Feature
Dataflow has added the following features to tune horizontal autoscaling for Streaming Engine jobs:
Dataflow has added the following features to tune horizontal autoscaling for Streaming Engine jobs:
- Latency tiers: You can configure the backlog threshold by choosing a latency tier (low_latency, medium_latency, or high_latency), allowing you to configure the autoscaler's sensitivity to backlog duration and balance latency requirements with resource costs. For more information, see Set the latency tier.
- Autoscaling schedules: You can configure recurring autoscaling schedules to automatically apply parameter overrides—such as minimum and maximum workers, worker utilization hints, or latency tiers—during specific time windows. For more information, see Schedule autoscaling changes.
You can configure these features when you create a job or update them while the job is running.
Read the original announcement
Get Dataflow updates by email
Don't miss the next one: a weekly email with Dataflow's new features and deprecations.