Google Cloud has announced a new feature in Dataflow that enables custom source reads with load balancing, aiming to cut costs and boost efficiency. This comes as a welcome solution to the challenge of scaling workloads, especially in streaming environments where latency is closely monitored.
Many modern autotuning strategies struggle to cope with hot keys or hot workers that bottleneck processing and create backlogs, impacting data freshness. For instance, a streaming environment like Apache Kafka can create hot spots in the pipeline. An autoscaler may try to compensate after the fact with additional compute units, but this is not only costly, it’s also slow. An autoscaler only reacts after there’s a backlog of accumulated messages and incurs overhead spinning up new workers.
The new load balancing feature works by better distributing workloads and proactively relieving overwhelmed workers. This allows pipelines to push more data with fewer resources and lower latencies. Real-world use cases from top Dataflow customers demonstrate the effectiveness of this feature in reducing operational costs and improving pipeline performance.
For example, one customer was able to reduce worker scaling events by 75%, resulting in a daily cost reduction of 64% in Google Compute Engine, and the backlog dropped from ~1min to ~10s.
This load balancing feature is turned on by default for all Dataflow customers across all regions, making it readily available to leverage without requiring additional configuration.
In conclusion, the introduction of custom source reads with load balancing in Dataflow marks a significant step towards improving pipeline efficiency and reducing costs, particularly in streaming environments where speed and efficiency are paramount.
