Dataflow release notes

Dataflow job builder now supports Delta Lake tables as a source. You can now import existing Delta Lake tables from Cloud Storage (gs://BUCKET_NAME) directly into Lakehouse for Apache Iceberg using Dataflow's job builder UI without writing code. For more information, see Import Delta Lake tables into Lakehouse using Dataflow.

You can now pause a Dataflow batch job using the pause_on_failure service option. This feature lets you preserve the state of your batch pipeline job, address external issues, and resume processing without losing completed work. You can use this option to automatically pause a job on failure, or use this feature to manually pause a job when you chose to. For more information, see Pause a Dataflow job.

You can now use Hyperdisk Balanced disks for Dataflow worker VMs. With Hyperdisk Balanced disks, you can provision IOPS and throughput independently of disk size by using the diskProvisionedIOPS and diskProvisionedThroughput pipeline options (Java SDK) or disk_provisioned_iops and disk_provisioned_throughput_mibps pipeline options (Python and Go SDKs). For more information, see Disk type and Provision IOPS and throughput.

Dataflow now supports NVIDIA RTX Pro 6000 GPUs. You can use this GPU model to run your Apache Beam pipelines on Dataflow. RTX Pro 6000 GPUs are recommended for large, medium, and small model inference workloads. To configure your workers with this GPU model, set the accelerator type to nvidia-rtx-pro-6000. For more information, see Dataflow support for GPUs.

Dataflow has updated and expanded its pipeline update features for streaming jobs:

• Automated stop-and-replace updates: You can perform automated, declarative stop-and-replace updates to streaming jobs.
• Parallel updates with the same job name: When you perform automated parallel updates, you can use the same job name for the new replacement job.
• Auto-cancel draining jobs: When performing parallel or stop-and-replace updates, you can configure Dataflow to automatically cancel the old job if it does not finish draining after a timeout you specify.
• Update strategy configuration: You can explicitly choose between a parallel update (update_strategy_parallel_job_update) and a standard in-place update (update_strategy_in_place_update) while keeping all other configuration the same.
• Template upsert functionality: When launching pipelines from classic templates, flex templates, Terraform, or Config Connector, you can use the create_or_update_job experiment to enable automatic create-or-update (upsert) behavior. If an active job with the specified name already exists, it is updated. Otherwise, a new job is created.

For more information, see Automated stop and replace, Automated parallel pipeline updates, and Automatic create or update (upsert) for templates.

Dataflow job builder now supports external Iceberg REST Catalogs as a source. You can now ingest data from external Apache Iceberg REST catalogs (IRC) directly into Lakehouse for Apache Iceberg tables using Dataflow's job builder UI without writing code. For more information, see Import data from external Iceberg catalogs to Lakehouse using Dataflow.

You can now add existing Apache Parquet files from cloud-based storage, for example, Cloud Storage or Amazon S3, to an Apache Iceberg table in Lakehouse for Apache Iceberg using the Dataflow job builder. This process registers the files without moving or rewriting the underlying data. For more information, see Import Parquet files from storage to Lakehouse using Dataflow.

When you use min_ram or cpu_count resource hints for pipeline steps that don't require accelerators, Auto VM Selection (Instance Flexibility) is enabled automatically. With Auto VM Selection, workers are provisioned from a curated list of machine types that meet your RAM and CPU requirements. For more information, see Auto VM Selection for worker machine types.

Dataflow support for the C4A machine series of Arm processors is now generally available. Arm-based VMs are optimized for power efficiency and can provide improved price-performance for many workloads. For more information, see Use Arm VMs on Dataflow.

Dataflow Managed I/O now supports rolling upgrades for streaming jobs. With this feature, Dataflow upgrades your Managed I/O connectors in running pipelines as new connector versions become available. For more information, see Automatic upgrades.

Dataflow is available in the Bangkok (asia-southeast3) region. Learn more about Google Cloud locations.

Dataflow now serves a notice for when the Dataflow Runner v2 container image of a streaming pipeline will be upgraded. To use a new image and avoid the scheduled maintenance, launch a replacement job before the upgrade. For more information, see Runner v2 harness update.

Dataflow now supports speculative execution for batch pipelines. This feature mitigates the impact of slow-running tasks (stragglers) by launching a redundant execution of these tasks. The first task to finish is used, and the other is canceled, which can improve the overall completion time of your pipeline. This feature is generally available. For more information, see Use speculative execution to avoid stragglers.

For jobs that use GPUs, Dataflow now supports the flex-start provisioning model. This flex-start provisioning model can improve your ability to get access to constrained GPU resources for short-duration workloads. This feature is available in Preview and is for batch pipelines only. For more information, see Configure a provisioning model.

Dataflow now supports using secure tags to set firewall rules on worker VMs. For more information, see Use secure tags with Dataflow.

Dataflow supports TPUs, Google's custom-designed AI accelerators that are optimized for large-scale AI/ML workloads. This feature lets you accelerate inference workloads on frameworks like PyTorch, JAX, and TensorFlow. This feature is generally available with an allowlist. For more information, see Dataflow support for TPUs.

Dataflow supports specifically targeted reservations for pipelines using accelerators (GPUs or TPUs). This functionality is generally available with an allowlist. For more information, see Use Compute Engine reservations with Dataflow.

Dataflow supports NVIDIA® H100 and NVIDIA® H100 Mega GPU types. For more information, see Dataflow support for GPUs.

Dataflow Runner v2 fixes an issue that could cause data discrepancies when using splittable DoFns, particularly when processing large datasets as side inputs. This fix ensures that all data is accurately processed and transmitted within the pipeline. This improvement is available in recent Dataflow service releases, and is automatically enabled when using Dataflow Runner v2.

Note: After this fix, pipelines that previously experienced data loss due to this issue might consume more resources (such as CPU, memory, and processing time) because more data is being processed. This increase in resource usage is expected and reflects the correct behavior of the pipeline.

Dataflow now automatically detects performance bottlenecks in streaming jobs. You can see the cause of the bottleneck in the Step Info panel to help with troubleshooting.

For more information, see Troubleshoot bottlenecks.

Dataflow now supports an automated parallel update workflow for streaming jobs. This feature helps minimize disruption by launching a new replacement job that runs in parallel with the existing job. After a duration of time you specify, the old job is automatically drained.

For more information, see Run parallel pipelines.

Dataflow now supports right fitting for streaming jobs. Right fitting lets you specify resource requirements for an entire pipeline or for specific pipeline steps. Previously, right fitting was only supported for batch pipelines. For more information, see Streaming right fitting.