Dataflow release notes

Dataflow job builder now supports Delta Lake tables as a source. You can now import existing Delta Lake tables from Cloud Storage (gs://BUCKET_NAME) directly into Lakehouse for Apache Iceberg using Dataflow's job builder UI without writing code. For more information, see Import Delta Lake tables into Lakehouse using Dataflow.

You can now pause a Dataflow batch job using the pause_on_failure service option. This feature lets you preserve the state of your batch pipeline job, address external issues, and resume processing without losing completed work. You can use this option to automatically pause a job on failure, or use this feature to manually pause a job when you chose to. For more information, see Pause a Dataflow job.

You can now use Hyperdisk Balanced disks for Dataflow worker VMs. With Hyperdisk Balanced disks, you can provision IOPS and throughput independently of disk size by using the diskProvisionedIOPS and diskProvisionedThroughput pipeline options (Java SDK) or disk_provisioned_iops and disk_provisioned_throughput_mibps pipeline options (Python and Go SDKs). For more information, see Disk type and Provision IOPS and throughput.

Dataflow now supports NVIDIA RTX Pro 6000 GPUs. You can use this GPU model to run your Apache Beam pipelines on Dataflow. RTX Pro 6000 GPUs are recommended for large, medium, and small model inference workloads. To configure your workers with this GPU model, set the accelerator type to nvidia-rtx-pro-6000. For more information, see Dataflow support for GPUs.

Dataflow has updated and expanded its pipeline update features for streaming jobs:

• Automated stop-and-replace updates: You can perform automated, declarative stop-and-replace updates to streaming jobs.
• Parallel updates with the same job name: When you perform automated parallel updates, you can use the same job name for the new replacement job.
• Auto-cancel draining jobs: When performing parallel or stop-and-replace updates, you can configure Dataflow to automatically cancel the old job if it does not finish draining after a timeout you specify.
• Update strategy configuration: You can explicitly choose between a parallel update (update_strategy_parallel_job_update) and a standard in-place update (update_strategy_in_place_update) while keeping all other configuration the same.
• Template upsert functionality: When launching pipelines from classic templates, flex templates, Terraform, or Config Connector, you can use the create_or_update_job experiment to enable automatic create-or-update (upsert) behavior. If an active job with the specified name already exists, it is updated. Otherwise, a new job is created.

For more information, see Automated stop and replace, Automated parallel pipeline updates, and Automatic create or update (upsert) for templates.

Dataflow job builder now supports external Iceberg REST Catalogs as a source. You can now ingest data from external Apache Iceberg REST catalogs (IRC) directly into Lakehouse for Apache Iceberg tables using Dataflow's job builder UI without writing code. For more information, see Import data from external Iceberg catalogs to Lakehouse using Dataflow.

You can now add existing Apache Parquet files from cloud-based storage, for example, Cloud Storage or Amazon S3, to an Apache Iceberg table in Lakehouse for Apache Iceberg using the Dataflow job builder. This process registers the files without moving or rewriting the underlying data. For more information, see Import Parquet files from storage to Lakehouse using Dataflow.

When you use min_ram or cpu_count resource hints for pipeline steps that don't require accelerators, Auto VM Selection (Instance Flexibility) is enabled automatically. With Auto VM Selection, workers are provisioned from a curated list of machine types that meet your RAM and CPU requirements. For more information, see Auto VM Selection for worker machine types.

Dataflow support for the C4A machine series of Arm processors is now generally available. Arm-based VMs are optimized for power efficiency and can provide improved price-performance for many workloads. For more information, see Use Arm VMs on Dataflow.

Dataflow Managed I/O now supports rolling upgrades for streaming jobs. With this feature, Dataflow upgrades your Managed I/O connectors in running pipelines as new connector versions become available. For more information, see Automatic upgrades.

Dataflow is available in the Bangkok (asia-southeast3) region. Learn more about Google Cloud locations.

Dataflow now serves a notice for when the Dataflow Runner v2 container image of a streaming pipeline will be upgraded. To use a new image and avoid the scheduled maintenance, launch a replacement job before the upgrade. For more information, see Runner v2 harness update.

Dataflow now supports speculative execution for batch pipelines. This feature mitigates the impact of slow-running tasks (stragglers) by launching a redundant execution of these tasks. The first task to finish is used, and the other is canceled, which can improve the overall completion time of your pipeline. This feature is generally available. For more information, see Use speculative execution to avoid stragglers.

For jobs that use GPUs, Dataflow now supports the flex-start provisioning model. This flex-start provisioning model can improve your ability to get access to constrained GPU resources for short-duration workloads. This feature is available in Preview and is for batch pipelines only. For more information, see Configure a provisioning model.

Dataflow now supports using secure tags to set firewall rules on worker VMs. For more information, see Use secure tags with Dataflow.

Dataflow supports TPUs, Google's custom-designed AI accelerators that are optimized for large-scale AI/ML workloads. This feature lets you accelerate inference workloads on frameworks like PyTorch, JAX, and TensorFlow. This feature is generally available with an allowlist. For more information, see Dataflow support for TPUs.

Dataflow supports specifically targeted reservations for pipelines using accelerators (GPUs or TPUs). This functionality is generally available with an allowlist. For more information, see Use Compute Engine reservations with Dataflow.

Dataflow supports NVIDIA® H100 and NVIDIA® H100 Mega GPU types. For more information, see Dataflow support for GPUs.

Dataflow Runner v2 fixes an issue that could cause data discrepancies when using splittable DoFns, particularly when processing large datasets as side inputs. This fix ensures that all data is accurately processed and transmitted within the pipeline. This improvement is available in recent Dataflow service releases, and is automatically enabled when using Dataflow Runner v2.

Note: After this fix, pipelines that previously experienced data loss due to this issue might consume more resources (such as CPU, memory, and processing time) because more data is being processed. This increase in resource usage is expected and reflects the correct behavior of the pipeline.

Dataflow now automatically detects performance bottlenecks in streaming jobs. You can see the cause of the bottleneck in the Step Info panel to help with troubleshooting.

For more information, see Troubleshoot bottlenecks.

Dataflow now supports an automated parallel update workflow for streaming jobs. This feature helps minimize disruption by launching a new replacement job that runs in parallel with the existing job. After a duration of time you specify, the old job is automatically drained.

For more information, see Run parallel pipelines.

Dataflow now supports right fitting for streaming jobs. Right fitting lets you specify resource requirements for an entire pipeline or for specific pipeline steps. Previously, right fitting was only supported for batch pipelines. For more information, see Streaming right fitting.

Dataflow now supports data lineage. Data lineage lets you track how data moves through your systems. This feature is generally available (GA). For more information, see Use data lineage in Dataflow.

Dataflow is now available in Stockholm (europe-north2).

Managed I/O now supports automatic upgrades for supported I/O connectors. Using this feature, Dataflow pipelines automatically use the latest reliable version of the connector. This feature is generally available (GA). For more information, see Dataflow managed I/O.

Dataflow is available in Queretaro, Mexico (northamerica-south1). Learn more about Google Cloud locations.

You can now use the Dataflow job builder UI to create and run Dataflow pipelines in the Google Cloud console, without writing any code. This feature is generally available (GA).

The remote code execution vulnerability, CVE-2024-6387, in OpenSSH has been mitigated. A patched Dataflow VM image that includes an updated OpenSSH is available. For more information about how to apply mitigations, see the GCP-2024-040 security bulletin.

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Dataflow jobs might create VMs that use an OS image with versions of OpenSSH that are vulnerable to CVE-2024-6387. For more information, see the GCP-2024-040 security bulletin.

Dataflow batch jobs are now cancelled after ten days. Previously, they were cancelled after 30 days. See Quotas and limits.

Dataflow SQL is deprecated. As of July 31, 2024, you can't access Dataflow SQL in the Google Cloud console. As of January 31, 2025, you can't use Dataflow SQL in the Google Cloud CLI. As a replacement, use Beam SQL.

Iceberg read/write support is available through the new Managed I/O Java API. For more information, see Dataflow managed I/O.

You can now use Metrics Explorer to find individual DoFns that cause latencies in streaming jobs. These metrics are available in streaming pipelines that use Apache Beam 2.53.0 and later versions. The following new metrics are available:

• Average message processing time per DoFn (job/dofn_latency_average)
• Maximum message processing time per DoFn (job/dofn_latency_max)
• Minimum message processing time per DoFn (job/dofn_latency_min)
• Number of messages processed per DoFn (job/dofn_latency_num_messages)
• Oldest active message processing time per DoFn (job/oldest_active_message_age)
• Total message processing time per DoFn (job/dofn_latency_total)

For more information about Dataflow metrics, see Google Cloud metrics.

Dataflow no longer supports the NVIDIA Tesla K80 GPU type. For a list of supported GPU types, see Dataflow support for GPUs.

The following Dataflow templates now support user-defined functions (UDFs) written in Python:

• Cloud Storage Text to BigQuery
• Cloud Storage Text to BigQuery (Stream)
• Pub/Sub to BigQuery
• Pub/Sub Proto to BigQuery

The following Dataflow templates are generally available (GA):

• Google Cloud to Neo4j
• Pub/Sub to Datadog

The Dataflow right fitting feature is now supported by non-Prime batch pipelines.

You can now use worker utilization hints to tune horizontal autoscaling for streaming pipelines.

Added new autoscaling metrics:

• Autoscaling rationale chart: explains the factors driving autoscaling decisions
• Worker CPU utilization chart: shows current user worker CPU utilization and customer autoscaling hint value
• Timer backlog per stage: shows an estimate of time needed to materialize the output for windows whose timer has expired
• Parallel processing: the number of keys available for parallel processing

You can now use committed use discounts (CUDs) with Dataflow streaming jobs. Committed use discounts provide discounted prices in exchange for your commitment to continuously use a certain amount of Dataflow compute resources for a year or longer.

Streaming jobs created after March 7, 2024 automatically encrypt all user data with customer-managed encryption keys (CMEK). To enable this encryption for jobs created before March 7, 2024, drain or cancel the job, and then restart it.

Dataflow now supports at-least-once streaming mode. You can use this mode to achieve lower latency and reduced costs for workloads that can tolerate duplicate records. This feature is generally available (GA). For more information, see Set the pipeline streaming mode.

You can now use Gemma models in your Apache Beam inference pipelines. For more information, see Use Gemma open models with Dataflow.

You can now use a turnkey transform to enrich streaming data in your Dataflow pipeline. When you enrich data, you augment the raw data from one source by adding related data from a second source. For more information, see Enrich streaming data.

Dataflow Streaming Engine now supports resource-based billing. When you enable resource-based billing with Streaming Engine, you're billed for the total resources consumed by your job.

Dataflow is available in Johannesburg, South Africa (africa-south1).

Dataflow now supports data sampling for pipeline exceptions. With this feature, you can see samples of the data being processed when an unhandled exception occurs. Use exception sampling to help troubleshoot pipeline errors. For more information, see Use exception sampling.

You can now run a job graph validation check to verify whether a replacement job is valid before you launch the new job. For more information, see Validate a replacement job.

You can now archive completed Dataflow jobs. When you archive a Dataflow job, the job is moved from the Dataflow Jobs page in the console to the Archived jobs page. For more information, see Archive Dataflow jobs.

The Dataflow web-based monitoring interface now includes a dashboard that monitors your Dataflow jobs at the project level. For more information, see Dataflow project monitoring dashboard.

Dataflow supports NVIDIA® L4 and NVIDIA® A100 80 GB GPU types. For more information, see Dataflow support for GPUs.

The Cloud Spanner to Vertex AI Vector Search template is generally available (GA).

Dataflow jobs now scale to 4,000 worker VMs.

The Cloud Spanner to BigQuery template for batch pipelines is available in preview.

Dataflow is now available in Dammam, Saudi Arabia (me-central2).

Dataflow now supports the Tau T2A Arm machine series as a worker machine type. This feature is generally available (GA). For more information, see Use Arm VMs on Dataflow.