Dataflow release notes

You can now pause a Dataflow batch job using the pause_on_failure service option. This feature lets you preserve the state of your batch pipeline job, address external issues, and resume processing without losing completed work. You can use this option to automatically pause a job on failure, or use this feature to manually pause a job when you chose to. For more information, see Pause a Dataflow job.

You can now use Hyperdisk Balanced disks for Dataflow worker VMs. With Hyperdisk Balanced disks, you can provision IOPS and throughput independently of disk size by using the diskProvisionedIOPS and diskProvisionedThroughput pipeline options (Java SDK) or disk_provisioned_iops and disk_provisioned_throughput_mibps pipeline options (Python and Go SDKs). For more information, see Disk type and Provision IOPS and throughput.

Dataflow now supports NVIDIA RTX Pro 6000 GPUs. You can use this GPU model to run your Apache Beam pipelines on Dataflow. RTX Pro 6000 GPUs are recommended for large, medium, and small model inference workloads. To configure your workers with this GPU model, set the accelerator type to nvidia-rtx-pro-6000. For more information, see Dataflow support for GPUs.

Dataflow has updated and expanded its pipeline update features for streaming jobs:

• Automated stop-and-replace updates: You can perform automated, declarative stop-and-replace updates to streaming jobs.
• Parallel updates with the same job name: When you perform automated parallel updates, you can use the same job name for the new replacement job.
• Auto-cancel draining jobs: When performing parallel or stop-and-replace updates, you can configure Dataflow to automatically cancel the old job if it does not finish draining after a timeout you specify.
• Update strategy configuration: You can explicitly choose between a parallel update (update_strategy_parallel_job_update) and a standard in-place update (update_strategy_in_place_update) while keeping all other configuration the same.
• Template upsert functionality: When launching pipelines from classic templates, flex templates, Terraform, or Config Connector, you can use the create_or_update_job experiment to enable automatic create-or-update (upsert) behavior. If an active job with the specified name already exists, it is updated. Otherwise, a new job is created.

For more information, see Automated stop and replace, Automated parallel pipeline updates, and Automatic create or update (upsert) for templates.

Dataflow job builder now supports external Iceberg REST Catalogs as a source. You can now ingest data from external Apache Iceberg REST catalogs (IRC) directly into Lakehouse for Apache Iceberg tables using Dataflow's job builder UI without writing code. For more information, see Import data from external Iceberg catalogs to Lakehouse using Dataflow.

You can now add existing Apache Parquet files from cloud-based storage, for example, Cloud Storage or Amazon S3, to an Apache Iceberg table in Lakehouse for Apache Iceberg using the Dataflow job builder. This process registers the files without moving or rewriting the underlying data. For more information, see Import Parquet files from storage to Lakehouse using Dataflow.

When you use min_ram or cpu_count resource hints for pipeline steps that don't require accelerators, Auto VM Selection (Instance Flexibility) is enabled automatically. With Auto VM Selection, workers are provisioned from a curated list of machine types that meet your RAM and CPU requirements. For more information, see Auto VM Selection for worker machine types.

Dataflow support for the C4A machine series of Arm processors is now generally available. Arm-based VMs are optimized for power efficiency and can provide improved price-performance for many workloads. For more information, see Use Arm VMs on Dataflow.

Dataflow Managed I/O now supports rolling upgrades for streaming jobs. With this feature, Dataflow upgrades your Managed I/O connectors in running pipelines as new connector versions become available. For more information, see Automatic upgrades.

Dataflow is available in the Bangkok (asia-southeast3) region. Learn more about Google Cloud locations.

Dataflow now serves a notice for when the Dataflow Runner v2 container image of a streaming pipeline will be upgraded. To use a new image and avoid the scheduled maintenance, launch a replacement job before the upgrade. For more information, see Runner v2 harness update.

Dataflow now supports speculative execution for batch pipelines. This feature mitigates the impact of slow-running tasks (stragglers) by launching a redundant execution of these tasks. The first task to finish is used, and the other is canceled, which can improve the overall completion time of your pipeline. This feature is generally available. For more information, see Use speculative execution to avoid stragglers.

For jobs that use GPUs, Dataflow now supports the flex-start provisioning model. This flex-start provisioning model can improve your ability to get access to constrained GPU resources for short-duration workloads. This feature is available in Preview and is for batch pipelines only. For more information, see Configure a provisioning model.

Dataflow now supports using secure tags to set firewall rules on worker VMs. For more information, see Use secure tags with Dataflow.

Dataflow supports TPUs, Google's custom-designed AI accelerators that are optimized for large-scale AI/ML workloads. This feature lets you accelerate inference workloads on frameworks like PyTorch, JAX, and TensorFlow. This feature is generally available with an allowlist. For more information, see Dataflow support for TPUs.

Dataflow supports specifically targeted reservations for pipelines using accelerators (GPUs or TPUs). This functionality is generally available with an allowlist. For more information, see Use Compute Engine reservations with Dataflow.

Dataflow supports NVIDIA® H100 and NVIDIA® H100 Mega GPU types. For more information, see Dataflow support for GPUs.

Dataflow Runner v2 fixes an issue that could cause data discrepancies when using splittable DoFns, particularly when processing large datasets as side inputs. This fix ensures that all data is accurately processed and transmitted within the pipeline. This improvement is available in recent Dataflow service releases, and is automatically enabled when using Dataflow Runner v2.

Note: After this fix, pipelines that previously experienced data loss due to this issue might consume more resources (such as CPU, memory, and processing time) because more data is being processed. This increase in resource usage is expected and reflects the correct behavior of the pipeline.

Dataflow now automatically detects performance bottlenecks in streaming jobs. You can see the cause of the bottleneck in the Step Info panel to help with troubleshooting.

For more information, see Troubleshoot bottlenecks.

Dataflow now supports an automated parallel update workflow for streaming jobs. This feature helps minimize disruption by launching a new replacement job that runs in parallel with the existing job. After a duration of time you specify, the old job is automatically drained.

For more information, see Run parallel pipelines.

Dataflow now supports right fitting for streaming jobs. Right fitting lets you specify resource requirements for an entire pipeline or for specific pipeline steps. Previously, right fitting was only supported for batch pipelines. For more information, see Streaming right fitting.

Dataflow now supports data lineage. Data lineage lets you track how data moves through your systems. This feature is generally available (GA). For more information, see Use data lineage in Dataflow.

Dataflow is now available in Stockholm (europe-north2).

Managed I/O now supports automatic upgrades for supported I/O connectors. Using this feature, Dataflow pipelines automatically use the latest reliable version of the connector. This feature is generally available (GA). For more information, see Dataflow managed I/O.

Dataflow is available in Queretaro, Mexico (northamerica-south1). Learn more about Google Cloud locations.

You can now use the Dataflow job builder UI to create and run Dataflow pipelines in the Google Cloud console, without writing any code. This feature is generally available (GA).

The remote code execution vulnerability, CVE-2024-6387, in OpenSSH has been mitigated. A patched Dataflow VM image that includes an updated OpenSSH is available. For more information about how to apply mitigations, see the GCP-2024-040 security bulletin.

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. Dataflow jobs might create VMs that use an OS image with versions of OpenSSH that are vulnerable to CVE-2024-6387. For more information, see the GCP-2024-040 security bulletin.

Dataflow batch jobs are now cancelled after ten days. Previously, they were cancelled after 30 days. See Quotas and limits.

Dataflow SQL is deprecated. As of July 31, 2024, you can't access Dataflow SQL in the Google Cloud console. As of January 31, 2025, you can't use Dataflow SQL in the Google Cloud CLI. As a replacement, use Beam SQL.

Iceberg read/write support is available through the new Managed I/O Java API. For more information, see Dataflow managed I/O.

You can now use Metrics Explorer to find individual DoFns that cause latencies in streaming jobs. These metrics are available in streaming pipelines that use Apache Beam 2.53.0 and later versions. The following new metrics are available:

• Average message processing time per DoFn (job/dofn_latency_average)
• Maximum message processing time per DoFn (job/dofn_latency_max)
• Minimum message processing time per DoFn (job/dofn_latency_min)
• Number of messages processed per DoFn (job/dofn_latency_num_messages)
• Oldest active message processing time per DoFn (job/oldest_active_message_age)
• Total message processing time per DoFn (job/dofn_latency_total)

For more information about Dataflow metrics, see Google Cloud metrics.

Dataflow no longer supports the NVIDIA Tesla K80 GPU type. For a list of supported GPU types, see Dataflow support for GPUs.

The following Dataflow templates now support user-defined functions (UDFs) written in Python:

• Cloud Storage Text to BigQuery
• Cloud Storage Text to BigQuery (Stream)
• Pub/Sub to BigQuery
• Pub/Sub Proto to BigQuery

The following Dataflow templates are generally available (GA):

• Google Cloud to Neo4j
• Pub/Sub to Datadog

The Dataflow right fitting feature is now supported by non-Prime batch pipelines.

You can now use worker utilization hints to tune horizontal autoscaling for streaming pipelines.

Added new autoscaling metrics:

• Autoscaling rationale chart: explains the factors driving autoscaling decisions
• Worker CPU utilization chart: shows current user worker CPU utilization and customer autoscaling hint value
• Timer backlog per stage: shows an estimate of time needed to materialize the output for windows whose timer has expired
• Parallel processing: the number of keys available for parallel processing

You can now use committed use discounts (CUDs) with Dataflow streaming jobs. Committed use discounts provide discounted prices in exchange for your commitment to continuously use a certain amount of Dataflow compute resources for a year or longer.

Streaming jobs created after March 7, 2024 automatically encrypt all user data with customer-managed encryption keys (CMEK). To enable this encryption for jobs created before March 7, 2024, drain or cancel the job, and then restart it.

Dataflow now supports at-least-once streaming mode. You can use this mode to achieve lower latency and reduced costs for workloads that can tolerate duplicate records. This feature is generally available (GA). For more information, see Set the pipeline streaming mode.

You can now use Gemma models in your Apache Beam inference pipelines. For more information, see Use Gemma open models with Dataflow.

You can now use a turnkey transform to enrich streaming data in your Dataflow pipeline. When you enrich data, you augment the raw data from one source by adding related data from a second source. For more information, see Enrich streaming data.

Dataflow Streaming Engine now supports resource-based billing. When you enable resource-based billing with Streaming Engine, you're billed for the total resources consumed by your job.

Dataflow is available in Johannesburg, South Africa (africa-south1).

Dataflow now supports data sampling for pipeline exceptions. With this feature, you can see samples of the data being processed when an unhandled exception occurs. Use exception sampling to help troubleshoot pipeline errors. For more information, see Use exception sampling.

You can now run a job graph validation check to verify whether a replacement job is valid before you launch the new job. For more information, see Validate a replacement job.

You can now archive completed Dataflow jobs. When you archive a Dataflow job, the job is moved from the Dataflow Jobs page in the console to the Archived jobs page. For more information, see Archive Dataflow jobs.

The Dataflow web-based monitoring interface now includes a dashboard that monitors your Dataflow jobs at the project level. For more information, see Dataflow project monitoring dashboard.

Dataflow supports NVIDIA® L4 and NVIDIA® A100 80 GB GPU types. For more information, see Dataflow support for GPUs.

The Cloud Spanner to Vertex AI Vector Search template is generally available (GA).

Dataflow jobs now scale to 4,000 worker VMs.

The Cloud Spanner to BigQuery template for batch pipelines is available in preview.

Dataflow is now available in Dammam, Saudi Arabia (me-central2).

Dataflow now supports the Tau T2A Arm machine series as a worker machine type. This feature is generally available (GA). For more information, see Use Arm VMs on Dataflow.

The following Dataflow templates are generally available (GA):

• BigQuery to Bigtable
• Pub/Sub to Splunk

Dataflow is available in Berlin (europe-west10).

You can now update streaming job options without stopping your job. For more information, see In-flight job option update.

Dataflow cost monitoring is generally available (GA).

The following Dataflow templates are generally available (GA):

• MySQL to BigQuery
• PostgreSQL to BigQuery
• SQL Server to BigQuery

Dynamic thread scaling is generally available (GA). Dynamic thread scaling is a part of Dataflow's suite of vertical scaling features.

When you run multiple SDK processes on a shared Dataflow GPU, you can improve GPU efficiency and utilization by enabling the NVIDIA Multi-Process Service (MPS).

You can now view streaming stragglers in the Google Cloud console. For more information, see Troubleshoot stragglers in streaming jobs.

Dataflow now supports Confidential VMs for Dataflow worker VMs. For more information, see Dataflow service options.

Data sampling is now generally available (GA). Data sampling lets you observe the data at each step of a pipeline. For more information, see Use data sampling to observe pipeline data.

Dataflow ML now supports the Automatic Model Refresh feature, which lets you update your machine learning model without stopping your Apache Beam pipeline.

You can now manage Dataflow jobs by using Eventarc. For more information, see Use Eventarc to manage Dataflow jobs.

Dataflow cost monitoring is now available in preview.

Dataflow is now available in Doha (me-central1).

The Dataflow VM image has been updated to include mitigations for multiple vulnerabilities by upgrading to cos-97-16919-235-30. For the full list of mitigations, see the Container-Optimized OS release notes.

Dataflow jobs started on or after March 29, 2023 will run VM instances that use this image.

Vertical Autoscaling now supports batch jobs.

Dataflow is now available in Turin (europe-west12).

Starting in version 2023-01-03-00_RC00, the Google-provided Dataflow templates support ES6 syntax for JavaScript user-defined functions (UDFs). This change is backwards-compatible. ES5 syntax and existing user-defined functions are still supported.

When you run Google-provided templates using the latest version, your jobs are upgraded automatically on restart. If you want to keep running an earlier version of a template, when you run the template, specify version 2022-12-15-00_RC00 or earlier.

Starting with Beam SDK version 2.44.0, Dataflow will not support running Dataflow jobs with workers in a region that is different from the Dataflow regional endpoint.

Dataflow now supports regional placement for workers.

The Dataflow VM image has been updated to include mitigations for OpenSSL CVE-2022-3602 by upgrading to cos-97-16919-189-12. For jobs that use GPUs, the NVIDIA drivers have also been updated to mitigate the vulnerability. Dataflow jobs started on or after December 14, 2022 will run VM instances that use this image.

Dataflow is now available in Tel Aviv (me-west1).

The Dataflow VM image has been updated to include several mitigations for a recently disclosed hardware speculative execution vulnerability named Retbleed. Dataflow jobs started on or after September 21, 2022 will run VM instances that use this image.

Dataflow now uses Regional Managed Instance Groups (MIGs). Previously, Dataflow used zonal MIGs.

If this change causes you to exceed your quota, set your Regional managed instance groups quota to the same limit assigned to your Managed instance groups quota. For more information, see Working with quotas.

Dataflow Prime is now in General Availability.

You can use the Apache Beam SDK for Go to create batch and streaming Dataflow pipelines. This feature is now in General Availability.

Dataflow is now available in Dallas, Texas (us-south1).

Dataflow is now available in Columbus (us-east5).

Dataflow now supports Flex Template images from private registries. To learn more, see Use an image from a private registry.

Dataflow is now available in Madrid (europe-southwest1).

Dataflow is now available in Paris (europe-west9).

Dataflow is now available in Milan (europe-west8).

Dataflow now supports Runner v2 in GA for all languages.

Hot key detection and logging is deprecated for streaming pipelines. For information on pipeline debugging, see Pipeline troubleshooting and debugging.

Dataflow now supports the following Google-provided templates in GA:

• Pub/Sub Subscription to BigQuery
• Pub/Sub Topic to BigQuery
• Pub/Sub Avro to BigQuery
• Pub/Sub Proto to BigQuery
• Pub/Sub to Pub/Sub
• Pub/Sub to Cloud Storage Avro
• Pub/Sub to Cloud Storage Text
• Cloud Storage Text to BigQuery (Stream)
• Cloud Storage Text to Pub/Sub (Stream)
• Change Data Capture to BigQuery (Stream)
• Apache Kafka to BigQuery
• BigQuery export to Parquet (via Storage API)
• Firestore to Cloud Storage Text
• Cloud Spanner to Cloud Storage Text
• Cloud Storage Text to BigQuery
• Cloud Storage Text to Firestore
• Cloud Storage Text to Pub/Sub (Batch)
• Apache Cassandra to Bigtable
• Datastream to Cloud Spanner
• File Format Conversion
• Bulk Compress Cloud Storage Files
• Bulk Decompress Cloud Storage Files
• Firestore Bulk Delete
• Streaming Data Generator to Pub/Sub, BigQuery, and Cloud Storage

You can now use the Apache Beam SDK for Go to create batch Dataflow pipelines. This feature is in Preview.

Profiling Dataflow pipelines with Cloud Profiler is generally available (GA). Use Dataflow integration with Cloud Profiler to monitor pipeline performance.

Dataflow now fully supports using Identity and Access Management (IAM) custom roles. You can create a custom IAM role and assign it to a user-managed service account used in Dataflow instead of assigning the Dataflow Worker role.

Dataflow is now available in Santiago (southamerica-west1).

Dataflow now uses Zonal DNS for worker resources. This enables Dataflow to offer higher reliability guarantees around Internal DNS registration.

Dataflow now supports Shielded VM workers.

Dataflow Prime is now available in Preview.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Toronto (northamerica-northeast2).

Dataflow now supports storing Flex Template images in Artifact Registry.

Dataflow now supports custom containers in GA.

GPU support on Dataflow is now in General Availability.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Delhi (asia-south2).

Dataflow snapshots are now available in GA.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Melbourne (australia-southeast2).

In addition to scalar functions, Dataflow SQL now supports aggregate user-defined functions (UDFs) for Java. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

Dataflow SQL now supports user-defined functions (UDFs) written using Java. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

You can now enable logging of human-readable hot keys. For more information, see the hot key entry in Pipeline options.

Dataflow Shuffle is now the default mode for all batch pipelines.

Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in europe-central2 (Warsaw).

Dataflow SQL now supports user-defined functions (UDFs) written using SQL. For more information, see Dataflow SQL user-defined functions. This feature is in Preview.

Execution details are now available in Preview.

Dataflow now supports Dataflow Shuffle, Streaming Engine, FlexRS, and the following regional endpoints in GA:

• asia-east2 Hong Kong
• asia-northeast2 - Japan (Osaka)
• asia-northeast3 - Seoul
• asia-southeast2 - Jakarta
• europe-north1 - Finland
• us-west3 - Salt Lake City
• us-west4 - Las Vegas

Flex templates now support updating streaming jobs and Flexible Resource Scheduling (FlexRS).

Dataflow snapshots are now available in Preview.

GPU support on Dataflow is currently available in Preview. To enroll in this Preview offering, contact Support or Sales.