Managed Service for Apache Spark release notes

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.86
• 2.2.86
• 2.3.39

Key updates in these runtime versions include:

• OpenLineage updates: In the 2.3 runtime:
  • Upgraded OpenLineage to version 1.49 to support lineage for tables created using the Lakehouse Runtime catalog.
  • Fixed a segmentation fault when OpenLineage parses complex SQL query strings.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

A critical bug related to Conda channels has been fixed in-place in image versions 1.4.81, 1.5.92, 2.1.117, and 2.2.85. These image versions were released without pre-configured Conda channels.

Required customer actions: To comply with Google requirements, recreate the following resources if they were created using these image versions on or before August 10, 2026:

• Custom images
• Clusters

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.85
• 2.2.85
• 2.3.38

Notes:

• Apache Spark upgraded to 3.5.3 in 2.2 runtime.

• Apache Gluten upgraded to 1.6 in 2.3 runtime.

Managed Service for Apache Spark latest image and runtime versions:

• Configured spark.scheduler.listenerbus.exitTimeout to 30s.

New Managed Service for Apache Spark on Google Kubernetes Engine (formerly Dataproc on Google Kubernetes Engine) subminor image version:

• 3.5-dataproc-28

Key updates in this image version include:

• Conda channels: The new 3.5-dataproc-28 subminor image version doesn't have preconfigured Conda channels, and is mapped to default aliases (such as 3.5 and latest).
  • Impact: When creating clusters with 3.5-dataproc-28 or using default aliases (3.5, latest), packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels, pin your clusters to the previous image versions before August 25, 2026.
  • Default change schedule: All workloads must transition to image versions without preconfigured Conda channels after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Starting with Managed Service for Apache Spark image version 3.0, clusters that are created without a specified machine type for a node are created with a Flex VM configuration for the node.

Announcing the General Availability (GA) release of Managed Service for Apache Spark cluster image version 3.0, as follows:

• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, and 3.0.0-ubuntu24.

  • The 3.0.0-ml-ubuntu24 image extends the 3.0 base image with ML-specific libraries.

Image version 3.0 is a lightweight image that contains the following pre-installed core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs):

• Apache Hadoop 3.5.0.
• Apache Hive 4.2.0
• Apache Spark 4.1.2
• Apache Tez 0.10.5
• Cloud Storage Connector 3.1.13
• Conscrypt 2.6
• Java 21
• Python 3.12
• R 4.5
• Scala 2.13.17
• Spark-BigQuery Connector 0.44.1-preview

You can add listed optional components when you create a 3.0 image version cluster.

Recommendation: Use the 3.0 (and later) image versions to meet security compliance requirements.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.34-debian12, 2.3.34-ml-ubuntu22, 2.3.34-rocky9, 2.3.34-ubuntu22, 2.3.34-ubuntu22-arm
• 3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, 3.0.0-ubuntu24

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

• The 2.1, 2.2 and 2.3 cluster image versions now support Confidential Compute for the g4-standard-48 GPU machine type.

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):

• The 3.0 runtime now uses fewer executors, as follows:

  • 0 min executors for spark.dynamicAllocation.minExecutors property
  • 1 min executor for spark.executor.instances and spark.dynamicAllocation.initialExecutors properties

• All runtimes now configure spark.scheduler.listenerbus.exitTimeout to 30 seconds.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.117-debian11, 2.1.117-rocky8, 2.1.117-ubuntu20, 2.1.117-ubuntu20-arm
• 2.2.85-debian12, 2.2.85-rocky9, 2.2.85-ubuntu22, 2.2.85-ubuntu22-arm
• 2.3.33-debian12, 2.3.33-ml-ubuntu22, 2.3.33-rocky9, 2.3.33-ubuntu22, 2.3.33-ubuntu22-arm

Key updates in these image versions include:

• Conda channels: The new 2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
  • Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
  • Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions (for example, 2.1.116-debian11 or 2.2.84-debian12).
  • Default change schedule: After August 25, 2026, the default aliases for 2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32, don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026. The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels, but note the following Default change schedule:, which will impact all supported image versions.

• Impact: When creating clusters with these image versions, specify the exact subminor version (for example, 2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.
• Mitigation: If your workloads require preconfigured Conda channels or default aliases, pin your clusters to the previous image versions.
• Default change schedule: The subminor versions 1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026. Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases. All workloads must use new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.

You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 1.3.96-debian10, 1.3.96-ubuntu18
• 1.4.81-debian10, 1.4.81-ubuntu18
• 1.5.92-debian10, 1.5.92-rocky8, 1.5.92-ubuntu18
• 2.0.161-debian10, 2.0.161-rocky8, 2.0.161-ubuntu18
• 2.1.115-debian11, 2.1.115-rocky8, 2.1.115-ubuntu20, 2.1.115-ubuntu20-arm
• 2.1.116-debian11, 2.1.116-rocky8, 2.1.116-ubuntu20, 2.1.116-ubuntu20-arm
• 2.2.83-debian12, 2.2.83-rocky9, 2.2.83-ubuntu22, 2.2.83-ubuntu22-arm
• 2.2.84-debian12, 2.2.84-rocky9, 2.2.84-ubuntu22, 2.2.84-ubuntu22-arm
• 2.3.31-debian12, 2.3.31-ml-ubuntu22, 2.3.31-rocky9, 2.3.31-ubuntu22, 2.3.31-ubuntu22-arm
• 2.3.32-debian12, 2.3.32-ml-ubuntu22, 2.3.32-rocky9, 2.3.32-ubuntu22, 2.3.32-ubuntu22-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:

• Iceberg support: Added support for Iceberg 1.10 in Dataproc 2.3 images. This change only applies to 2.3 clusters created with the ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.
• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default in new GCE image versions.
• Scheduler exitTimeout: Configured spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.82
• 2.2.82
• 2.3.35

Key updates in these runtime versions include:

• Spark optimizations: Enabled Spark skewed-join and self-join optimizations by default.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.

• Introduced confidentialInstanceType enum in the API.
• The --enable-confidential-compute flag and enableConfidentialCompute field are deprecated in favor of the new type-specific flag/field.
• Clusters created with the deprecated boolean flag will default to SEV.
• Added validation for machine type compatibility for SEV, SEV-SNP, and TDX.
• Updated live migration logic to support compatible machine types and CPU platforms for each technology, including N2D and C3D for SEV.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on May 19, 2026 have been rolled back:

• 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.114-debian11, 2.1.114-rocky8, 2.1.114-ubuntu20, 2.1.114-ubuntu20-arm
• 2.2.82-debian12, 2.2.82-rocky9, 2.2.82-ubuntu22, 2.2.82-ubuntu22-arm
• 2.3.30-debian12, 2.3.30-ml-ubuntu22, 2.3.30-rocky9, 2.3.30-ubuntu22, 2.3.30-ubuntu22-arm

Rollback Notice: The 2.2.82 image versions were rolled back on May 25, 2026.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.

This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.

• Impact: This change only affects users who are programmatically setting the configuration in code using spark.conf.set() with an integer literal.
  • Impacted example: spark.conf.set("spark.sql.shuffle.partitions", 100)
• User action: Update your code to pass a string literal instead of an integer.
  • Example fix: spark.conf.set("spark.sql.shuffle.partitions", "100")
• Not impacted: Setting the configuration via command-line arguments (e.g., spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.80
• 2.2.80
• 2.3.33

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):

• 3.0.0-RC2-debian13, 3.0.0-RC2-ml-ubuntu24, 3.0.0-RC2-rocky9, 3.0.0-RC2-ubuntu24

Key updates in this release:

• The debian image for 3.0 is upgraded to Debian 13.
• The default Java runtime is upgraded to Java 21.
• Apache Hadoop upgraded to 3.5.0.
• Apache Spark upgraded to 4.1.1.
• Apache Hive upgraded to 4.2.0.
• Trino upgraded to 480.
• Apache Solr upgraded to 9.10.1.
• Apache Kafka upgraded to 3.9.2.
• Docker upgraded to 28.1.
• Apache Flink upgraded to 2.2.0.
• Scala upgraded to 2.13.17.
• Cloud Storage Connector upgraded to 3.1.13.
• Apache Zookeeper upgraded to 3.9.5.
• BigQuery Connector upgraded to 0.44.1-Preview.
• Zeppelin Notebook upgraded to 0.12.0.
• JupyterLab upgraded to 4.5.7.
• Pixi is used as python package manager instead of conda.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.79
• 2.2.79
• 2.3.32

Key updates in this release:

• Upgraded Metastore Proxy version to v0.0.79.
• Upgraded Spark RAPIDS to version 26.04.0 version in the Managed Service for Apache Spark version 3.0 serverless runtime.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.1.113-debian11, 2.1.113-rocky8, 2.1.113-ubuntu20, 2.1.113-ubuntu20-arm
• 2.2.81-debian12, 2.2.81-rocky9, 2.2.81-ubuntu22, 2.2.81-ubuntu22-arm
• 2.3.29-debian12, 2.3.29-ml-ubuntu22, 2.3.29-rocky9, 2.3.29-ubuntu22, 2.3.29-ubuntu22-arm

Key updates in this release:

• The native runtime engine is now supported on base Ubuntu cluster images starting from image version 2.3.29-ubuntu22 (not supported on Arm and ML images).

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:

• Flexible VMs: Minimize stockouts and improve machine obtainability by defining prioritized lists for your master, primary, and secondary worker VM types. Managed Spark for Apache Spark utilizes these lists to create your cluster, and selects the optimal VM type based on current capacity, quotas, and existing reservations.
• Cluster Scheduled Stop: Optimize cost and maintain your cluster configuration by stopping clusters after a specified idle period, at a specified future time, or after a specified period from cluster creation or a cluster update request. All custom cluster configurations are restored once you restart the cluster.
• Zero-scale clusters: Reduce costs by creating only secondary workers, scaling them down to zero when they are not in use.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

• 1.2.78
• 2.2.78
• 2.3.31

Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

• 2.3.28-debian12, 2.3.28-ml-ubuntu22, 2.3.28-rocky9, 2.3.28-ubuntu22, 2.3.28-ubuntu22-arm

• Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Upgraded Apache Zookeeper to version 3.9.5 in image version 2.3.

• Upgraded Dataproc Metastore Proxy to v0.0.79 to fix CVEs.
• Fixed CVEs CVE-2026-24308 and CVE-2026-24281.

New Serverless for Apache Spark runtime versions:

• 1.2.77
• 2.2.77
• 2.3.30
• 3.0.13

New Dataproc on Compute Engine subminor image versions:

• 2.1.112-debian11, 2.1.112-rocky8, 2.1.112-ubuntu20, 2.1.112-ubuntu20-arm
• 2.2.80-debian12, 2.2.80-rocky9, 2.2.80-ubuntu22, 2.2.80-ubuntu22-arm
• 2.3.27-debian12, 2.3.27-ml-ubuntu22, 2.3.27-rocky9, 2.3.27-ubuntu22, 2.3.27-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.76
• 2.2.76
• 2.3.29
• 3.0.12

New Dataproc on Compute Engine subminor image versions:

• 2.1.111-debian11, 2.1.111-rocky8, 2.1.111-ubuntu20, 2.1.111-ubuntu20-arm
• 2.2.79-debian12, 2.2.79-rocky9, 2.2.79-ubuntu22, 2.2.79-ubuntu22-arm
• 2.3.26-debian12, 2.3.26-ml-ubuntu22, 2.3.26-rocky9, 2.3.26-ubuntu22, 2.3.26-ubuntu22-arm

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924, and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.
• Default JDK is set to Temurin JDK - 11.0.30 in all 2.1, 2.2 and 2.3 images.

New Serverless for Apache Spark runtime versions:

• 1.2.75
• 2.2.75
• 2.3.28
• 3.0.11

Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.1.110-debian11, 2.1.110-rocky8, 2.1.110-ubuntu20, 2.1.110-ubuntu20-arm
• 2.2.78-debian12, 2.2.78-rocky9, 2.2.78-ubuntu22, 2.2.78-ubuntu22-arm
• 2.3.25-debian12, 2.3.25-ml-ubuntu22, 2.3.25-rocky9, 2.3.25-ubuntu22, 2.3.25-ubuntu22-arm

Rollback Notice: These image versions were rolled back on March 11, 2026.

• Fixed CVEs CVE-2025-58057, CVE-2025-53864, CVE-2025-68161, CVE-2025-48924 (partial), and CVE-2025-33042.
• Upgraded Dataproc Metastore Proxy to v0.0.78 to fix CVEs.

New Serverless for Apache Spark runtime versions:

• 1.2.74
• 2.2.74
• 2.3.27
• 3.0.10

Serverless for Apache Spark 3.0 runtime version:

• Updated Spark version to 4.0.1.
• Updated BigQuery Spark Connector version to 0.44.0.
• Applied patch for SPARK-52601.
• Applied patch for SPARK-52614.

Serverless for Apache Spark 1.2, 2.2 and 2.3 runtime versions:

• Updated Delta Lake version to 3.2.1.

New Serverless for Apache Spark runtime versions:

• 1.2.73
• 2.2.73
• 2.3.26

New Serverless for Apache Spark runtime versions:

• 1.2.72
• 2.2.72
• 2.3.25
• 3.0.9

New Dataproc on Compute Engine subminor image versions:

• 2.0.160-debian10, 2.0.160-rocky8, 2.0.160-ubuntu18
• 2.1.109-debian11, 2.1.109-rocky8, 2.1.109-ubuntu20, 2.1.109-ubuntu20-arm
• 2.2.77-debian12, 2.2.77-rocky9, 2.2.77-ubuntu22, 2.2.77-ubuntu22-arm
• 2.3.24-debian12, 2.3.24-ml-ubuntu22, 2.3.24-rocky9, 2.3.24-ubuntu22, 2.3.24-ubuntu22-arm

Fixed an issue that could cause gsutil failures on clusters using custom images built from new base images. This problem occurred when the gcloud core/universe_domain property was not set in the build environment, resulting in a malformed /etc/boto.cfg. The image build process now ensures a correct default configuration, restoring proper Google Cloud Storage connectivity.

New Serverless for Apache Spark runtime versions:

• 1.2.71
• 2.2.71
• 2.3.24
• 3.0.8

Hyperdisks can now be added to a Dataproc cluster.

New Dataproc on Compute Engine subminor image versions:

• 2.0.159-debian10, 2.0.159-rocky8, 2.0.159-ubuntu18
• 2.1.108-debian11, 2.1.108-rocky8, 2.1.108-ubuntu20, 2.1.108-ubuntu20-arm
• 2.2.76-debian12, 2.2.76-rocky9, 2.2.76-ubuntu22, 2.2.76-ubuntu22-arm
• 2.3.23-debian12, 2.3.23-ml-ubuntu22, 2.3.23-rocky9, 2.3.23-ubuntu22, 2.3.23-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.70
• 2.2.70
• 2.3.23
• 3.0.7

New Serverless for Apache Spark runtime versions:

• 1.2.69
• 2.2.69
• 2.3.22
• 3.0.5

Serverless for Apache Spark: Added support for removing conscrypt from Serverless for Apache Spark 2.3+ runtimes using the dataproc.artifacts.remove property .

Dataproc on Compute Engine: Sharing snapshot diagnostic data: Setting the --tarball-access=GOOGLE_DATAPROC_DIAGNOSE flag with the gcloud dataproc clusters diagnose command shares all of the output Cloud Storage bucket contents with Google Cloud support if uniform bucket-level access is enabled on the output Cloud Storage bucket. If object-level access control is enabled on the output Cloud Storage bucket, only the generated diagnostic tar file is shared.

New Dataproc on Compute Engine subminor image versions:

• 2.0.158-debian10, 2.0.158-ubuntu18, 2.0.158-rocky8
• 2.1.107-debian11, 2.1.107-ubuntu20, 2.1.107-ubuntu20-arm, 2.1.107-rocky8
• 2.2.75-debian12, 2.2.75-ubuntu22, 2.2.75-ubuntu22-arm, 2.2.75-rocky9
• 2.3.22-debian12, 2.3.22-ml-ubuntu22, 2.3.22-rocky9, 2.3.22-ubuntu22, 2.3.22-ubuntu22-arm

Zookeeper version upgraded to 3.9.4 in Dataproc image 2.3.22.

Parquet CLI version upgraded to 1.15.2 in 2.1 and 2.2 images.

Delta subminor version upgraded to 3.2.1 in Dataproc on Compute Engine image 2.2 and 2.3.

Apache Pig is now available in ARM images.

Added a new dataproc:pypi.repository property to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.

Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

Upcoming Spark data lineage changes See the upcoming May, 2026 Dataproc and Serverless for Apache Spark release notes for an announcement of a change that will automatically enable Dataproc Spark data lineage and Serverless for Apache Spark data lineage when you enable the Data Lineage API (see Control lineage ingestion for a service) without requiring additional project, cluster, batch workload, or interactive session settings.

New Serverless for Apache Spark runtime versions:

• 3.0.4

Dataproc on Compute Engine: The following subminor image versions announced on January 24, 2026 have been rolled back:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

New Dataproc on Compute Engine subminor image versions:

• 2.0.157-debian10, 2.0.157-ubuntu18, 2.0.157-rocky8
• 2.1.106-debian11, 2.1.106-ubuntu20, 2.1.106-ubuntu20-arm, 2.1.106-rocky8
• 2.2.74-debian12, 2.2.74-ubuntu22, 2.2.74-ubuntu22-arm, 2.2.74-rocky9
• 2.3.21-debian12, 2.3.21-ml-ubuntu22, 2.3.21-rocky9, 2.3.21-ubuntu22, 2.3.21-ubuntu22-arm

Rollback Notice: These image versions were rolled back on January 30, 2026.

Upgraded the Delta subminor version to 3.2.1 in images 2.2 and 2.3.

Apache Pig is now available in ARM images.

Added a new dataproc:pypi.repository property to customize the PyPI repository that pip uses. The value can be a internet-accessible URL, or you can specify google to use a Google-hosted cache of PyPI, which is accessible without public internet connectivity. Starting with Managed Service for Apache Spark image version 3.1, google will be the default (you can specify the pypi property value to use public PyPI instead of the default google 3.1 value).

Removed the use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

New Serverless for Apache Spark runtime versions:

• 1.2.67
• 2.2.67
• 2.3.20
• 3.0.3

• Applied patch for SPARK-48292 in Serverless for Apache Spark 1.2 and 2.2 runtimes.

Dataproc on Compute Engine: The following subminor image versions announced on January 06, 2026 have been rolled back:

• 2.0.156-debian10, 2.0.156-ubuntu18, 2.0.156-rocky8
• 2.1.105-debian11, 2.1.105-ubuntu20, 2.1.105-ubuntu20-arm, 2.1.105-rocky8
• 2.2.73-debian12, 2.2.73-ubuntu22, 2.2.73-ubuntu22-arm, 2.2.73-rocky9
• 2.3.20-debian12, 2.3.20-ml-ubuntu22, 2.3.20-rocky9, 2.3.20-ubuntu22, 2.3.20-ubuntu22-arm

New Serverless for Apache Spark runtime versions:

• 1.2.66
• 2.2.66
• 2.3.19
• 3.0.1

New Dataproc on Compute Engine subminor image versions:

• 2.0.156-debian10, 2.0.156-ubuntu18, 2.0.156-rocky8
• 2.1.105-debian11, 2.1.105-ubuntu20, 2.1.105-ubuntu20-arm, 2.1.105-rocky8
• 2.2.73-debian12, 2.2.73-ubuntu22, 2.2.73-ubuntu22-arm, 2.2.73-rocky9
• 2.3.20-debian12, 2.3.20-ml-ubuntu22, 2.3.20-rocky9, 2.3.20-ubuntu22, 2.3.20-ubuntu22-arm

Rollback Notice: These image versions were rolled back on January 13, 2026.

Added a new property dataproc:pypi.repository to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.

Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.

Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.

New Serverless for Apache Spark runtime versions:

• 1.2.65
• 2.2.65
• 2.3.18

The following Dataproc on Compute Engine subminor image versions, released on December 05, 2025, are now blocklisted:

• 2.2.72-debian12, 2.2.72-ubuntu22, 2.2.72-ubuntu22-arm, 2.2.72-rocky9
• 2.3.19-debian12, 2.3.19-ubuntu22, 2.3.19-ubuntu22-arm, 2.3.19-ml-ubuntu22, 2.3.19-rocky9

New Dataproc on Compute Engine subminor image versions:

• 2.0.155-debian10, 2.0.155-ubuntu18, 2.0.155-rocky8
• 2.1.104-debian11, 2.1.104-ubuntu20, 2.1.104-ubuntu20-arm, 2.1.104-rocky8
• 2.2.72-debian12, 2.2.72-ubuntu22, 2.2.72-ubuntu22-arm, 2.2.72-rocky9
• 2.3.19-debian12, 2.3.19-ubuntu22, 2.3.19-ubuntu22-arm, 2.3.19-ml-ubuntu22, 2.3.19-rocky9

dataproc-ml is available by default in 2.3 -ml images.

Serverless for Apache Spark: Runtime version 3.0 is now generally available. This version simplifies onboarding, improves reliability, reduces startup latency, and adds support for Spark 4.0.

• Features and improvements:

  • Regional and multi-zonal workloads are used by default to increase obtainability of compute resources
  • Faster startup than previous runtimes
  • Fast resource cleanup that allows faster release of VPC IPs after workload completion
  • End-user credentials are used for all workloads by default
  • New bigquery Spark catalog, pre-configured for out-of-the-box BigQuery native table interactions
  • New Spark Serverless-specific IAM roles
  • New dataproc-rm.googleapis.com API enablement is required