Serverless compute release notes

This article explains the features and behaviors that are currently available and upcoming on serverless compute for notebooks and jobs.

For more information on serverless compute, see Connect to serverless compute.

Azure Databricks periodically releases updates to serverless compute, automatically upgrading the serverless compute runtime to support enhancements and upgrades to the platform. All users get the same updates, rolled out over a short period of time.

Serverless environment versions

Serverless compute for notebooks and jobs uses environment versions, which provide a stable client API based on Spark Connect to ensure application compatibility. This allows Databricks to upgrade the server independently, delivering performance improvements, security enhancements, and bug fixes without requiring any code changes to workloads.

Each environment version includes a specific Python version and a set of Python packages with defined versions. Databricks introduces new features and fixes in the latest environment version while applying security updates to all supported environment versions.

For serverless environment version release notes, see Environment versions.

Release notes

This section includes release notes for serverless compute. Release notes are organized by year and week of year. Serverless compute always runs using the most recently released version listed here.

JAR tasks on serverless compute are now generally available

August 6, 2026

Package Scala or Java code as a JAR and run it as a task in a Lakeflow Job on serverless compute. Your JAR must match the Scala, Java Development Kit (JDK), and Databricks Connect versions of your serverless environment version. See Create and run JARs on serverless compute.

July 6, 2026

July 6, 2026

This serverless compute release includes updates from Databricks Runtime 18.

New features

  • Spark Declarative Pipelines on Lakeflow streaming query ID: Spark Declarative Pipelines on Lakeflow can now set the streaming query ID on demand.

  • IP address functions (Public Preview): New SQL functions are available for working with IPv4 and IPv6 addresses and CIDR blocks, including ip_host, ip_cidr, ip_version, ip_prefix_length, ip_network, ip_network_last, ip_cidr_contains, ip_as_binary, ip_as_string, and try_* variants for null-safe behavior. See ip_host and related functions.

  • On-demand state repartitioning (Public Preview): Structured Streaming now supports changing the number of shuffle partitions for stateful queries without losing checkpoint state. See On-demand state repartitioning for stateful streaming queries.

Behavior changes

  • CREATE OR REPLACE TABLE preserves comments: CREATE OR REPLACE TABLE now preserves existing column and table comments by default. Previously, comments were dropped when recreating a table. Managed tables and views now match the existing behavior of materialized views and streaming tables.

  • DataFrame by-name writes cast compatible columns: writeTo().append(), writeTo().overwrite(), writeTo().overwritePartitions(), and write.mode("append").saveAsTable() now automatically cast type-compatible columns (for example, int to long) to match the target Delta table schema. Previously, these operations failed with a DELTA_FAILED_TO_MERGE_FIELDS error when column types were compatible but not identical. Behavior now matches SQL INSERT INTO ... BY NAME.

  • ALTER TABLE SET TBLPROPERTIES for pipelines.pipelineId: ALTER TABLE <table> SET TBLPROPERTIES('pipelines.pipelineId' = '<pipeline-id>') now attempts to make the specified table eligible for writes by the pipeline. Previously, setting this property on a regular table had no effect. If the table isn't eligible for pipeline writes, the command throws SETTING_PIPELINES_PIPELINE_ID_NOT_SUPPORTED.

  • DESCRIBE EXTENDED AS JSON includes predictive optimization results: DESCRIBE EXTENDED ... AS JSON now includes predictive optimization evaluation results in its output. Previously, this information wasn't returned in the JSON output.

  • Metric view window measures return correct results: Metric view window measures now return correct results when queries use GROUP BY, IN/BETWEEN filters, or mixed predicates on the window's order column. Previously, these filter patterns could produce incorrect results.

  • Structured Streaming deduplication with NaN keys: Structured Streaming deduplication now treats NaN (Not-a-Number) values that have different bit patterns as duplicates when a double or float column is used as a deduplication key. Previously, NaN values with different internal representations were treated as distinct and were not deduplicated.

  • NATURAL JOIN case-insensitive column matching: NATURAL JOIN now matches common columns case-insensitively, consistent with the equivalent USING join. Previously, column matching was case-sensitive, causing columns that differ only in case (for example, ID vs id) to not be recognized as common columns, resulting in a silent cross join instead of the expected equi-join.

Version 18.2

May 13, 2026

This serverless compute release roughly corresponds to Databricks Runtime 18.2.

New features

  • CREATE OR REPLACE support for temporary tables: CREATE OR REPLACE TEMP TABLE syntax is now supported, allowing you to create or replace temporary tables in a single statement. This eliminates the need to explicitly drop and recreate temporary tables.

  • agg() alias for measure() function: agg() is now available as an alias for the measure() function. This change is fully backward compatible. Existing queries that use measure() continue to work without modification, and agg() produces identical results when used with the same arguments.

  • Snowflake JDBC driver upgrade: The Snowflake JDBC driver is upgraded from 3.22.0 to 3.28.0.

  • pyspark.pipelines.testing namespace alias: pyspark.pipelines.testing is now available as a convenience alias for dlt.testing APIs. Import Spark Declarative Pipelines on Lakeflow pipeline testing utilities through either namespace.

  • Delta table history includes write option flags: Delta table history (DESCRIBE HISTORY) now includes write option flags in the operationParameters column for WRITE and REPLACE TABLE operations. When the following options are explicitly enabled, they appear as boolean flags in the history (only included when true):

    For WRITE and REPLACE TABLE operations:

    • isDynamicPartitionOverwrite: present when dynamic partition overwrite mode was used
    • canOverwriteSchema: present when schema overwrite (overwriteSchema) was enabled
    • canMergeSchema: present when schema merge (mergeSchema) was enabled

    For REPLACE TABLE operations:

    • predicate: present when replaceWhere was used
    • isV1WriterSaveAsTableOverwrite: present when the replace was triggered by a .saveAsTable overwrite

  • Selectively replace data with replaceOn and replaceUsing DataFrame APIs: The replaceOn and replaceUsing options in the Scala and Python DataFrame APIs are now generally available. Use these options to replace part of the table with the result of a DataFrame. replaceOn replaces rows that match a user-defined condition. replaceUsing replaces rows where specified columns are equal. These APIs complement the INSERT REPLACE ON and INSERT REPLACE USING SQL statements. See Selectively overwrite data with Delta Lake.

Behavior changes

  • NULL struct preservation in INSERT, MERGE, and streaming writes with schema evolution: For INSERT, MERGE, and streaming writes that use schema evolution, a NULL struct in the source is now stored as NULL in the target. Previously, that value was incorrectly materialized as a non-null struct with every field set to NULL, while the same operations without schema evolution preserved NULL structs correctly. If your code relied on receiving a non-null struct whose fields were all NULL, update your code to handle a NULL struct instead.

  • Fix for LEFT OUTER JOIN LATERAL dropping rows: A bug that incorrectly dropped rows from LEFT OUTER JOIN LATERAL queries is now fixed. Queries using this construct now return the correct results. To temporarily revert to the previous behavior, set spark.databricks.sql.optimizer.lateralJoinPreserveOuterSemantic to true.

  • NATURAL JOIN respects case-insensitive column matching: NATURAL JOIN now correctly uses case-insensitive column matching when spark.sql.caseSensitive is set to false (the default). Previously, NATURAL JOIN used case-sensitive comparison to identify common columns, causing columns that differed only in case (for example, ID versus id) to not be recognized as matching. This caused NATURAL JOIN to silently produce cross-join results. This fix aligns NATURAL JOIN behavior with USING joins, which already handled case-insensitivity correctly. Queries affected by this bug now return correct results with properly joined columns.

  • SQL UDF dependency validation in Unity Catalog: Unity Catalog now enforces dependency validation for SQL user-defined functions (UDFs) to prevent access control bypass. Previously, SQL functions created through the REST API could reference dependencies the user did not have access to. SQL UDFs with invalid dependency configurations are now blocked from execution.

  • AWS SDK v1 dependencies are shaded: AWS SDK v1 dependencies bundled with the Azure Databricks runtime are now shaded and no longer directly available on the classpath. If your code depends on AWS SDK v1 libraries previously provided by the Azure Databricks runtime, add them as explicit dependencies in your project. This change prepares for the migration to AWS SDK v2, following the end of AWS support for SDK v1.

  • Fix incorrect EPSG authority for ESRI-defined SRID 102100: The Coordinate Reference System (CRS) mapping for SRID 102100 now correctly uses ESRI:102100 instead of the incorrect EPSG:102100. This fix ensures geospatial data is stored with the correct authority for better interoperability with other systems.

Version 18.1

April 20, 2026

This serverless compute release roughly corresponds to Databricks Runtime 18.1.

New features

  • DATETIMEOFFSET data type support for Microsoft Azure Synapse: The DATETIMEOFFSET data type is supported for Microsoft Azure Synapse connections.

  • Google BigQuery table comments: Google BigQuery table descriptions are resolved and exposed as table comments.

  • JDBC connection: Use a JDBC connection to read and write to a data source with the Spark Data Source API or the Databricks Remote Query SQL API.

  • Schema evolution with INSERT statements: Use the WITH SCHEMA EVOLUTION clause with SQL INSERT statements to automatically evolve the target table's schema during insert operations. The clause is supported for INSERT INTO, INSERT OVERWRITE, and INSERT INTO ... REPLACE forms. See schema evolution.

  • Preserved NULL struct values in INSERT operations: INSERT operations with schema evolution or implicit casting preserve NULL struct values when the source and target tables have differing struct field orders.

  • Delta Sharing multi-statement transaction support: Delta Sharing tables that use pre-signed URL or cloud token sharing modes support multi-statement transactions. On first access within a transaction, the table version is pinned and reused for all subsequent reads in that transaction. Time travel, change data feed, and streaming aren't supported.

  • parse_timestamp SQL function: The parse_timestamp SQL function parses timestamp strings using multiple patterns. The function runs on the Photon engine for improved performance.

  • max_by and min_by with optional limit: The aggregate functions max_by and min_by now accept an optional third argument limit (up to 100,000), returning an array of top- or bottom-K values without window functions or CTEs.

  • Vector aggregate and scalar functions: New SQL functions operate on ARRAY<FLOAT> vectors for embedding and similarity workloads, including vector_avg, vector_sum, vector_cosine_similarity, vector_inner_product, vector_l2_distance, vector_norm, and vector_normalize. See Built-in functions.

  • SQL cursor support in compound statements: SQL scripting compound statements now support cursor processing. Use DECLARE CURSOR to define a cursor, then OPEN statement, FETCH statement, and CLOSE statement to run the query and consume rows one at a time.

  • Approximate top-k sketch functions: New functions enable building and combining approximate top-K sketches for distributed top-K aggregation: approx_top_k_accumulate, approx_top_k_combine, and approx_top_k_estimate. See approx_top_k aggregate function.

  • Tuple sketch functions: New aggregate and scalar functions for tuple sketch support distinct counting and aggregation over key-summary pairs. See Built-in functions.

  • New geospatial functions: The following geospatial functions are now available:

    • st_estimatesrid function: Estimates the best projected spatial reference identifier (SRID) for an input geometry.
    • st_force2d function: Converts a geography or geometry to its 2D representation.
    • st_nrings function: Counts the total number of rings in a polygon or multipolygon, including both exterior and interior rings.
    • st_numpoints function: Counts the number of non-empty points in a geography or geometry.

  • Photon support for geospatial functions: st_difference function, st_intersection function, and st_union function now run on the Photon engine for faster performance.

Behavior changes

  • Observation metric errors no longer fail queries: Errors during observation metric collection no longer cause query execution failures. Previously, errors in OBSERVE clauses (such as division by zero) could block or fail the entire query. Now, the query completes successfully and the error is raised when you call observation.get.

  • DESCRIBE FLOW reserved keyword: The DESCRIBE FLOW command is now available. If you have a table named flow, use DESCRIBE schema.flow, DESCRIBE TABLE flow, or DESCRIBE `flow` with backticks.

  • SpatialSQL boolean set operations: ST_Difference, ST_Intersection, and ST_Union use a new implementation with approximately 2x faster performance. Valid input geometries always produce a result. Results are normalized for consistent output and can differ after the 15th decimal place for line-segment intersections due to different formulas and order of operations.

  • Exception types for SQLSTATE: Exception types are updated to support SQLSTATE. If your code parses exceptions by string matching or catches specific exception types, update your error handling logic.

Version 18.0

February 27, 2026

This serverless compute release roughly corresponds to Databricks Runtime 18.0.

New features

  • SQL scripting is now generally available: SQL scripting is now generally available.

  • Redshift JDBC driver upgraded to 2.1.0.28: The Redshift JDBC driver has been upgraded to version 2.1.0.28.

  • Shared isolation execution environment for Unity Catalog Python UDFs: Unity Catalog Python UDFs with the same owner can now share an isolation environment by default, improving performance and reducing memory usage. To ensure a UDF always runs in a fully isolated environment, add the STRICT ISOLATION characteristic clause. See Environment isolation.

  • SQL window functions in metric views: You can now use SQL window functions in metric views to calculate running totals, rankings, and other window-based calculations.

  • Dynamic shuffle partition adjustment in stateless streaming queries: You can now change the number of shuffle partitions in stateless streaming queries without restarting the query.

  • Adaptive Query Execution and auto-optimized shuffle in stateless streaming queries: Adaptive Query Execution (AQE) and auto-optimized shuffle (AOS) are now supported in stateless streaming queries.

  • Literal string coalescing everywhere: The ability to coalesce sequential string literals such as 'Hello' ' World' into 'Hello World' has been expanded to any place string literals are allowed. See STRING type.

  • Parameter markers everywhere: You can now use named (:param) and unnamed (?) parameter markers virtually anywhere a literal value of the appropriate type can be used, including DDL statements and column types. See Parameter markers.

  • IDENTIFIER clause everywhere: The IDENTIFIER clause, which casts strings to SQL object names, has been expanded to nearly everywhere an identifier is permitted, including column aliases and column definitions. See IDENTIFIER clause.

  • New BITMAP_AND_AGG function: A new bitmap_and_agg aggregate function function is now available.

  • New Theta sketch functions: A new library of functions for approximate distinct count and set operations using Datasketches Theta Sketch is now available, including theta_sketch_agg, theta_union_agg, theta_intersection_agg, and related functions.

  • New KLL Sketch function library: A new library of functions for building KLL Sketches for approximate quantile computation is now available, including kll_sketch_agg_bigint, kll_sketch_agg_double, and related functions.

  • Apache Parquet library upgraded to 1.16.0: The Apache Parquet library has been upgraded to version 1.16.0.

  • New geospatial functions: The following new geospatial functions are now supported:

  • Improved spatial join performance: Spatial join performance is now improved by introducing shuffled spatial join support.

  • Improved performance for geospatial functions: Photon implementations are now available for st_isvalid function, st_makeline function, and st_makepolygon function.

  • EWKT input support: The try_to_geography, try_to_geometry, to_geography, and to_geometry functions now accept Extended Well-Known Text (EWKT) as input.

Behavior changes

  • FSCK REPAIR TABLE includes metadata repair by default: The FSCK REPAIR TABLE command now includes an initial metadata repair step before checking for missing data files. The command can work on tables with corrupt checkpoints or invalid partition values.

  • Proration factors aligned between reads and auto-optimized writes: Proration factors for partition sizing now use fractional values consistently across read operations and auto-optimized writes. This change might result in a different number of tasks for read operations.

  • Python UDF execution unified across PySpark and Unity Catalog: Unity Catalog Python UDFs now use Apache Arrow as the default interchange format, improving overall performance. As part of this change, TIMESTAMP values passed to Python UDFs no longer include timezone information in the datetime object's tzinfo attribute. If your UDF relies on timezone information, restore it with date = date.replace(tzinfo=timezone.utc). See Timestamp timezone behavior for inputs.

  • Improved error messages for Kafka connector login module issues: When using the Kafka connector with an unshaded login module class, Azure Databricks now provides error messages that suggest using the correct shaded class prefix (kafkashaded.org.apache.kafka or kafkashaded.software.amazon.msk.auth.iam).

  • Time travel restrictions and VACUUM retention behavior: Azure Databricks now blocks time travel queries beyond the deletedFileRetentionDuration threshold for all tables. The VACUUM command ignores the retention duration argument except when the value is 0 hours. You can't set deletedFileRetentionDuration larger than logRetentionDuration or vice versa.

  • BinaryType maps to bytes by default in PySpark: In PySpark, BinaryType now consistently maps to Python bytes. Previously, PySpark mapped BinaryType to either bytes or bytearray depending on the context. To restore the old behavior, set spark.sql.execution.pyspark.binaryAsBytes to false.

  • Partition columns materialized in Parquet files: Partitioned Delta tables now materialize partition columns in newly written Parquet data files. Previously, partition values were stored in the Delta transaction log metadata and reflected in directory paths, but not written as columns in the Parquet files themselves. This change might affect workloads that directly read Parquet files written by Delta.

  • DESCRIBE TABLE output includes metadata column: The output of DESCRIBE TABLE [EXTENDED] now includes a new metadata column for all table types, containing semantic metadata (display name, format, and synonyms) defined on the table as a JSON string.

Serverless environment version 5 is now available

February 25, 2026

You can now use serverless environment version 5 in your serverless notebooks and jobs. This includes both the CPU and GPU environment versions. See Environment version 5.

Version 17.3

October 28, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.3 LTS.

New features

  • LIMIT ALL support for recursive CTEs: You can now use the LIMIT ALL clause with recursive common table expressions (rCTEs) to explicitly specify that no row limit should be applied to the query results. See Common table expression (CTE).

  • Appending to files in Unity Catalog volumes returns correct error: Attempting to append to existing files in Unity Catalog volumes now returns a more descriptive error message to help you understand and resolve the issue.

  • st_dump function support: You can now use the st_dump function to decompose a geometry object into its constituent parts, returning a set of simpler geometries. See st_dump function.

  • Polygon interior ring functions are now supported: You can now use the following functions to work with polygon interior rings:

  • EXECUTE IMMEDIATE using constant expressions: The EXECUTE IMMEDIATE statement now supports using constant expressions in the query string, allowing for more flexible dynamic SQL execution. See EXECUTE IMMEDIATE.

  • Allow spark.sql.files.maxPartitionBytes in serverless compute: You can now configure the spark.sql.files.maxPartitionBytes Spark configuration parameter on serverless compute to control the maximum number of bytes to pack into a single partition when reading files. See Configure Spark properties for serverless notebooks and jobs.

Behavior changes

  • Support MV/ST refresh information in DESCRIBE EXTENDED AS JSON: The DESCRIBE EXTENDED AS JSON command now includes refresh information for materialized views and streaming tables, providing visibility into the last refresh time and status.

  • Add metadata column to DESCRIBE QUERY and DESCRIBE TABLE: The DESCRIBE QUERY and DESCRIBE TABLE commands now include a metadata column in their output, providing additional information about each column's properties and characteristics.

  • Correct handling of null structs when dropping NullType columns: Azure Databricks now correctly handles null struct values when dropping columns with NullType, preventing potential data corruption or unexpected behavior.

  • Improved handling of null structs in Parquet: This release includes improvements to how null struct values are handled when reading from and writing to Parquet files, ensuring more consistent and correct behavior.

  • Upgrade aws-msk-iam-auth library for Kafka: The aws-msk-iam-auth library used for Amazon MSK IAM authentication has been upgraded to the latest version, providing improved security and compatibility.

Version 17.2

September 25, 2025

This serverless compute release roughly corresponds to Databricks Runtime 17.2.

New features

  • ST_ExteriorRing function is now supported: You can now use the ST_ExteriorRing function to extract the outer boundary of a polygon and return it as a linestring. See st_exteriorring function.

  • Support TEMPORARY keyword for metric view creation: You can now use the TEMPORARY keyword when creating a metric view. Temporary metric views are visible only in the session that created them and are dropped when the session ends. See CREATE VIEW.

  • Use native I/O for LokiFileSystem.getFileStatus on S3: LokiFileSystem.getFileStatus now uses the native I/O stack for Amazon S3 traffic and returns org.apache.hadoop.fs.FileStatus objects instead of shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileStatus.

  • Auto Loader infers partition columns in singleVariantColumn mode: Auto Loader now infers partition columns from file paths when ingesting data as a semi-structured variant type using the singleVariantColumn option. Previously, partition columns were not automatically detected. See Auto Loader.