BigQuery release notes archive

BigQuery is available in the Mexico (northamerica-south1) region.

The Sovereign Controls for EU control package now supports BigQuery Data Transfer Service. For more information, see Supported products by control package. This feature is generally available (GA).

You can now manage data canvases, data preparations, notebooks, saved queries, and workflows in Dataplex. Metadata of data canvases, data preparations, notebooks, saved queries, and workflows is automatically available in Dataplex, without additional configuration. This feature is generally available (GA).

You can now search for and view the metadata of data canvases, data preparations, notebooks, saved queries, and workflows in the Dataplex console. This feature is in preview.

You can now use the Google Cloud Code extension for VS Code to work with BigQuery datasets and notebooks in your VS Code environment. This feature is in preview.

Regional endpoints, which help you run your workloads in compliance with data residency and data sovereignty requirements, are now generally available (GA). With regional endpoints, your request traffic is routed directly to the region specified in the endpoint. For more information, see BigQuery regional endpoints.

You can now discover, procure, and commercialize your Analytics Hub listings on Google Cloud Marketplace to share data offerings at scale. This feature is in preview.

You can now create remote models in BigQuery ML based on the gemini-2.0-flash-exp model in Vertex AI. To create remote models, you can use either SQL or BigQuery DataFrames.

You can use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. You can also use the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try this feature by using either the Generate text by using the ML.GENERATE_TEXT function how-to topic, or the BigFrames Gemini 2.0 Text Generation Simple Example notebook.

This feature is in preview.

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now generally available (GA).

BigQuery Managed Disaster Recovery provides managed failover and redundant compute capacity for business-critical workloads. It is intended for use in the case of a total region outage and is supported with the BigQuery Enterprise Plus edition only. This feature is now generally available (GA).

You can create a search index on columns containing INT64 or TIMESTAMP data and BigQuery can optimize predicates that use those columns. This feature is generally available (GA).

The following BigQuery ML features are now available:

• Creating remote models based on the Vertex AI gemini-1.5-flash and gemini-1.5-pro models.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

These features are now generally available (GA).

You can try Gemini in BigQuery at no charge until January 27, 2025. After that date, to continue to use Gemini in BigQuery you must do one of the following:

• Purchase and assign BigQuery Enterprise Plus edition reservations to projects that use Gemini in BigQuery.
• Purchase Gemini Code Assist Enterprise.

The following BigQuery ML features are now available:

• You can perform supervised tuning on a remote model based on a Vertex AI Gemini 1.5 flash or Gemini 1.5 pro model.
• You can evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained PaLM and Gemini models and tuned Gemini models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic or the Use tuning and evaluation to improve model performance tutorial.

These BigQuery ML features are generally available (GA).

BigQuery now offers the following Gemini-enhanced SQL translation features:

• In interactive translation mode, you can use Gemini-enhanced SQL translations to customize translated GoogleSQL queries. This feature is generally available (GA).

• You can generate AI suggestions for batch translations using the Gemini model. The suggestions are based on a Gemini-based configuration YAML file. This feature is in Preview.

• After running an interactive SQL translation, you can request a Gemini-generated text explanation that includes a summary of the translated SQL query. This feature is in Preview.

The BigQuery Data Transfer Service data source change log provides details about upcoming changes to data source schemas and schema mappings.

Dataplex automatic discovery lets you scan data in Cloud Storage buckets to extract and catalog metadata. Automatic discovery creates BigLake or external tables and object tables you can use for analytics and AI, and catalogs that data in Dataplex Catalog. This feature is available in public preview.

BigQuery provides context-aware transformation recommendations from Gemini for cleansing data for analysis. Data preparation is available in Preview.

You can now view, trigger, and pause Airflow DAGs in BigQuery. This feature is in Preview.

You can now manage notebook schedules on the Orchestration page. Notebook scheduling is in Preview.

Custom organization policies let you allow or deny specific operations on BigQuery Data Transfer Service transfer configurations to meet your organization's compliance and security requirements. This feature is generally available (GA).

You can now use fine-grained DML to optimize the execution of UPDATE, DELETE, and MERGE statements on tables. This feature is in Preview.

Use the BigQuery migration assessment for Oracle to assess the complexity of migrating data from your Oracle data warehouse to BigQuery. This feature is in preview.

You can now export and load Parquet files that include GeoParquet metadata. This feature is generally available (GA).

BigQuery tables for Apache Iceberg bring the convenience of BigQuery storage optimization to Apache Iceberg tables that reside in your own cloud buckets. BigQuery tables for Apache Iceberg let you use BigQuery without moving data out of buckets that you control. This feature is now in preview.

You can now use pipe syntax anywhere you write GoogleSQL. Pipe syntax supports a linear query structure designed to make your queries easier to read, write, and maintain. This feature is in Preview.

You can now create an external dataset in BigQuery that links to an existing database in Spanner. This feature is in preview.

ODBC driver update, release 3.0.7 1016

• [New] Connector authentication on Google Cloud VMs: The connector now supports authentication through Application Default Credentials using the Google internal metadata server, eliminating the need for a keyfile. This feature works only on Google Cloud Compute Engine VMs.
• [Resolved] The output for PrimaryKeys previously denoted the Key Sequence as a 0-indexed value. This has been corrected to a 1-indexed value, indicating the sequential order of the primary key's column within the primary key itself.

You can now enable, disable, and analyze history-based optimizations for queries. This feature is generally available (GA).

You can now use the operational health dashboard to get a single-pane view of key metrics such as slot usage, shuffle usage, errors, and total storage in real time. This feature is generally available (GA).

You can now use flexible column names with BigQuery tables and views for extracting, loading, streaming, and querying data. This feature is generally available (GA).

You can now create a materialized view replica directly from the Google Cloud console. This feature is generally available (GA).

Cloud console updates: You can now use keyboard shortcuts to control tab navigation in the details pane. This feature is generally available (GA).

You can now use Cloud KMS Autokey to automate the creation and use of customer-managed encryption keys (CMEKs), including the Cloud HSM service. This feature is generally available (GA).

BigQuery ML now offers the following AI features:

• You can process documents from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Document AI API, including specifying a document processor to use.
  2. Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.

  Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.

• You can transcribe audio files from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  2. Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.

  Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

These BigQuery ML feature are generally available (GA).

BigQuery ML now offers the following expanded embedding support features:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text, image, and video into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try these capabilities with the following tutorials:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate video embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

These features are generally available (GA).

You can now create workflows to execute code assets in sequence at a scheduled time. This feature is in Preview.

You can perform model monitoring in BigQuery ML. The following model monitoring functions are now generally available (GA):

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow tfdv.validate_statistics API.

You can now batch migrate classic saved queries to saved queries. This feature is in Preview for projects that have fewer than 2500 classic saved queries.

You can now use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. You can use a contribution analysis model with the ML.GET_INSIGHTS function to generate insights about changes to key metrics in your multi-dimensional data.

Try this feature with the Get data insights from a contribution analysis model tutorial.

This feature is in preview.

You can store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is Generally Available.

You can now use the partial ordering mode in BigQuery DataFrames to generate more efficient queries. This feature is in Preview.

You can now use Terraform to manage IAM tags on datasets and tables. This feature is generally available (GA).

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery, including Creative, Partner, and Advertiser tables. This feature is generally available (GA).

You can now use vector search and vector index features in BigQuery.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

You can try the vector search and vector index capabilities by using the Search embeddings with vector search tutorial.

The BigQuery vector search and vector index features are generally available (GA).

The BigQuery Data Transfer Service now supports incremental transfers when you migrate your data from your Teradata data warehouses to BigQuery. This feature is generally available (GA).

Delta Lake BigLake tables are now generally available (GA). Delta Lake is an open source, tabular data storage format that supports petabyte scale data tables.

Phrase support for the SEARCH function is now generally available (GA).

The following Gemini in BigQuery features are now generally available (GA):

• Data insights
• Data canvas
• SQL and Python code assistance features:
  • Use the SQL generation tool
  • Prompt to generate SQL queries
  • Explain a SQL query
  • Generate Python code
• Partitioning and clustering recommendations

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

You can now use the GROUP BY clause and the SELECT DISTINCT clause with the ARRAY and STRUCT data types. This feature is in Preview.

You can now query data in AlloyDB using a federated query. This feature is now generally available (GA).

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable (reverse ETL). This feature is generally available (GA).

You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

This feature is in preview.

Python code completion is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

You can now perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models. This feature lets you detect anomalies in historical time series data or in new data with multiple feature columns. You can try this feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial. This feature is generally available (GA).