Use the Bigtable Spark connector

The Bigtable Spark connector lets you read and write data to and from Bigtable. You can read data from your Spark application using Spark SQL and DataFrames. The following Bigtable operations are supported using the Bigtable Spark connector:

  • Write data
  • Read data
  • Create a new table

This document shows you how to convert a Spark SQL DataFrames table to a Bigtable table, and then compile and create a JAR file to submit a Spark job.

Spark and Scala support status

The Bigtable Spark connector supports the following Scala versions:

The Bigtable Spark connector supports the following Spark versions:

The Bigtable Spark connector supports the following Managed Service for Apache Spark versions:

For more information, see Managed Service for Apache Spark cluster image version lists.

Calculate costs

If you decide to use any of the following billable components of Google Cloud, you are billed for the resources that you use:

  • Bigtable (You are not charged for using the Bigtable emulator)
  • Managed Service for Apache Spark
  • Cloud Storage

Managed Service for Apache Spark pricing applies to the use of Managed Service for Apache Spark on Compute Engine clusters. Managed Service for Apache Spark Serverless pricing applies to workloads and sessions run on Managed Service for Apache Spark Serverless for Spark.

To generate a cost estimate based on your projected usage, use the pricing calculator.

Before you begin

Complete the following prerequisites before using the Bigtable Spark connector.

Required roles

To get the permissions that you need to use Bigtable Spark connector, ask your administrator to grant you the following IAM roles on your project:

  • Bigtable Administrator (roles/bigtable.admin)(Optional): lets you read or write data and create a new table.
  • Bigtable User (roles/bigtable.user): lets you read or write data, but doesn't let you create a new table.

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

If you are using Managed Service for Apache Spark or Cloud Storage, additional permissions might be required. For more information, see Managed Service for Apache Spark permissions and Cloud Storage permissions.

Set up Spark

Apart from creating a Bigtable instance, you also need to set up your Spark instance. You can do so locally or select either of these options to use Spark with Managed Service for Apache Spark:

  • Managed Service for Apache Spark cluster
  • Managed Service for Apache Spark Serverless

For more information about choosing between a Managed Service for Apache Spark cluster or serverless option, see the Managed Service for Apache Spark Serverless for Spark compared to Managed Service for Apache Spark on Compute Engine documentation.

Download the connector JAR file

You can find the Bigtable Spark connector source code with examples in the Bigtable Spark connector GitHub repository.

Based on your Spark setup, you can access the JAR file as follows:

  • If you are running PySpark locally, you should download the connector's JAR file from the gs://spark-lib/bigtable/spark-bigtable_SCALA_VERSION-CONNECTOR_VERSION.jar Cloud Storage location.

    Replace SCALA_VERSION with 2.12 or 2.13, which are the only supported Scala versions, and replace the CONNECTOR_VERSION with the connector version that you want to use.

  • For Managed Service for Apache Spark cluster or serverless option, use the latest JAR file as an artifact that can be added in your Scala or Java Spark applications. For more information about using the JAR file as an artifact, see Manage dependencies.

  • If you are submitting your PySpark job to Managed Service for Apache Spark, use the gcloud dataproc jobs submit pyspark --jars flag to set the URI to the JAR file location in Cloud Storage—for example gs://spark-lib/bigtable/spark-bigtable_SCALA_VERSION-CONNECTOR_VERSION.jar.

Determine compute type

For high-throughput read jobs, such as those executed by Spark applications, you can use Data Boost serverless compute to avoid impacting your application-serving clusters. Your Spark application must use version 1.1.0 or later of the Spark connector to use Data Boost.

To use Data Boost, you must create a Data Boost app profile and then provide the app profile ID for the spark.bigtable.app_profile.id Spark option when you add your Bigtable configuration to your Spark application. If you've already created an app profile for your Spark read jobs and you want to continue using it without changing your application code, you can convert the app profile to a Data Boost app profile. For more information, see Convert an app profile.

For more information, see the Bigtable Data Boost overview.