Schedule production runs

This quickstart walks you through the following steps to schedule production runs in Dataform:

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the BigQuery and Dataform APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  6. Verify that billing is enabled for your Google Cloud project.

  7. Enable the BigQuery and Dataform APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

    8.

Additionally, select or create a custom service account to run workflows in BigQuery.

Required roles

To get the permissions that you need to perform all tasks in this quickstart, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Grant required roles

To run workflows in BigQuery, you can use a custom service account or your Google Account (Preview). However, custom service account credentials are the default option for scheduled runs. Using Google Account user account credentials is discouraged for scheduled runs.

Your custom service account must have the following required roles:

  • BigQuery Data Editor (roles/bigquery.dataEditor) on projects or specific BigQuery datasets to which Dataform needs both read and write access. This usually includes the project hosting your Dataform repository.
  • BigQuery Data Viewer (roles/bigquery.dataViewer) on projects or specific BigQuery datasets to which Dataform needs read-only access.
  • BigQuery Job User (roles/bigquery.jobUser) on the project hosting your Dataform repository.

To let Dataform use your custom service account, the default Dataform service agent must have the following roles on the custom service account resource:

To grant these roles, follow these steps:

  1. In the Google Cloud console, go to the IAM page.

    Go to IAM

  2. Click Grant access.

  3. In the New principals field, enter your custom service account ID.

  4. In the Select a role menu, select the following roles one by one, using Add another role for each additional role:

    • BigQuery Data Editor
    • BigQuery Data Viewer
    • BigQuery Job User

  5. Click Save.

  6. In the Google Cloud console, go to the Service accounts page.

    Go to Service accounts

  7. Select your custom service account.

  8. Go to Principals with access, and then click Grant access.

  9. In the New principals field, enter your default Dataform service agent ID.

    Your default Dataform service agent ID is in the following format:

    service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

    Replace PROJECT_NUMBER with the numeral ID of your Google Cloud project. You can find your Google Cloud project ID in the Google Cloud console dashboard. For more information, see Find the project name, number, and ID.

  10. In the Select a role list, add the following roles:

    • Service Account User
    • Service Account Token Creator

  11. Click Save.

For more information on granting roles, see Grant Dataform the required access.

Create a Dataform repository

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Click Create repository.

  3. On the Create repository page, do the following:

    1. In the Repository ID field, enter quickstart-production.

    2. In the Region list, select europe-west4.

    3. In the Service account list, select a custom service account for the repository.

    4. In the actAs permission checks section, enforce the permission checks on user actions on the repository.

    5. Click Create.

  4. Click Go to repositories.

Create a release configuration and workflow configuration

To create production compilation results of the quickstart-production repository and schedule a run of production tables, follow these steps: