Schedule runs

This document shows you how to do the following in Dataform:

The following table compares the methods:

Method Goal Trigger Type
Workflow configurations Standard, time-based production runs within Dataform. Time-based
Workflows and Cloud Scheduler Lightweight, serverless orchestration of Dataform alongside other APIs. Time-based or event-driven
Managed Airflow Complex pipelines with dependencies outside of BigQuery. Time-based or event-driven
Cloud Build triggers Automating releases immediately after a Git push. Event-driven

Before you begin

To schedule runs with workflow configurations or schedule runs with workflows and Cloud Scheduler, do the following:

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Select or create a repository.

  3. Create a release configuration.

To schedule runs with Managed Service for Apache Airflow, do the following:

  1. Select or create a Dataform repository.
  2. Grant Dataform access to BigQuery.
  3. Select or create a Dataform workspace.
  4. Create at least one table.
  5. Create a Managed Airflow 2 environment.

Required roles

To get the permissions that you need to complete the tasks in this document, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

To use a custom service account when creating a workflow configuration, grant access to the custom service account.

To use user credentials for a Google Account when creating a workflow configuration (Preview), grant access to the Google Account.

To enable scheduled executions for a workflow configuration, you must grant the iam.serviceAccounts.actAs permission to the default Dataform service agent for the custom service account used in the workflow configuration. This permission is available in the Service Account User role (roles/iam.serviceAccountUser). For more information, see Use strict act-as mode.

To enhance security for scheduling, see Implement enhanced scheduling permissions.

Schedule runs with workflow configurations

This section shows you how to create a workflow configuration in Dataform to schedule and configure workflow runs. You can use workflow configurations to execute Dataform workflows on a schedule.

About workflow configurations

To schedule Dataform runs of all or selected workflow actions in BigQuery, you can create workflow configurations. In a workflow configuration, you select a compilation release configuration, select workflow actions for execution, and set the run schedule.

Then, during a scheduled run of your workflow configuration, Dataform deploys your selection of actions from the latest compilation result in your release configuration to BigQuery. You can also manually trigger the run of a workflow configuration with the Dataform API workflowConfigs.

A Dataform workflow configuration contains the following run settings:

  • ID of the workflow configuration.
  • Release configuration.

  • Service account.

    This is the custom service account that's associated with the workflow configuration. You can select a custom service account that's associated with your Google Cloud project, or you can manually enter a different service account. By default, workflow configurations use the same service accounts as their repositories.

    Service account credentials are the default authorization method for scheduled workflow configuration creation and executions.

  • Google Account user credentials (Preview)

    Google Account user credentials are the default authorization method for manual, non-scheduled workflow configuration creation and executions. For more information, see Authorize your Google Account.

  • Workflow actions to be executed:

    • All actions.
    • Selection of actions.
    • Selection of tags.

  • Run schedule and time zone.

Create a workflow configuration

To create a Dataform workflow configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. In the Workflow configurations section, click Create.

  3. In the Create workflow configuration pane, in the Configuration ID field, enter a unique ID for the workflow configuration.

    IDs can only include numbers, letters, hyphens, and underscores.

  4. In the Release configuration menu, select a compilation release configuration.

  5. In the Authentication section, authorize the workflow configuration with your Google Account user credentials or a service account.

    • To use your Google Account user credentials (Preview), select Execute with my user credentials.

      Optional: In the Extended access options section, select the additional services that your workflow requires:

      • Knowledge Catalog: Allows Google Cloud Knowledge Catalog metadata updates.
      • Google Drive: Allows read-only access to Google Drive files.
      • Bigtable: Allows read-only access to Google Bigtable data.

    • To use a custom service account, select Execute with selected service account, and then select the service account associated with your Google Cloud project that you have access to. If you don't select a service account, the workflow configuration uses the service account of the repository.

  6. Optional: In the Schedule frequency field, enter the frequency of runs in the unix-cron format.

    To verify that Dataform executes the latest compilation result in the corresponding release configuration, keep a minimum one-hour break between the time of compilation result creation and the time of scheduled execution.

  7. Optional: In the Timezone menu, select the time zone for runs.

    The default time zone is UTC.

  8. Select the workflow actions to be executed:

    • To execute the entire workflow, click All actions.
    • To execute selected actions in the workflow, click Selection of actions, and then select actions.
    • To execute actions with selected tags, click Selection of tags, and then select tags.
    • Optional: To execute selected actions or tags and their dependencies, select the Include dependencies option.
    • Optional: To execute selected actions or tags and their dependents, select the Include dependents option.

    • Optional: To rebuild all tables from scratch, select the Run with full refresh option.

      Without this option, Dataform updates incremental tables without rebuilding them from scratch.

    • Optional: Set the BigQuery query job priority with the Execute as interactive job with high priority (default) option. By default, BigQuery runs queries as interactive query jobs, which are intended to start running as quickly as possible. Clearing this option runs the queries as batch query jobs, which have lower priority.

  9. Click Create. If you selected Execute with my user credentials for your authentication method, you must authorize your Google Account (Preview).

For example, the following workflow configuration executes actions with the hourly tag every hour in the CEST timezone:

  • Configuration ID: production-hourly
  • Release configuration: -
  • Frequency: 0 * * * *
  • Timezone: Central European Summer Time (CEST)
  • Selection of workflow actions: selection of tags, hourly tag

Authorize your Google Account

To authenticate the resource with your Google Account user credentials, you must manually grant permission for BigQuery pipelines to get the access token for your Google Account and access the source data on your behalf. You can grant manual approval with the OAuth dialog interface. If you select any Extended access options, you must grant access to those services—for example, Google Drive or Knowledge Catalog.

You only need to give permission to BigQuery pipelines once.

To revoke the permission that you granted, follow these steps:

  1. Go to your Google Account page.
  2. Click BigQuery Pipelines.
  3. Click Remove access.

Changing the workflow configuration owner by updating credentials also requires manual approval if the new Google Account owner has never created a workflow configuration before.

Edit a workflow configuration

To edit a workflow configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. By the workflow configuration that you want to edit, click the More menu, and then click Edit.
  3. In the Edit workflow configuration pane, edit the workflow configuration settings, and then click Save.

Delete a workflow configuration

To delete a workflow configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. By the workflow configuration that you want to delete, click the More menu, and then click Delete.
  3. In the Delete release configuration dialog, click Delete.

Schedule runs with Workflows and Cloud Scheduler

This section shows you how to schedule runs of Dataform workflows using Workflows and Cloud Scheduler.

About scheduled workflow runs

You can set the frequency of your Dataform workflow runs by creating a Cloud Scheduler job that triggers a Workflows workflow. Workflows executes services in an orchestration workflow that you define.

Workflows executes your Dataform workflow in a two-step process. First, it pulls your Dataform repository code from your Git provider and compiles it into a compilation result. Then, it uses the compilation result to create a Dataform workflow and executes it at the frequency that you set.

Create a scheduled orchestration workflow

To schedule runs of your Dataform workflow, use Workflows to create an orchestration workflow and add a Cloud Scheduler job as a trigger.

  1. Workflows uses service accounts to give workflows access to Google Cloud resources. Create a service account and grant it the following permissions:

  2. Create an orchestration workflow and use the following YAML source code as your workflow definition:

    main:
    steps:
    - init:
        assign:
        - repository: projects/PROJECT_ID/locations/REPOSITORY_LOCATION/repositories/REPOSITORY_ID
    - createCompilationResult:
        call: http.post
        args:
            url: ${"https://dataform.googleapis.com/v1/" + repository + "/compilationResults"}
            auth:
                type: OAuth2
            body:
                gitCommitish: GIT_COMMITISH
        result: compilationResult
    - createWorkflowInvocation:
        call: http.post
        args:
            url: ${"https://dataform.googleapis.com/v1/" + repository + "/workflowInvocations"}
            auth:
                type: