Configure compilations

This document shows you how to do the following in Dataform:

Before you begin

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Select or create a repository.

  3. Select or create a development workspace.

  4. Optional: To override the default Google Cloud project in the release configuration, grant your custom service account access in the project you plan to use.

Required roles

To get the permissions that you need to complete the tasks in this document, ask your administrator to grant you the following IAM roles:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

To enhance security for scheduling, see Implement enhanced scheduling permissions.

To be able to update the release version of a release configuration for a Dataform repository, you must grant the iam.serviceAccounts.actAs permission to the Dataform service agent for every custom service account in the workflow configurations that are using this release configuration. This permission is available in the Service Account User role (roles/iam.serviceAccountUser). For more information, see Grant the required IAM roles.

Introduction to the workflow lifecycle in Dataform

This section describes the workflow lifecycle in Dataform and ways to configure compilation and execution within Dataform.

The Dataform workflow lifecycle consists of the following phases:

Development
You develop a workflow in a Dataform workspace.
Compilation

Dataform compiles the workflow code in your workspace to SQL in real time, creating a compilation result of the workspace that you can run in BigQuery. Dataform uses the settings that you define in your workflow settings file to create the compilation result.

Dataform compilation is hermetic to ensure compilation consistency, meaning that the same code compiles to the same SQL compilation result every time. Dataform compiles your code in a sandbox environment with no internet access. No additional actions, such as calling external APIs, are available during compilation.

Execution

In a workflow invocation, Dataform runs the workspace compilation result in BigQuery.

To tailor the Dataform workflow lifecycle to your needs, you can configure the compilation result to influence where and how Dataform runs your workflow. Then, you can manually trigger or schedule executions to influence when Dataform runs your whole workflow or its selected elements.

Ways to configure Dataform compilation

By default, Dataform uses settings in the workflow settings file to create compilation results. You can override the default settings with compilation overrides to create custom compilation results. You can then manually trigger an execution of a custom compilation result, or schedule executions.

Dataform provides the following options for configuring compilation results:

Workspace compilation overrides
You can configure compilation overrides that apply to all workspaces in a repository. You can use workspace compilation overrides to create isolated development environments.
Release configurations
You can create release configurations to configure templates for creating compilation results of a Dataform repository. You can then create a workflow configuration to schedule executions of compilation results created in a selected release configuration.
Dataform API compilation overrides
You can pass Dataform API requests in the terminal to create and run a single compilation result with compilation overrides.

Configure workspace compilation overrides

With workspace compilation overrides, you can create compilation overrides for all workspaces in a Dataform repository. You can create one configuration of workspace compilation overrides for each repository.

When you manually trigger an execution in a workspace in a repository with workspace compilation overrides, Dataform applies these overrides to the compilation result of the workspace.

You can configure the following workspace compilation overrides:

  • Google Cloud project in which Dataform runs the contents of the workspace
  • Table prefix
  • Schema suffix

You can use workspace compilation overrides to create isolated development environments by isolating workspace compilation results in BigQuery with dynamic compilation overrides. Dynamic table prefix and schema suffix compilation overrides contain the ${workspaceName} variable. When you trigger an execution in a workspace, Dataform replaces the ${workspaceName} variable with the name of the current workspace, creating compilation overrides unique to the workspace.

Keep in mind that you cannot schedule executions of compilation results created with workspace compilation overrides.

Create release configurations

With release configurations, you can configure templates of settings for creating compilation results of repositories.

In a release configuration, you can configure compilation overrides of workflow settings, compilation variables, and the frequency of creating compilation results of your whole repository.

In a release configuration, you can configure the following compilation overrides:

You can create multiple release configurations in a Dataform repository, one for each stage of your development lifecycle, creating isolated repository compilation results.

You can then create workflow configurations to schedule executions of compilation results created in a selected release configuration.

You can also manually trigger the execution of a compilation result in a selected release configuration.

Configure a single compilation result with Dataform API compilation overrides

By passing Dataform API requests in the terminal, you can configure compilation overrides for a single compilation result.

In the compilationResults.create request, you can create a single compilation result of a Dataform workspace or a specified Git commitish.

In the CodeCompilationConfig object of the compilationResults.create request, you can configure compilation overrides for the compilation request.

You can configure the following Dataform API compilation overrides:

Keep in mind that Dataform API compilation overrides apply to a single compilation result and a single execution. You cannot use them to schedule Dataform executions.

You can run a compilation result in the workflowInvocations.create request.

Ways to configure Dataform execution

Dataform provides the following options for configuring execution:

Manual execution in a workspace
You can manually trigger the instant execution of a workflow in a Dataform workspace, outside of any schedule. You can run selected actions in the workflow.
Workflow configurations
You can schedule executions of compilation results created in a selected release configuration. You can select workflow actions to run, and set the frequency and time zone of executions.

Trigger instant execution in a workspace

In a Dataform workspace, you can manually trigger the instant execution of a workflow in your workspace, outside of any schedule.

You can manually run the following elements of the workflow in your workspace:

If your repository contains workspace compilation overrides, you can view what compilation overrides Dataform will apply to the workspace compilation result.

Create workflow configurations

With workflow configurations, you can schedule executions of compilation results from a selected release configuration. You can create multiple workflow configurations in a Dataform repository.

In a workflow configuration, you can configure the following execution settings:

  • Applied compilation release configuration.
  • Selection of workflow actions to be run.
  • Schedule and time zone of executions.

You can select the following workflow actions to be run:

  • All actions
  • Selected actions
  • Actions with selected tags

Then, during a scheduled execution of your workflow configuration, Dataform deploys your selection of actions from the applied compilation result to BigQuery.

Dataform release configurations and workflow configurations let you configure compilation and schedule executions within Dataform, without the need to rely on additional services.

Expiration of lifecycle resources

Dataform stores compilation results and workflow invocations for a specific period of time.

Expiration of workflow invocations

Workflow invocations expire after 90 days, or when you manually delete them.

In a workflow configuration, you can view a list of most recent workflow invocations created by the configuration. When a workflow invocation created by a workflow configuration expires, Dataform removes that workflow invocation from the list of recent invocations.

Expiration of compilation results

The expiration of compilation results depends on whether they were created in a development workspace, in a release configuration, or by a workflow invocation.

When you develop a workflow in a Dataform workspace, Dataform compiles your code into a compilation result in real time to provide query validation. Compilation results created this way expire after 24 hours.

In a release configuration, the latest compilation result becomes the live compilation result. A new compilation result replaces the current live compilation result. Dataform retains the live compilation result until it is replaced with a new compilation result. A replaced compilation result expires in up to 24 hours.

Dataform removes expired compilation results from the list of past compilation results on the Details page of a release configuration.

Dataform retains compilation results created by workflow invocations for the whole life of the workflow invocation, up to 24 hours after workflow invocation expires or is deleted.

Create workspace compilation overrides

This section shows you how to create workspace compilation overrides to isolate tables and views created from your Dataform workspaces in BigQuery. You can use workspace compilation overrides to create isolated Dataform development environments.

When you develop workflow code in a Dataform workspace, Dataform compiles the code in your workspace in real-time to create a compilation result of the workspace. Dataform uses settings defined in the workflow settings file to create the workspace compilation result. Then, when you trigger execution in a workspace, Dataform runs the workspace compilation result in BigQuery.

To override the default settings set in workflow settings for all workspaces in your repository, you can create workspace compilation overrides.

With workspace compilation overrides, you can override the following settings for all workspaces in your repository:

Project

The Google Cloud project in which Dataform runs the workspace compilation result, set in workflow_settings.yaml as defaultProject or in dataform.json as defaultDatabase.

Table prefix

The custom prefix added to all the table names in all the workspaces in the repository.

Schema suffix

The custom suffix appended to the schema of tables. Set as defaultDataset in workflow_settings.yaml, defaultSchema in dataform.json, or in the schema parameter in the config block of a table.

To create isolated development environments, you can isolate workspaces with unique compilation overrides. You can dynamically modify the table prefix and schema suffix compilation overrides with the ${workspaceName} variable.

When you manually trigger execution in a workspace, the ${workspaceName} variable injects the name of the workspace into the workspace compilation overrides.

When you set ${workspaceName} as the table prefix, Dataform adds the name of the workspace to names of all tables in the workspace. After execution, in BigQuery, you can identify which workspace a table originates from.

When you set ${workspaceName} as the schema suffix, Dataform appends the name of the workspace to defaultSchema, creating a custom schema dedicated to the workspace. After execution, in BigQuery, you can locate all the tables that are run from a specific workspace in the dedicated schema.

Example of dynamic workspace compilation overrides

The following example shows dynamic workspace compilation overrides applied to a repository that contains workspaces named after developers who work on the repository: Sasha and Kai.

The goal of workspace compilation overrides in this example is to create isolated development environments for Sasha and Kai.

The following default settings are set in workflow_settings.yaml:

  • defaultProject : analytics
  • defaultDataset: dataform

The following workspace compilation overrides create a dynamic table prefix and schema suffix for each workspace in the repository:

  • Google Cloud Project ID: analytics_dev
  • Table prefix: ${workspaceName}
  • Schema suffix: ${workspaceName}

When Sasha manually triggers execution in the Sasha workspace, Dataform runs its content with the following settings:

  • Google Cloud project: analytics_dev
  • Schema: dataform_sasha
  • Table names: sasha_name, for example, sasha_orders

When Kai manually triggers execution in the Kai workspace, Dataform runs its content with the following settings:

  • Google Cloud project: analytics_dev
  • Schema: dataform_kai
  • Table names: kai_name, for example, kai_orders

Create workspace compilation overrides

To create Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. In the Workspace compilation overrides pane, in the Google Cloud Project ID field, enter the ID of the project.
  4. In the Table prefix field, enter a prefix for all table names.
    1. Optional: To create a dynamic table prefix that is unique for each workspace, enter ${workspaceName} as the table prefix.
  5. In the Schema suffix field, enter a suffix to append to the schema of the created table or view.
    1. Optional: To create a dynamic schema suffix that is unique for each workspace, enter ${workspaceName} as the table suffix.
  6. Click Save.

Dataform applies workspace compilation overrides to all the workspaces in your repository.

Edit workspace compilation overrides

To edit Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. Edit workspace compilation overrides, and then click Save.

Delete workspace compilation overrides

To delete Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. In the Workspace compilation overrides pane, click Clear all, and then click Save.

Configure compilation overrides with the Dataform API

This section shows you how to create and run a compilation result with compilation overrides by using the Dataform API.

About Dataform API compilation overrides

To run your workflow, Dataform compiles your code to SQL to create a compilation result. Then, during a workflow invocation, Dataform runs the compilation result in BigQuery.

By default, Dataform uses the settings in the workflow settings file to create the compilation result. To isolate data run at different stages of your development lifecycle, you can override the default settings with compilation overrides.

By passing Dataform API requests in the terminal, you can create and run a single compilation result with compilation overrides. You can create a compilation result of a workspace or of a selected Git commitish.

To create a compilation result with compilation overrides, you need to raise the Dataform API compilationResults.create request. In the request, you need to specify a source, which should be a workspace or Git commitish, for Dataform to compile into the compilation result. In the CodeCompilationConfig object of the compilationResults.create request, you can configure compilation overrides.

You can then run the created compilation result in a Dataform API workflowInvocations.create request.

You can configure the following Dataform API compilation overrides:

Google Cloud project

The project in which Dataform runs the compilation result, set in the workflow_settings.yaml file as the defaultProject property or in the dataform.json file as the defaultDatabase property.

Table prefix

The custom prefix added to all table names in the compilation result.

Schema suffix

The custom suffix appended to the schema of tables defined in the defaultDataset property in the workflow_settings.yaml file, the defaultSchema property in the dataform.json file, or in the schema parameter in the config block of a table.