Control access with IAM

This document shows you how to do the following in Dataform:

• Grant Dataform required access.
• Control access to Dataform with IAM.
• Control access to individual tables with IAM.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

Enable the APIs

Grant Dataform the required access

This section shows you how to grant the Identity and Access Management (IAM) roles that Dataform service agents and custom service accounts require to run workflows in BigQuery.

About custom service accounts and Dataform service agents

You can configure custom service accounts to run workflows on your behalf in the following ways:

• At the repository level, to run all the workflows in a given repository.
• Individually for each workflow configuration.

When you create a Dataform repository or workflow configuration, you can select any service account that you have act-as permissions on. You must configure the required act-as permissions for all the service accounts associated with your Dataform resources.

When you create your first Dataform repository, Dataform automatically generates a service agent. Dataform uses the service agent to interact with BigQuery on your behalf.

Your Dataform service agent ID is in the following format:

service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com

Replace PROJECT_NUMBER with the numeral ID of your Google Cloud project. You can find your Google Cloud project ID in the Google Cloud console dashboard. For more information, see Find the project name, number, and ID.

Required roles for Dataform service agents, custom service accounts, and Google Accounts

Dataform service agents, custom service accounts, and user credentials for a Google Account (Preview) used to authenticate in Dataform require the following BigQuery IAM roles to be able to run workflows in BigQuery:

• BigQuery Data Editor (roles/bigquery.dataEditor) on projects to which Dataform needs both read and write access. These usually include the project hosting your Dataform repository.
• BigQuery Data Viewer (roles/bigquery.dataViewer) on projects to which Dataform needs read-only access.
• BigQuery Job User (roles/bigquery.jobUser) on the project hosting your Dataform repository.
• BigQuery Data Owner (roles/bigquery.dataOwner) if you want to query BigQuery datasets.
• BigQuery roles for column-level access control if you want to use BigQuery policy tags.

Additionally, grant the following roles to the Dataform service agent on the effective service account for the workflow configuration. These roles are required for strict act-as mode to work.

• Service Account User (roles/iam.serviceAccountUser)
• Service Account Token Creator (roles/iam.serviceAccountTokenCreator)

For automatic repository releases and automatic workflow runs, grant the Dataform service agent the iam.serviceAccounts.actAs permission on the effective service account.

To enable metadata enrichment in Knowledge Catalog (Preview) and data quality scorecard integration (Preview), grant the Dataplex Catalog Editor (roles/dataplex.catalogEditor) role on the project or @bigquery entry group to the custom service accounts or Google Accounts used for workflow execution.

Security considerations

Granting the roles required by Dataform to a Dataform service agent, custom service account, or a user's Google Account (Preview) comes with the following security considerations:

• Any custom service account granted the required roles might gain access to BigQuery, Secret Manager, or Knowledge Catalog in the project that the service account belongs to, regardless of VPC Service Controls. When Dataform uses a custom service account to execute workflows, the Dataform requests are treated as originating from within the VPC Service Controls perimeter of the project that contains your repository. Consequently, VPC Service Controls doesn't block communication between Dataform and other services, such as BigQuery, Secret Manager, or Knowledge Catalog, if those resources are located within the same service perimeter.

  For more details on how to manage these security boundaries, see Configure VPC Service Controls.

• Any user who has the dataform.repositories.create IAM permission can run code using the Dataform service agent and all the permissions granted to that service agent or service account.

  For more information, see Security considerations for Dataform permissions.

• To maintain a secure permissions model, regularly audit Dataform service agent role bindings. For detailed monitoring instructions, see Monitor service agent permissions with Security Command Center.

To restrict the data that a user, service agent, or service account can read or write in BigQuery, you can grant granular BigQuery IAM permissions to selected BigQuery datasets or tables. For more information, see Controlling access to datasets and Controlling access to tables and views.

To prevent users from performing actions while using the user credentials for the Google Account of another user, the following restrictions are enforced:

• To modify a workflow configuration with the user credentials of another Google Account attached to it, you need to attach the user credentials for your own Google Account to the workflow configuration or change the workflow configuration to authenticate with a custom service account.
• You can't modify a compilation result for a release configuration if there are workflow configurations referencing the release configuration that have another Google Account user's credentials attached.

• You can't set a workflow configuration to authenticate with Google Account user credentials and reference a release configuration with a schedule. This limitation has the following consequences:

  • You can't update a release configuration to use a schedule if there are workflow configurations referencing the release configuration that are set to authenticate with Google Account user credentials.
  • You can't create a workflow configuration that authenticates with Google Account user credentials and points to a release configuration with a schedule.
  • You can't create or update a workflow configuration to use Google Account user credentials and point to a release configuration with a schedule.

Grant the required BigQuery roles

To grant the required BigQuery IAM roles to your Dataform service agent, a custom service account that you want to use in Dataform, or a user's Google Account that you want to use to authenticate in Dataform (Preview), follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select or create a repository.

3. In the Google Cloud console, go to the IAM page.

   Go to IAM

4. Click Grant Access.

5. In the New principals field, enter the service agent ID, service account ID, or the user's Google Account email (Preview).

6. In the Select a role list, select the BigQuery Job User role.

7. Click Add another role, and then in the Select a role list, select the BigQuery Data Editor role.

8. Click Add another role, and then in the Select a role list, select the BigQuery Data Viewer role.

9. Click Save.

Grant roles required for automatic workflows

To use a custom service account in Dataform, the Dataform service agent must be able to access the custom service account. This lets Dataform run your workflows using the permissions defined on your custom service account instead of on the service agent's account.

To grant this access, you need to grant the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) to the Dataform service agent as the principal. This lets the Dataform service agent impersonate the service account by creating short-lived credentials known as tokens. These tokens are required for Dataform to run workflows using the custom service account's identity.

You also need to grant the Service Account User role (roles/iam.serviceAccountUser) to the Dataform service agent. This lets the Dataform service agent start new automatic workflow runs for workflow configurations that are run by the custom service account.

To grant the Dataform service agent access to a custom service account, follow these steps:

1. In the Google Cloud console, go to IAM > Service accounts.

   Go to Service accounts

2. Select the Google Cloud project where your custom service account is located.

3. On the Service accounts for project "PROJECT_NAME" page, select your custom service account.

4. Go to Principals with access, and then click Grant Access.

5. In the New principals field, enter your Dataform service agent ID.

   Your Dataform service agent ID is in the following format:

   service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com
   

6. In the Select a role list, select the Service Account Token Creator role and the Service Account User role.

7. Click Save.

The custom service account is now ready to be configured within your Dataform repository.

Audit service account configurations

This section shows you how to audit your Dataform resources to ensure proper service account usage and permission grants. Auditing is especially important when using custom service accounts, as they require specific permissions for the Dataform service agent to operate.

When using a custom service account for a Dataform repository or workflow configuration, you must verify that the Dataform service agent has the Service Account User role (roles/iam.serviceAccountUser) on the custom service account. This role grants the iam.serviceAccounts.actAs permission, which lets scheduled runs, initiated by the Dataform service agent, impersonate the custom service account. Additionally, verify that the Dataform service agent has the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) on the effective service account.

Verify repository service accounts

First, identify the dataform.Repository assets that are in scope for Dataform's own scheduling and execution. Then, verify the service account configurations for those in-scope repositories.

1. Use Cloud Asset Inventory to list all the resources of the dataform.Repository type. For more information, see View your assets.

2. For each repository in the Cloud Asset Inventory output, check the resource.data.labels field to determine if it's in scope. The exact path might vary slightly based on your export format.

3. Identify out-of-scope repositories by inspecting the labels map for the single-file-asset-type key. The presence of this key indicates that the repository is used by a BigQuery feature. If the value is sql or data_canvas, the repository can be excluded from the service account permission checks.

   The remaining repositories lacking this key or these values are in scope for the service account permission checks.

4. For each in-scope repository, check the resource.data.serviceAccount field in the Cloud Asset Inventory output to determine if a custom service account is configured:

   • If the resource.data.serviceAccount field is present and its value is different from the project's Dataform service agent email address, then the repository uses a custom service account.

   • If the resource.data.serviceAccount field is absent, or if the field's value matches the project's Dataform service agent, then the repository uses the service agent.

5. If a custom service account is used, verify that the Dataform service agent has both the Service Account User role (roles/iam.serviceAccountUser) and the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) on that custom service account.

Verify workflow configuration service accounts

Using dedicated custom service accounts for Dataform workflow configurations is a security best practice, aligning with the principle of least privilege.

To verify service account usage for dataform.WorkflowConfig resources, do the following:

1. Use Cloud Asset Inventory to list all resources of the dataform.WorkflowConfig type.

2. For each workflow configuration, examine the Cloud Asset Inventory output to determine the effective service account:

   • If the resource.data.serviceAccount field is present, this value is the email address of the service account explicitly set on the workflow configuration.
   • If the resource.data.serviceAccount field is absent, the workflow configuration inherits the service account from its parent repository. Check the parent repository's configuration to find the effective service account.

3. Identify if a custom service account is being used by comparing the email address of the effective service account with the email address of the project's Dataform service agent. If they are different, a custom service account is in use.

4. If a custom service account is in use, ensure that the Dataform service agent has both the Service Account User role (roles/iam.serviceAccountUser) and the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) granted on that custom service account. These permissions let the Dataform service agent initiate workflow executions impersonating the custom service account.

Monitor service agent permissions with Security Command Center

To ensure ongoing adherence to the principle of least privilege, actively monitor your project in Security Command Center for findings related to over-privileged service accounts and service agents.

For instructions on how to review and apply remediation findings in Security Command Center, see IAM recommender findings and Investigating identity findings in the console.

When you remediate a SERVICE_AGENT_GRANTED_BASIC_ROLE finding for a Dataform service agent, replace the overly permissive role with a predefined role that has the minimal required permissions, scoped to the specific resource. For example, grant the Service Account User role (roles/iam.serviceAccountUser) on a specific service account.

Control access to Dataform with IAM

This section describes the access control options for Dataform and shows you how to view and grant Dataform roles. Dataform uses Identity and Access Management (IAM) for access control. For more information about roles and permissions in IAM, see IAM roles and permissions index.

Predefined Dataform roles

The following table lists the predefined roles that give you access to Dataform resources: