Use the open-source Dataform CLI

With the open-source Dataform CLI, you can initialize, compile, test, and run Dataform core locally, outside of Google Cloud.

The Dataform CLI supports Application Default Credentials (ADC). With ADC, you can make credentials available to your application in a variety of environments, such as local development or production, without needing to modify your application code. To use ADC, you must first provide your credentials to ADC.

Before you begin

Before installing the Dataform CLI, install NPM.

Install Dataform CLI

• To install Dataform CLI, run the following command:

  npm i -g @dataform/cli
  

Initialize a Dataform project

• To initialize a new Dataform project, run the following command inside your project directory:

  dataform init . PROJECT_NAME DEFAULT_LOCATION
  

  Replace the following:

  • PROJECT_NAME: the name of your project.

  • DEFAULT_LOCATION (optional): the location where you want Dataform to write BigQuery data. If unset, Dataform determines the location based on the datasets that your SQL query references. This works as follows:

    • If your query references datasets from the same location, Dataform uses that location.
    • If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see Cross-region dataset replication.
    • If your query doesn't reference any datasets, the default location for Dataform is the US multi-region. To choose a different location, set the default location. Alternatively, use the @@location system variable in your query. For more information, see Specify locations.

Update Dataform core

• To update the Dataform core framework, update the dataformCoreVersion in workflow_settings.yaml file, then re-run NPM install:

  npm i
  

Update Dataform CLI

• To update the Dataform CLI tool, run the following command:

  npm i -g @dataform/cli@^3.0.50
  

Create a credentials file

Dataform requires a credentials file to connect to remote services and create the .df-credentials.json file on your disk.

To create the credentials file, follow these steps:

1. Run the following command:

   dataform init-creds
   

2. Follow the init-creds wizard that walks you through credentials file creation.

Create a project

An empty Dataform project in Dataform core 3.0.0 or later has the following structure:

   project-dir
   ├── definitions
   ├── includes
   └── workflow_settings.yaml

• To create a Dataform project to deploy assets to BigQuery, run the following command:

  dataform init PROJECT_NAME --default-database YOUR_GOOGLE_CLOUD_PROJECT_ID --default-location DEFAULT_LOCATION
  

  Replace the following:

  • PROJECT_NAME: the name of your project.
  • YOUR_GOOGLE_CLOUD_PROJECT_ID: your Google Cloud project ID.

  • DEFAULT_LOCATION (optional): the location where you want Dataform to write BigQuery data. If unset, Dataform determines the location based on the datasets that your SQL query references. This works as follows:

    • If your query references datasets from the same location, Dataform uses that location.
    • If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see Cross-region dataset replication.
    • If your query doesn't reference any datasets, the default location for Dataform is the US multi-region. To choose a different location, set the default location. Alternatively, use the @@location system variable in your query. For more information, see Specify locations.

Clone a project

To clone an existing Dataform project from a third-party Git repository, follow the instructions from your Git provider.

• Once the repository is cloned, run the following command inside the cloned repository directory:

  dataform install
  

Define a table

Store definitions in the definitions/ folder.

• To define a table, run the following command:

  echo "config { type: 'TABLE_TYPE' } SELECT_STATEMENT" > definitions/FILE.sqlx
  

  Replace the following:

  • TABLE_TYPE: the type of the table: table, incremental, or view.
  • SELECT_STATEMENT: a SELECT statement that defines the table.
  • FILE: the name for the table definition file.

The following code sample defines a view in the example SQLX file.

echo "config { type: 'view' } SELECT 1 AS test" > definitions/example.sqlx

Define a manual assertion

Store definitions in the definitions/ folder.

• To define a manual assertion, run the following command:

  echo "config { type: 'assertion' } SELECT_STATEMENT" > definitions/FILE.sqlx
  

  Replace the following:

  • SELECT_STATEMENT: a SELECT statement that defines the assertion.
  • FILE: the name for the custom SQL operation definition file.

Define a custom SQL operation

Store definitions in the definitions/ folder.

• To define a custom SQL operation, run the following command:

  echo "config { type: 'operations' } SQL_QUERY" > definitions/FILE.sqlx
  

  Replace the following:

  • SQL_QUERY: your custom SQL operation.
  • FILE: the name for the custom SQL operation definition file.

View compilation output

Dataform compiles your code in real time.

• To view the output of the compilation process in the terminal, run the following command:

  dataform compile