Deployment configuration

This page explains the deployment configuration options for Cortex Framework across the following areas:

This page also provides How-to guides with step-by-step instructions for common deployment use cases and scenarios.

Configuration file: config/config.yaml

The config/config.yaml file — typically initialized from the config/config.yaml.example template — serves as the primary configuration for the Cortex Framework deployment. The configuration is divided into these structural blocks:

  1. Build environment (buildEnvironment): Governs the build orchestration layer, specifying the central Google Cloud project where intermediate metadata calculations, database validations, and schema lookups are billed and executed.
  2. Data (data): Governs the logical data architecture. This block configures dataset locations, namespace boundaries, connection details for raw ingestion sources, destination datasets, and registers the data module instances (foundations, catalogs, and products).
  3. Deployment (deployment): Configures physical target system deployments. It specifies the Dataform repository details (project ID, location, repository name, and development workspace) where compiled SQLX/JS transformation pipelines are deployed.

The following sections provide a detailed breakdown of each block.

Build environment

The build environment project is the project that gets billed for build actions, such as BigQuery jobs reading DD03L.

buildEnvironment:
  buildProjectId: YOUR_BUILD_PROJECT_ID

The following table describes the build environment parameters.

Parameter Meaning Default value Description
buildEnvironment.buildProjectId Build project ID YOUR_BUILD_PROJECT_ID Google Cloud Project ID where build operations are executed.

Data section overview

The data: section of the configuration file defines your data sources, targets, and the specific modules for the data foundation and data products. Its general structure is as follows:

data:
   # Geographic location for BigQuery datasets (for example: US, EU, us-central1)
   # For full list see: https://docs.cloud.google.com/cortex/docs/supported-locations
  bigQueryLocation: US
  # List of namespaces for data foundation and product modules.
  namespaces:
    - name: cortex
      path: ../src/data_modules/cortex
  # List of datasets mapping.
  datasets:
    - ...

  # Configuration for data foundation, data product, and external catalog modules.
  modules:
    # List of foundation modules.
    foundations:
    - ... 
    # List of external catalog modules.
    catalogs:
    - ...
    # List of data product modules.
    products:
    - ...

Data: BigQuery location

Defines the location of the BigQuery source and target datasets.

Parameter Meaning Default value Description
data.bigQueryLocation BigQuery Location US BigQuery dataset location (for example, US, us-central1, or europe-west1).

Data: Cortex namespace

Defines Cortex Framework namespace.

Parameter Meaning Default value Description
data.namespaces.name Namespace name - Cortex Framework namespace name. For example, cortex.
data.namespaces.path Namespace path - Cortex Framework namespace path for subdirectories used within src and config folder. For example, cortex.

Data: BigQuery sources and target datasets

The list of datasets defines the inbound raw data connection points and outbound storage locations for the framework. Each dataset registers a unique identifier mapped to a specific Google Cloud project and BigQuery dataset.

Datasets are referenced from the modules using their unique ID.

# Dataset mapping
datasets:
  - id: sap_raw
    projectId: YOUR_SOURCE_PROJECT_ID
    datasetId: cortex_sap_raw
  - id: sap_foundation
    projectId: YOUR_TARGET_PROJECT_ID
    datasetId: cortex7_sap_data_foundation

The following table describes the dataset mapping parameters.

Parameter Meaning Default value Description
data.datasets.id Dataset ID - Defines a unique identifier for the dataset (e.g., sap_raw or sap_foundation).
data.datasets.projectId Project ID - References the Google Cloud Project ID hosting the dataset.
data.datasets.datasetId BigQuery Dataset ID - References the actual BigQuery dataset name.

Data: Modules

The modules define the structure and components of the Dataform data pipelines.

Data: Modules: Foundations

This section configures the data foundation layer modules that process data from the raw layer into standardized latest records representation of the source data. In case the source provides a view on the latest records directly, or such transformations are performed by the source system connector, the module can be configured as an external data foundation source.

modules:
  # List of foundation modules.
  foundations:
    # Unique identifier for the module instance.
    - moduleId: erp
      # Path of the module format: {namespace}.{systemtype:sap}.{module_type:foundations}.{subsystemtype:sap}, for example, cortex.sap.foundations.sap.
      modulePath: cortex.sap.foundations.sap
      # Reference to the source dataset ID.
      dataSourceId: sap_raw
      # Reference to the target dataset ID.
      dataTargetId: sap_foundation
      # Module-specific configuration settings.
      moduleSettings:
        # SAP version (for example, ecc, s4).
        sapVersion: ecc
        # SAP client number.
        mandt: "100"
      # Whether the module is enabled.
      enabled: true
      # Whether the foundation is external (does not create target dataset).
      external: false
      # Custom table settings file, relative to 'config/' file directory
      # Recommended path: '{namespace_dir}/{system_type}/foundations/{system_sub_type}/table_settings.yaml' (e.g. 'cortex/sap/foundations/sap/table_settings.yaml')
      # Default path: '../src/data_modules/{namespace_dir}/{system_type}/foundations/{system_sub_type}/table_settings.default.yaml'
      tableSettings: "custom_table_settings.yaml"

The following table describes the data foundation modules parameters for modules.foundations configuration.

Parameter Meaning Default value Description
moduleId Module Identifier erp