Deployment configuration

This page explains the deployment configuration options for Cortex Framework across the following areas:

• Deployment configurations (config/config.yaml): Defines global variables, build environments, and module mapping (data foundation and data product targets).
• Table configurations (table_settings.yaml): Module-specific performance and schema specifications, outlining how base tables are compiled and conformed in BigQuery.

This page also provides How-to guides with step-by-step instructions for common deployment use cases and scenarios.

Configuration file: config/config.yaml

The config/config.yaml file — typically initialized from the config/config.yaml.example template — serves as the primary configuration for the Cortex Framework deployment. The configuration is divided into these structural blocks:

1. Build environment (buildEnvironment): Governs the build orchestration layer, specifying the central Google Cloud project where intermediate metadata calculations, database validations, and schema lookups are billed and executed.
2. Data (data): Governs the logical data architecture. This block configures dataset locations, namespace boundaries, connection details for raw ingestion sources, destination datasets, and registers the data module instances (foundations, catalogs, and products).
3. Deployment (deployment): Configures physical target system deployments. It specifies the Dataform repository details (project ID, location, repository name, and development workspace) where compiled SQLX/JS transformation pipelines are deployed.

The following sections provide a detailed breakdown of each block.

Build environment

The build environment project is the project that gets billed for build actions, such as BigQuery jobs reading DD03L.

buildEnvironment:
  buildProjectId: YOUR_BUILD_PROJECT_ID

The following table describes the build environment parameters.

Data section overview

The data: section of the configuration file defines your data sources, targets, and the specific modules for the data foundation and data products. Its general structure is as follows:

data:
   # Geographic location for BigQuery datasets (for example: US, EU, us-central1)
   # For full list see: https://docs.cloud.google.com/cortex/docs/supported-locations
  bigQueryLocation: US
  # List of namespaces for data foundation and product modules.
  namespaces:
    - name: cortex
      path: ../src/data_modules/cortex
  # List of datasets mapping.
  datasets:
    - ...

  # Configuration for data foundation, data product, and external catalog modules.
  modules:
    # List of foundation modules.
    foundations:
    - ... 
    # List of external catalog modules.
    catalogs:
    - ...
    # List of data product modules.
    products:
    - ...

Data: BigQuery location

Defines the location of the BigQuery source and target datasets.

Data: Cortex namespace

Defines Cortex Framework namespace.

Data: BigQuery sources and target datasets

The list of datasets defines the inbound raw data connection points and outbound storage locations for the framework. Each dataset registers a unique identifier mapped to a specific Google Cloud project and BigQuery dataset.

Datasets are referenced from the modules using their unique ID.

# Dataset mapping
datasets:
  - id: sap_raw
    projectId: YOUR_SOURCE_PROJECT_ID
    datasetId: cortex_sap_raw
  - id: sap_foundation
    projectId: YOUR_TARGET_PROJECT_ID
    datasetId: cortex7_sap_data_foundation

The following table describes the dataset mapping parameters.

Data: Modules

The modules define the structure and components of the Dataform data pipelines.

Data: Modules: Foundations

This section configures the data foundation layer modules that process data from the raw layer into standardized latest records representation of the source data. In case the source provides a view on the latest records directly, or such transformations are performed by the source system connector, the module can be configured as an external data foundation source.

modules:
  # List of foundation modules.
  foundations:
    # Unique identifier for the module instance.
    - moduleId: erp
      # Path of the module format: {namespace}.{systemtype:sap}.{module_type:foundations}.{subsystemtype:sap}, for example, cortex.sap.foundations.sap.
      modulePath: cortex.sap.foundations.sap
      # Reference to the source dataset ID.
      dataSourceId: sap_raw
      # Reference to the target dataset ID.
      dataTargetId: sap_foundation
      # Module-specific configuration settings.
      moduleSettings:
        # SAP version (for example, ecc, s4).
        sapVersion: ecc
        # SAP client number.
        mandt: "100"
      # Whether the module is enabled.
      enabled: true
      # Whether the foundation is external (does not create target dataset).
      external: false
      # Custom table settings file, relative to 'config/' file directory
      # Recommended path: '{namespace_dir}/{system_type}/foundations/{system_sub_type}/table_settings.yaml' (e.g. 'cortex/sap/foundations/sap/table_settings.yaml')
      # Default path: '../src/data_modules/{namespace_dir}/{system_type}/foundations/{system_sub_type}/table_settings.default.yaml'
      tableSettings: "custom_table_settings.yaml"

The following table describes the data foundation modules parameters for modules.foundations configuration.

Data: Modules: Catalogs

External lakehouse catalogs allow Cortex Framework to ingest external tables from BigLake Delta Sharing catalogs and shares without physical manifests.

modules:
  # List of external catalog modules.
  catalogs:
    # Unique identifier for the catalog.
    - id: sap_bdc_catalog
      # Type of the catalog.
      type: lakehouse_delta_share
      # Logical namespace prefixes bound by this catalog.
      bindsNamespaces: [sap_bdc]
      # Connection settings for the catalog.
      connectionSettings:
        # Unique identifier for the catalog.
        catalogId: sap_bdc_catalog
        # Unique identifier for the project hosting the catalog.
        projectId: sap_bdc_delta_share
        # Geographic region location for the catalog.
        location: europe-west3
        # List of shares to import.
        shares:
          - shareId: customer_v1_he2_100_p8123
          - shareId: salesorder_v1_he2_100_p8124
      # Whether the catalog is enabled.
      # enabled: true

The following table describes the external catalog configuration parameters.

Data: Modules: Products

Data product modules define the aggregations, calculations, and joins necessary to transform raw data into insights that fulfill specific business use cases.

The configuration of the data products allows setting of unique ID, definition of dependencies as well as reference of the data foundation module and target dataset where the results will be stored into.

Detailed configuration of given data products is defined within files referenced by the key: tableSettings.

modules:
  # List of data product modules.
  products:
    # Unique identifier for the data product instance.
    - moduleId: sap_purchasing_organizational_structure
      # Path of the data product (namespaced).
      modulePath: cortex.sap.products.purchasing_organizational_structure
      # Map of module dependencies.
      dependencyBindings:
        sapModule: erp
      # Reference to the target dataset ID.
      dataTargetId: product_target
      # Whether the module is enabled.
      enabled: true
      # Whether this data product is synced to the Knowledge Catalog. Defaults to true.
      syncToKc: true

      # Custom table settings file, relative to 'config/' file directory
      # Recommended path: '{namespace_dir}/{system_type}/products/{product_name}/table_settings.yaml'
      # If omitted, defaults to '../src/data_modules/{namespace_dir}/{system_type}/products/{product_name}/table_settings.default.yaml'
      # tableSettings: "custom_dataproduct_table_settings.yaml"

The following table describes the data product modules parameters for modules.products configuration.

Deployment environment

Cortex Framework uses Dataform to orchestrate SQL transformations within BigQuery. The deployment: block defines the Dataform configuration, responsible for the execution of the data pipelines, including the repository project, location, repository name, and the Dataform workspace name.

deployment:
  targets:
    - type: dataform
      enabled: true
      targetSettings:
        repositoryProjectId: YOUR_REPO_PROJECT_ID
        repositoryRegion: us-central1
        repositoryName: cortex-repository
        workspaceName: dev
        # serviceAccount: "example@example.com"

The following table describes the deployment targets location parameters (deployment.targets:).

Configuration file: table_settings.yaml

This guide explains how to use table_settings.yaml file to configure data foundation and data product tables in Google Cloud Cortex Framework.

The data module specific table_settings.yaml file controls how raw source tables are conformed and how analytical data models are materialized within BigQuery. Using this file, you can configure tags, materialization strategies, and advanced BigQuery performance features like partitioning or clustering.

Dynamic dependency resolution

By default, Cortex Framework optimizes deployment footprint and execution time by only deploying and compiling the foundation tables that are required as dependencies of your enabled data products. If a table configured in table_settings.yaml does not have any active downstream data products depending on it, it is omitted from deployment.

To override this optimization and force the deployment of a foundation table, you can set the deployAlways attribute to true (see Data foundation style parameter reference).

In Google Cloud Cortex Framework, each module (foundation or product) can be assigned a specific table settings file in the deployment configuration file: config/config.yaml using the tableSettings property.

Configuration paths

• Custom settings (Recommended): To customize table behaviors, copy the default file to your configuration directory, modify it, and reference its path in config/config.yaml. The recommended paths to use (relative to config/ directory), are:
  • Foundation modules: namespace_dir/system_type/foundations/system_sub_type/custom_table_settings.yaml (e.g., config/cortex/sap/foundations/sap/table_settings.yaml)
  • Product modules: namespace_dir/system_type/products/product_name/custom_table_settings.yaml (e.g., config/cortex/sap/products/accounting_documents/table_settings.yaml)
• Default fallback: If tableSettings is omitted, the framework automatically falls back to:
  • Foundation modules: ../src/data_modules/namespace_dir/system_type/foundations/system_sub_type/table_settings.default.yaml
  • Product modules: ../src/data_modules/namespace_dir/system_type/products/product_name/table_settings.default.yaml

Configuration styles

There are two distinct schema styles for table_settings.yaml depending on the category of the module:

1. Data Foundation Style: List-based mapping that defines the source-to-target schema relationships, CDC (Change Data Capture) handling, and BigQuery layout. Notice that the data foundation table-settings layout is source system specific.

2. Data Product Style: Map-based mapping (dictionary) that defines how analytical views or tables are materialized (e.g., as views, tables, or incremental tables) and optimized.

Both styles support three root-level sections to segregate configurations by source system version (primarily used for SAP Data Foundation and SAP-dependent products):

• ecc: Settings applied only when deploying an SAP ECC source system.
• s4: Settings applied only when deploying an SAP S/4HANA source system.
• common: Settings applied regardless of the SAP version (used for conformed or universal settings).

Data foundation style for SAP ERP

In a data foundation module for SAP ERP source systems, the table_settings.yaml file is structured as a list of table items under the ecc, s4, and common keys. Each item maps a raw source table to a conformed target table and configures its BigQuery settings.

YAML syntax example

common:
  - source:
      tableName: bkpf
      isCdc: true
    target:
      tableName: bkpf # Optional: defaults to source tableName if omitted
      bigQueryLabels:
        - key: data_class
          value