This page describes Managed Service for Apache Airflow versioning and lists differences between major Managed Airflow versions.
Major versions of Managed Airflow
Managed Airflow has the following major versions:
Managed Airflow (Gen 3). This version has simplified network setup and hides infrastructure components including the environment's cluster and environment's dependencies on other services.
Managed Airflow (Gen 2). In this version, the environment's cluster scales automatically to meet the resource demands.
Managed Airflow (Legacy Gen 1). Has manual environment scaling, the infrastructure is deployed to your projects and networks.
Comparison of Managed Airflow versions
The following table lists major differences between Managed Airflow (Legacy Gen 1), Managed Airflow (Gen 2), and Managed Airflow (Gen 3).
| Feature | Managed Airflow (Legacy Gen 1) | Managed Airflow (Gen 2) | Managed Airflow (Gen 3) |
|---|---|---|---|
| Image version | composer-1.b.c-airflow-x.y.z |
composer-2.b.c-airflow-x.y.z |
composer-3-airflow-x.y.z-build.t |
| Airflow versions | Airflow 1.10.* and Airflow 2 | Airflow 2 | Airflow 2, Airflow 3 |
| Python versions | 3.8.12 | 3.11.8 (3.11.5 and 3.8.12 in earlier versions) | 3.11.8 |
| Environment's cluster | Standard mode VPC-native or Routes-based Google Kubernetes Engine cluster | Autopilot mode VPC-native Google Kubernetes Engine cluster | Environment's cluster is not deployed into your project. |
| Horizontal scaling | Can adjust the number of nodes in the environment's cluster. This changes the number of Airflow workers. Can adjust the number of Airflow schedulers. |
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers. Can adjust the number of Airflow schedulers and triggerers |
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers. Can adjust the number of Airflow schedulers, triggerers, and DAG processors |
| Vertical scaling | Can set machine types for cluster nodes, Airflow web server and database when creating an environment. Can change machine types for Airflow web server and database. | Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and the environment size. | Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, DAG processors, and the environment size. |
| Pricing model | Managed Airflow (Legacy Gen 1) pricing model | Managed Airflow (Gen 2) pricing model | Managed Airflow (Gen 3) pricing model |
| Access control | Managed Airflow (Legacy Gen 1) roles | Managed Airflow roles. Uses Workload Identity Federation for GKE and requires additional permissions for the Managed Airflow Service Agent account. | Managed Airflow roles. Uses Workload Identity Federation for GKE. Extra permissions are not required |
| Access through Workforce Identity Federation | Not supported | Supported | Supported |
| Highly Resilient environments | Not supported | Supported | Supported |
| Database retention policy | Not supported | Not yet available | Supported |
| Private IP networking | VPC peerings | Private Service Connect | Simplified network setup. Can be changed between Public and Private IP in an existing environment. |
| VPC Service Controls | Supported | Supported | Supported |
| Airflow executors | Celery Executor | Celery Executor | CeleryKubernetes Executor |
| Airflow triggerer (deferrable operators) | Not supported | Supported | Supported |
| Custom service account for KubernetesPodOperator | Supported | Supported | Not supported |
| CMEK | Supported | Supported | Supported |
| Web server access controls | Supported | Supported | Supported |
| Restarting web server | Supported | Supported | Supported |
| Web server plugins | Supported in Airflow 1 environments with disabled DAG serialization. | Supported | Supported, plugins can be disabled and enabled on demand. |
| Scheduler plugins | Not supported | Supported | Not supported |
| Listeners | Not supported | Supported | Only task-level listeners |
| TimeTables | Only Built-In | Custom and Built-In | Only Built-In |
| Scheduler custom PyPI packages | Not supported | Supported | Not supported |
| Installing custom plugins | Supported | Supported | Supported |
| Maintenance operations | All tasks can be impacted. | Tasks that take less than 55 minutes to execute are not impacted. | Tasks that take less than 24 hours to execute are not impacted. |
| Storing Snapshots | Supported for Airflow 2.x and 1.10.15 versions | Supported | Supported |
| Loading Snapshots | Not supported | Supported | Supported |
| Scheduled Snapshots | Not supported | Supported | Supported |
| Custom environment's bucket | Not supported | Supported | Supported |
| Saving Airflow task logs only in Cloud Logging | Not supported | Supported | Supported |
| Data lineage integration | Not supported | Supported | Supported |
| Composer Local Development CLI tool | Not supported | Supported | Supported |
Comparison of Airflow versions
Managed Airflow (Gen 3) provides environments with Airflow 2 and Airflow 3.
Support for Managed Airflow (Gen 3) features
The following table summarizes Managed Airflow (Gen 3) features that aren't yet supported in Airflow 3.
| Feature | Status | Description |
|---|