Top / Google Cloud Platform / Google Cloud Composer / Environment

Google Cloud Composer Environment

This page shows how to write Terraform for Cloud Composer Environment and write them securely.

terraform-logo

Review your .tf file for Google best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

google_composer_environment (Terraform)

The Environment in Cloud Composer can be configured in Terraform with the resource name google_composer_environment. The following sections describe 4 examples of how to use the resource and its parameters.

Example Usage from GitHub

main.tf#L1
resource "google_composer_environment" "composer_env" {
       name    = var.composer_env_name
       project = var.project_id
       region  = var.region

      config {
main.tf#L1
resource "google_composer_environment" "main" {
  provider = google-beta

  name    = var.config.name
  region  = var.region
  labels  = var.labels
main.tf#L1
resource "google_composer_environment" "composers_1" {
  for_each = { for c in var.composers : c.name => c if substr(c.software_config.image_version, 9, 1) == "1" }
  provider = google
  name     = each.value.name
  region   = var.region
  labels   = each.value.labels
composer.tf#L1
resource "google_composer_environment" "procamp_env" {
  provider = google-beta

  name   = "procamp-env"
  region = var.region
  labels = {

Review your Terraform file for Google best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

  • id optional computed - string
  • labels optional - map from string to string

User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions: Label keys must be between 1 and 63 characters long and must conform to the following regular expression: a-z?. Label values must be between 0 and 63 characters long and must conform to the regular expression (a-z?)?. No more than 64 labels can be associated with a given environment. Both keys and values must be <= 128 bytes in size.

Name of the environment.

The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

The location or Compute Engine region for the environment.

  • config list block

    The URI of the Apache Airflow Web UI hosted within this environment.

    The Cloud Storage prefix of the DAGs for this environment. Although Cloud Storage objects reside in a flat namespace, a hierarchical file tree can be simulated using '/'-delimited object name prefixes. DAG objects for this environment reside in a simulated directory with this prefix.

    The Kubernetes Engine cluster used to run this environment.

    The number of nodes in the Kubernetes Engine cluster that will be used to run this environment.

    • node_config list block

      The disk size in GB used for node VMs. Minimum size is 20GB. If unspecified, defaults to 100GB. Cannot be updated.

      Configuration for controlling how IPs are allocated in the GKE cluster. Cannot be updated.

      The Compute Engine machine type used for cluster instances, specified as a name or relative resource name. For example: "projects/[project]/zones/[zone]/machineTypes/[machineType]". Must belong to the enclosing environment's project and region/zone.

      The Compute Engine machine type used for cluster instances, specified as a name or relative resource name. For example: "projects/[project]/zones/[zone]/machineTypes/[machineType]". Must belong to the enclosing environment's project and region/zone. The network must belong to the environment's project. If unspecified, the "default" network ID in the environment's project is used. If a Custom Subnet Network is provided, subnetwork must also be provided.

      The set of Google API scopes to be made available on all node VMs. Cannot be updated. If empty, defaults to ["https://www.googleapis.com/auth/cloud-platform"].

      The Google Cloud Platform Service Account to be used by the node VMs. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated. If given, note that the service account must have roles/composer.worker for any GCP resources created under the Cloud Composer Environment.

      The Compute Engine subnetwork to be used for machine communications, , specified as a self-link, relative resource name (e.g. "projects/[project]/regions/[region]/subnetworks/[subnetwork]"), or by name. If subnetwork is provided, network must also be provided and the subnetwork must belong to the enclosing environment's project and region.

      • tags optional - set of string

      The list of instance tags applied to all node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC1035. Cannot be updated.

      The Compute Engine zone in which to deploy the VMs running the Apache Airflow software, specified as the zone name or relative resource name (e.g. "projects/[project]/zones/[zone]"). Must belong to the enclosing environment's project and region.

    • private_environment_config list block

      The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. Needs to be disjoint from web_server_ipv4_cidr_block.

      If true, access to the public endpoint of the GKE cluster is denied.

      The IP range in CIDR notation to use for the hosted master network. This range is used for assigning internal IP addresses to the cluster master or set of masters and to the internal load balancer virtual IP. This range must not overlap with any other ranges in use within the cluster's network. If left blank, the default value of '172.16.0.0/28' is used.

      The CIDR block from which IP range for web server will be reserved. Needs to be disjoint from master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block.

    • software_config list block

      Apache Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation". Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and cannot contain "=" or ";". Section and property names cannot contain characters: "." Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format. Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

      Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes. Environment variable names must match the regular expression [a-zA-Z_][a-za-z0-9_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression AIRFLOW[A-Z0-9_]+[A-Z0-9_]+), and they cannot match any of the following reserved names: AIRFLOW_HOME C_FORCE_ROOT CONTAINER_NAME DAGS_FOLDER GCP_PROJECT GCS_BUCKET GKE_CLUSTER_NAME SQL_DATABASE SQL_INSTANCE SQL_PASSWORD SQL_PROJECT SQL_REGION SQL_USER.

      The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-[0-9]+.[0-9]+(.[0-9]+)?-airflow-[0-9]+.[0-9]+(.[0-9]+.*)?. The Cloud Composer portion of the version is a semantic version. The portion of the image version following 'airflow-' is an official Apache Airflow repository release name. See documentation for allowed release names.

      Custom Python Package Index (PyPI) packages to be installed in the environment. Keys refer to the lowercase package name (e.g. "numpy"). Values are the lowercase extras and version specifier (e.g. "==1.12.0", "[devel,gcp_api]", "[devel]>=1.8.2, <1.9.2"). To specify a package without pinning it to a version specifier, use the empty string as the value.

      The major version of Python used to run the Apache Airflow scheduler, worker, and webserver processes. Can be set to '2' or '3'. If not specified, the default is '2'. Cannot be updated.

  • timeouts single block

>> from Terraform Registry

Explanation in Terraform Registry

An environment for running orchestration tasks. Environments run Apache Airflow software on Google infrastructure. To get more information about Environments, see:

>> from Terraform Registry

Frequently asked questions

What is Google Cloud Composer Environment?

Google Cloud Composer Environment is a resource for Cloud Composer of Google Cloud Platform. Settings can be wrote in Terraform.

Where can I find the example code for the Google Cloud Composer Environment?

For Terraform, the brodriguesmclara/desafio, AtsushiKitano/assets and ICTEAM-Test-AMPDWH/symphony_adp_infra_lib source code examples are useful. See the Terraform Example section for further details.

security-icon

Automate config file reviews on your commits

Fix issues in your infrastructure as code with auto-generated patches.

Table of Contents