Top / Google Cloud Platform / Google Kubernetes (Container) Engine / Cluster

Google Kubernetes (Container) Engine Cluster

This page shows how to write Terraform for Kubernetes (Container) Engine Cluster and write them securely.

terraform-logo

Review your .tf file for Google best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

google_container_cluster (Terraform)

The Cluster in Kubernetes (Container) Engine can be configured in Terraform with the resource name google_container_cluster. The following sections describe how to use the resource and its parameters.

Example Usage from GitHub

An example could not be found in GitHub.

Review your Terraform file for Google best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Security Best Practices for google_container_cluster

There are 12 settings in google_container_cluster that should be taken care of for security reasons. The following section explain an overview and example code.

risk-label

Ensure legacy authentication is disabled

The legacy authentication for your GKE cluster is enabled. It is better to use OAuth authentication instead.

risk-label

Ensure a legacy ABAC is disabled

The legacy ABAC permission for the GKE cluster should be enabled to utilize RBAC permissions.

risk-label

Ensure legacy metadata endpoints for your GKE cluster are disabled

It's better to disable legacy metadata endpoints for the GKE cluster. If it is enabled, legacy metadata APIs might pose a potential attacker to retrieve instance metadata.

risk-label

Ensure the network policy for your GKE cluster is enabled

It's better to enable the network policy for the GKE cluster to control the communication between the clusters and services.

risk-label

Ensure your GKE cluster uses private nodes if possible

GKE cluster should use private nodes if possible. It should be enabled for the nodes which, are only available internally.

risk-label

Ensure a custom service account is set to your GKE nodes

The service account for the GKE cluster should be configured. It is better to create and select a service account that has limited privileges.

risk-label

Ensure cluster labels are configured

It is better to configure cluster labels for the management of complex resources.

risk-label

Ensure a logging service of your GKE cluster is specified

It's better to specify a logging service of your GKE cluster.

risk-label

Ensure master authorized networks of your GKE cluster is configured

It's better to configure master authorized networks of your GKE cluster to restrict master access to a set of CIDR ranges.

risk-label

Ensure a monitoring service of your GKE cluster is specified

It's better to set a monitoring service of the GKE cluster to monitor the metrics.

risk-label

Ensure Shielded GKE nodes for your GKE cluster is enabled

It is better to enable Shielded GKE nodes for providing verifiable node identity to enhance the security.

risk-label

Ensure GKE control plane should not be publicly accessible on the Internet

The control plane of the GKE cluster should not be publicly accessible. It's better to utilize private nodes and master authorised networks to prevent it.

Review your Google Kubernetes (Container) Engine settings

You can check if the google_container_cluster setting in your .tf file is correct in 3 min with Shisho Cloud.

Parameters

The IP address range of the Kubernetes pods in this cluster in CIDR notation (e.g. 10.96.0.0/14). Leave blank to have one automatically chosen or specify a /14 block in 10.0.0.0/8. This field will only work for routes-based clusters, where ip_allocation_policy is not defined.

The desired datapath provider for this cluster. By default, uses the IPTables-based kube-proxy implementation.

The default maximum number of pods per node in this cluster. This doesn't work on "routes-based" clusters, clusters that don't have IP Aliasing enabled.

Description of the cluster.

Enable Autopilot for this cluster.

Enable Binary Authorization for this cluster. If enabled, all container images will be validated by Google Binary Authorization.

Whether Intra-node visibility is enabled for this cluster. This makes same node pod to pod traffic visible for VPC network.

Whether to enable Kubernetes Alpha features for this cluster. Note that when this option is enabled, the cluster cannot be upgraded and will be automatically deleted after 30 days.

Whether the ABAC authorizer is enabled for this cluster. When enabled, identities in the system, including service accounts, nodes, and controllers, will have statically granted permissions beyond those provided by the RBAC configuration or IAM. Defaults to false.

Enable Shielded Nodes features on all nodes in this cluster.

Whether to enable Cloud TPU resources in this cluster.

The IP address of this cluster's Kubernetes master.

The number of nodes to create in this cluster's default node pool. In regional or multi-zonal clusters, this is the number of nodes per zone. Must be set if node_pool is not set. If you're using google_container_node_pool objects with no default node pool, you'll need to set this to a value of at least 1, alongside setting remove_default_node_pool to true.

List of instance group URLs which have been assigned to the cluster.

The fingerprint of the set of labels for this cluster.

The location (region or zone) in which the cluster master will be created, as well as the default node location. If you specify a zone (such as us-central1-a), the cluster will be a zonal cluster with a single cluster master. If you specify a region (such as us-west1), the cluster will be a regional cluster with multiple masters spread across zones in the region, and with default node locations in those zones as well.

The logging service that the cluster should write logs to. Available options include logging.googleapis.com(Legacy Stackdriver), logging.googleapis.com/kubernetes(Stackdriver Kubernetes Engine Logging), and none. Defaults to logging.googleapis.com/kubernetes.

The current version of the master in the cluster. This may be different than the min_master_version set in the config if the master has been updated by GKE.

The minimum version of the master. GKE will auto-update the master to new versions, so this does not guarantee the current master version--use the read-only master_version field to obtain that. If unset, the cluster's version will be set by GKE to the version of the most recent official release (which is not necessarily the latest version).

The monitoring service that the cluster should write metrics to. Automatically send metrics from pods in the cluster to the Google Cloud Monitoring API. VM metrics will be collected by Google Compute Engine regardless of this setting Available options include monitoring.googleapis.com(Legacy Stackdriver), monitoring.googleapis.com/kubernetes(Stackdriver Kubernetes Engine Monitoring), and none. Defaults to monitoring.googleapis.com/kubernetes.

The name of the cluster, unique within the project and location.

The name or self_link of the Google Compute Engine network to which the cluster is connected. For Shared VPC, set this to the self link of the shared network.

Determines whether alias IPs or routes will be used for pod IPs in the cluster.

The list of zones in which the cluster's nodes are located. Nodes must be in the region of their regional cluster or in the same region as their cluster's zone for zonal clusters. If this is specified for a zonal cluster, omit the cluster's zone.

The Kubernetes version on the nodes. Must either be unset or set to the same value as min_master_version on create. Defaults to the default version set by GKE which is not necessarily the latest version. This only affects nodes in the default node pool. While a fuzzy version can be specified, it's recommended that you specify explicit versions as Terraform will see spurious diffs when fuzzy versions are used. See the google_container_engine_versions data source's version_prefix field to approximate fuzzy versions in a Terraform-compatible way. To update nodes in other node pools, use the version attribute on the node pool.

The desired state of IPv6 connectivity to Google Services. By default, no private IPv6 access to or from Google Services (all access will be via IPv4).

The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

If true, deletes the default node pool upon cluster creation. If you're using google_container_node_pool resources with no default node pool, this should be set to true, alongside setting initial_node_count to at least 1.

The GCE resource labels (a map of key/value pairs) to be applied to the cluster.

Server-defined URL for the resource.

The IP address range of the Kubernetes services in this cluster, in CIDR notation (e.g. 1.2.3.4/29). Service addresses are typically put in the last /16 from the container CIDR.

The name or self_link of the Google Compute Engine subnetwork in which the cluster's instances are launched.

The IP address range of the Cloud TPUs in this cluster, in CIDR notation (e.g. 1.2.3.4/29).

  • addons_config list block
  • authenticator_groups_config list block

    The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format gke-security-groups@yourdomain.com.

  • cluster_autoscaling list block

    Whether node auto-provisioning is enabled. Resource limits for cpu and memory must be defined to enable node auto-provisioning.

    • auto_provisioning_defaults list block

      Scopes that are used by NAP when creating node pools.

      The Google Cloud Platform Service Account to be used by the node VMs.

    • resource_limits list block

      Maximum amount of the resource in the cluster.

      Minimum amount of the resource in the cluster.

      The type of the resource. For example, cpu and memory. See the guide to using Node Auto-Provisioning for a list of types.

  • database_encryption list block

    The key to use to encrypt/decrypt secrets.

    ENCRYPTED or DECRYPTED.

  • default_snat_status list block

    When disabled is set to false, default IP masquerade rules will be applied to the nodes to prevent sNAT on cluster internal traffic.

  • ip_allocation_policy list block

    The IP address range for the cluster pod IPs. Set to blank to have a range chosen with the default size. Set to /netmask (e.g. /14) to have a range chosen with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use.

    The name of the existing secondary range in the cluster's subnetwork to use for pod IP addresses. Alternatively, cluster_ipv4_cidr_block can be used to automatically create a GKE-managed one.

    The IP address range of the services IPs in this cluster. Set to blank to have a range chosen with the default size. Set to /netmask (e.g. /14) to have a range chosen with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use.

    The name of the existing secondary range in the cluster's subnetwork to use for service ClusterIPs. Alternatively, services_ipv4_cidr_block can be used to automatically create a GKE-managed one.

  • maintenance_policy list block
  • master_auth list block

    Base64 encoded public certificate used by clients to authenticate to the cluster endpoint.

    Base64 encoded private key used by clients to authenticate to the cluster endpoint.

    Base64 encoded public certificate that is the root of trust for the cluster.

    The password to use for HTTP basic authentication when accessing the Kubernetes master endpoint.

    The username to use for HTTP basic authentication when accessing the Kubernetes master endpoint. If not present basic auth will be disabled.

  • master_authorized_networks_config list block
    • cidr_blocks set block

      External network that can access Kubernetes master through HTTPS. Must be specified in CIDR notation.

      Field for users to identify CIDR blocks.

  • network_policy list block

    Whether network policy is enabled on the cluster.

    The selected network policy provider. Defaults to PROVIDER_UNSPECIFIED.

  • node_config list block

    Size of the disk attached to each node, specified in GB. The smallest allowed disk size is 10GB.

    Type of the disk attached to each node.

    List of the type and count of accelerator cards attached to the instance.

    The image type to use for this node. Note that for a given image type, the latest version of it will be used.

    • labels optional computed - map from string to string

    The map of Kubernetes labels (key/value pairs) to be applied to each node. These will added in addition to any default label(s) that Kubernetes may apply to the node.

    The number of local SSD disks to be attached to the node.

    The name of a Google Compute Engine machine type.

    • metadata optional computed - map from string to string

    The metadata key/value pairs assigned to instances in the cluster.

    Minimum CPU platform to be used by this instance. The instance may be scheduled on the specified or newer CPU platform.

    The set of Google API scopes to be made available on all of the node VMs.

    Whether the nodes are created as preemptible VM instances.

    The Google Cloud Platform Service Account to be used by the node VMs.

    • tags optional - list of string

    The list of instance tags applied to all nodes.

    • taint optional computed - list of object

    List of Kubernetes taints to be applied to each node.

  • node_pool list block

    The initial number of nodes for the pool. In regional or multi-zonal clusters, this is the number of nodes per zone. Changing this will force recreation of the resource.

    The resource URLs of the managed instance groups associated with this node pool.

    The maximum number of pods per node in this node pool. Note that this does not work on node pools which are "route-based" - that is, node pools belonging to clusters that do not have IP Aliasing enabled.

    • name optional computed - string

    The name of the node pool. If left blank, Terraform will auto-generate a unique name.

    Creates a unique name for the node pool beginning with the specified prefix. Conflicts with name.

    The number of nodes per instance group. This field can be used to update the number of nodes per instance group but should not be used alongside autoscaling.

    The list of zones in which the node pool's nodes should be located. Nodes must be in the region of their regional cluster or in the same region as their cluster's zone for zonal clusters. If unspecified, the cluster-level node_locations will be used.

    The Kubernetes version for the nodes in this pool. Note that if this field and auto_upgrade are both specified, they will fight each other for what the node version should be, so setting both is highly discouraged. While a fuzzy version can be specified, it's recommended that you specify explicit versions as Terraform will see spurious diffs when fuzzy versions are used. See the google_container_engine_versions data source's version_prefix field to approximate fuzzy versions in a Terraform-compatible way.

    • autoscaling list block

      Maximum number of nodes in the NodePool. Must be >= min_node_count.

      Minimum number of nodes in the NodePool. Must be >=0 and <= max_node_count.

    • management list block

      Whether the nodes will be automatically repaired.

      Whether the nodes will be automatically upgraded.

    • node_config list block

      Size of the disk attached to each node, specified in GB. The smallest allowed disk size is 10GB.

      Type of the disk attached to each node.

      List of the type and count of accelerator cards attached to the instance.

      The image type to use for this node. Note that for a given image type, the latest version of it will be used.

      • labels optional computed - map from string to string

      The map of Kubernetes labels (key/value pairs) to be applied to each node. These will added in addition to any default label(s) that Kubernetes may apply to the node.

      The number of local SSD disks to be attached to the node.

      The name of a Google Compute Engine machine type.

      • metadata optional computed - map from string to string

      The metadata key/value pairs assigned to instances in the cluster.

      Minimum CPU platform to be used by this instance. The instance may be scheduled on the specified or newer CPU platform.

      The set of Google API scopes to be made available on all of the node VMs.

      Whether the nodes are created as preemptible VM instances.

      The Google Cloud Platform Service Account to be used by the node VMs.

      • tags optional - list of string

      The list of instance tags applied to all nodes.

      • taint optional computed - list of object

      List of Kubernetes taints to be applied to each node.

    • upgrade_settings list block

      The number of additional nodes that can be added to the node pool during an upgrade. Increasing max_surge raises the number of nodes that can be upgraded simultaneously. Can be set to 0 or greater.

      The number of nodes that can be simultaneously unavailable during an upgrade. Increasing max_unavailable raises the number of nodes that can be upgraded in parallel. Can be set to 0 or greater.

  • pod_security_policy_config list block

    Enable the PodSecurityPolicy controller for this cluster. If enabled, pods must be valid under a PodSecurityPolicy to be created.

  • private_cluster_config list block

    Enables the private cluster feature, creating a private endpoint on the cluster. In a private cluster, nodes only have RFC 1918 private addresses and communicate with the master's private endpoint via private networking.

    When true, the cluster's private endpoint is used as the cluster endpoint and access through the public endpoint is disabled. When false, either endpoint can be used. This field only applies to private clusters, when enable_private_nodes is true.

    The IP range in CIDR notation to use for the hosted master network. This range will be used for assigning private IP addresses to the cluster master(s) and the ILB VIP. This range must not overlap with any other ranges in use within the cluster's network, and it must be a /28 subnet. See Private Cluster Limitations for more details. This field only applies to private clusters, when enable_private_nodes is true.

    The name of the peering between this cluster and the Google owned VPC.

    The internal IP address of this cluster's master endpoint.

    The external IP address of this cluster's master endpoint.

  • release_channel list block

    The selected release channel. Accepted values are: UNSPECIFIED: Not set. RAPID: Weekly upgrade cadence; Early testers and developers who requires new features. REGULAR: Multiple per month upgrade cadence; Production users who need features not yet offered in the Stable channel. STABLE: Every few months upgrade cadence; Production users who need stability above all else, and for whom frequent upgrades are too risky.

  • resource_usage_export_config list block

    Whether to enable network egress metering for this cluster. If enabled, a daemonset will be created in the cluster to meter network egress traffic.

    Whether to enable resource consumption metering on this cluster. When enabled, a table will be created in the resource export BigQuery dataset to store resource consumption data. The resulting table can be joined with the resource usage table or with BigQuery billing export. Defaults to true.

  • timeouts single block
  • vertical_pod_autoscaling list block

    Enables vertical pod autoscaling.

  • workload_identity_config list block

    Enables workload identity.

>> from Terraform Registry

Explanation in Terraform Registry

-> Visit the Provision a GKE Cluster (Google Cloud) Learn tutorial to learn how to provision and interact with a GKE cluster. -> See the Using GKE with Terraform guide for more information about using GKE with Terraform. Manages a Google Kubernetes Engine (GKE) cluster. For more information see the official documentation and the API reference.

Note: All arguments and attributes, including basic auth username and passwords as well as certificate outputs will be stored in the raw state as plaintext. Read more about sensitive data in state.

>> from Terraform Registry

Tips: Best Practices for The Other Google Kubernetes (Container) Engine Resources

In addition to the google_container_node_pool, Google Kubernetes (Container) Engine has the other resources that should be configured for security reasons. Please check some examples of those resources and precautions.

risk-label

google_container_node_pool

Ensure to use Container-Optimized OS (cos) for node images

It is better to use Container-Optimized OS (cos) for node images. GKE supports several OS image types. However, COS_CONTAINERD should be used for enhanced security.

Review your Google Kubernetes (Container) Engine settings

In addition to the above, there are other security points you should be aware of making sure that your .tf files are protected in Shisho Cloud.

The Other Related Google Kubernetes (Container) Engine Resources

Frequently asked questions

What is Google Kubernetes (Container) Engine Cluster?

Google Kubernetes (Container) Engine Cluster is a resource for Kubernetes (Container) Engine of Google Cloud Platform. Settings can be wrote in Terraform.

security-icon

Automate config file reviews on your commits

Fix issues in your infrastructure as code with auto-generated patches.

Table of Contents