Google Kubernetes (Container) Engine Cluster
This page shows how to write Terraform for Kubernetes (Container) Engine Cluster and write them securely.
google_container_cluster (Terraform)
The Cluster in Kubernetes (Container) Engine can be configured in Terraform with the resource name google_container_cluster
. The following sections describe how to use the resource and its parameters.
Example Usage from GitHub
An example could not be found in GitHub.
Security Best Practices for google_container_cluster
There are 12 settings in google_container_cluster that should be taken care of for security reasons. The following section explain an overview and example code.
Ensure legacy authentication is disabled
The legacy authentication for your GKE cluster is enabled. It is better to use OAuth authentication instead.
Ensure a legacy ABAC is disabled
The legacy ABAC permission for the GKE cluster should be enabled to utilize RBAC permissions.
Ensure legacy metadata endpoints for your GKE cluster are disabled
It's better to disable legacy metadata endpoints for the GKE cluster. If it is enabled, legacy metadata APIs might pose a potential attacker to retrieve instance metadata.
Ensure the network policy for your GKE cluster is enabled
It's better to enable the network policy for the GKE cluster to control the communication between the clusters and services.
Ensure your GKE cluster uses private nodes if possible
GKE cluster should use private nodes if possible. It should be enabled for the nodes which, are only available internally.
Ensure a custom service account is set to your GKE nodes
The service account for the GKE cluster should be configured. It is better to create and select a service account that has limited privileges.
Ensure cluster labels are configured
It is better to configure cluster labels for the management of complex resources.
Ensure a logging service of your GKE cluster is specified
It's better to specify a logging service of your GKE cluster.
Ensure master authorized networks of your GKE cluster is configured
It's better to configure master authorized networks of your GKE cluster to restrict master access to a set of CIDR ranges.
Ensure a monitoring service of your GKE cluster is specified
It's better to set a monitoring service of the GKE cluster to monitor the metrics.
Ensure Shielded GKE nodes for your GKE cluster is enabled
It is better to enable Shielded GKE nodes for providing verifiable node identity to enhance the security.
Ensure GKE control plane should not be publicly accessible on the Internet
The control plane of the GKE cluster should not be publicly accessible. It's better to utilize private nodes and master authorised networks to prevent it.
Parameters
-
cluster_ipv4_cidr
optional computed - string
The IP address range of the Kubernetes pods in this cluster in CIDR notation (e.g. 10.96.0.0/14). Leave blank to have one automatically chosen or specify a /14 block in 10.0.0.0/8. This field will only work for routes-based clusters, where ip_allocation_policy is not defined.
-
datapath_provider
optional computed - string
The desired datapath provider for this cluster. By default, uses the IPTables-based kube-proxy implementation.
-
default_max_pods_per_node
optional computed - number
The default maximum number of pods per node in this cluster. This doesn't work on "routes-based" clusters, clusters that don't have IP Aliasing enabled.
-
description
optional - string
Description of the cluster.
-
enable_autopilot
optional - bool
Enable Autopilot for this cluster.
-
enable_binary_authorization
optional - bool
Enable Binary Authorization for this cluster. If enabled, all container images will be validated by Google Binary Authorization.
-
enable_intranode_visibility
optional computed - bool
Whether Intra-node visibility is enabled for this cluster. This makes same node pod to pod traffic visible for VPC network.
-
enable_kubernetes_alpha
optional - bool
Whether to enable Kubernetes Alpha features for this cluster. Note that when this option is enabled, the cluster cannot be upgraded and will be automatically deleted after 30 days.
-
enable_legacy_abac
optional - bool
Whether the ABAC authorizer is enabled for this cluster. When enabled, identities in the system, including service accounts, nodes, and controllers, will have statically granted permissions beyond those provided by the RBAC configuration or IAM. Defaults to false.
-
enable_shielded_nodes
optional computed - bool
Enable Shielded Nodes features on all nodes in this cluster.
-
enable_tpu
optional - bool
Whether to enable Cloud TPU resources in this cluster.
-
endpoint
optional computed - string
The IP address of this cluster's Kubernetes master.
-
id
optional computed - string -
initial_node_count
optional - number
The number of nodes to create in this cluster's default node pool. In regional or multi-zonal clusters, this is the number of nodes per zone. Must be set if node_pool is not set. If you're using google_container_node_pool objects with no default node pool, you'll need to set this to a value of at least 1, alongside setting remove_default_node_pool to true.
-
instance_group_urls
optional computed - list of string
List of instance group URLs which have been assigned to the cluster.
-
label_fingerprint
optional computed - string
The fingerprint of the set of labels for this cluster.
-
location
optional computed - string
The location (region or zone) in which the cluster master will be created, as well as the default node location. If you specify a zone (such as us-central1-a), the cluster will be a zonal cluster with a single cluster master. If you specify a region (such as us-west1), the cluster will be a regional cluster with multiple masters spread across zones in the region, and with default node locations in those zones as well.
-
logging_service
optional computed - string
The logging service that the cluster should write logs to. Available options include logging.googleapis.com(Legacy Stackdriver), logging.googleapis.com/kubernetes(Stackdriver Kubernetes Engine Logging), and none. Defaults to logging.googleapis.com/kubernetes.
-
master_version
optional computed - string
The current version of the master in the cluster. This may be different than the min_master_version set in the config if the master has been updated by GKE.
-
min_master_version
optional - string
The minimum version of the master. GKE will auto-update the master to new versions, so this does not guarantee the current master version--use the read-only master_version field to obtain that. If unset, the cluster's version will be set by GKE to the version of the most recent official release (which is not necessarily the latest version).
-
monitoring_service
optional computed - string
The monitoring service that the cluster should write metrics to. Automatically send metrics from pods in the cluster to the Google Cloud Monitoring API. VM metrics will be collected by Google Compute Engine regardless of this setting Available options include monitoring.googleapis.com(Legacy Stackdriver), monitoring.googleapis.com/kubernetes(Stackdriver Kubernetes Engine Monitoring), and none. Defaults to monitoring.googleapis.com/kubernetes.
-
name
required - string
The name of the cluster, unique within the project and location.
-
network
optional - string
The name or self_link of the Google Compute Engine network to which the cluster is connected. For Shared VPC, set this to the self link of the shared network.
-
networking_mode
optional computed - string
Determines whether alias IPs or routes will be used for pod IPs in the cluster.
-
node_locations
optional computed - set of string
The list of zones in which the cluster's nodes are located. Nodes must be in the region of their regional cluster or in the same region as their cluster's zone for zonal clusters. If this is specified for a zonal cluster, omit the cluster's zone.
-
node_version
optional computed - string
The Kubernetes version on the nodes. Must either be unset or set to the same value as min_master_version on create. Defaults to the default version set by GKE which is not necessarily the latest version. This only affects nodes in the default node pool. While a fuzzy version can be specified, it's recommended that you specify explicit versions as Terraform will see spurious diffs when fuzzy versions are used. See the google_container_engine_versions data source's version_prefix field to approximate fuzzy versions in a Terraform-compatible way. To update nodes in other node pools, use the version attribute on the node pool.
-
operation
optional computed - string -
private_ipv6_google_access
optional computed - string
The desired state of IPv6 connectivity to Google Services. By default, no private IPv6 access to or from Google Services (all access will be via IPv4).
-
project
optional computed - string
The ID of the project in which the resource belongs. If it is not provided, the provider project is used.
-
remove_default_node_pool
optional - bool
If true, deletes the default node pool upon cluster creation. If you're using google_container_node_pool resources with no default node pool, this should be set to true, alongside setting initial_node_count to at least 1.
-
resource_labels
optional - map from string to string
The GCE resource labels (a map of key/value pairs) to be applied to the cluster.
-
self_link
optional computed - string
Server-defined URL for the resource.
-
services_ipv4_cidr
optional computed - string
The IP address range of the Kubernetes services in this cluster, in CIDR notation (e.g. 1.2.3.4/29). Service addresses are typically put in the last /16 from the container CIDR.
-
subnetwork
optional computed - string
The name or self_link of the Google Compute Engine subnetwork in which the cluster's instances are launched.
-
tpu_ipv4_cidr_block
optional computed - string
The IP address range of the Cloud TPUs in this cluster, in CIDR notation (e.g. 1.2.3.4/29).
-
addons_config
list block-
cloudrun_config
list block-
disabled
required - bool -
load_balancer_type
optional - string
-
-
horizontal_pod_autoscaling
list block-
disabled
required - bool
-
-
http_load_balancing
list block-
disabled
required - bool
-
-
network_policy_config
list block-
disabled
required - bool
-
-
-
authenticator_groups_config
list block-
security_group
required - string
The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format gke-security-groups@yourdomain.com.
-
-
cluster_autoscaling
list block-
enabled
required - bool
Whether node auto-provisioning is enabled. Resource limits for cpu and memory must be defined to enable node auto-provisioning.
-
auto_provisioning_defaults
list block-
oauth_scopes
optional computed - list of string
Scopes that are used by NAP when creating node pools.
-
service_account
optional - string
The Google Cloud Platform Service Account to be used by the node VMs.
-
-
resource_limits
list block-
maximum
optional - number
Maximum amount of the resource in the cluster.
-
minimum
optional - number
Minimum amount of the resource in the cluster.
-
resource_type
required - string
The type of the resource. For example, cpu and memory. See the guide to using Node Auto-Provisioning for a list of types.
-
-
-
database_encryption
list block-
key_name
optional - string
The key to use to encrypt/decrypt secrets.
-
state
required - string
ENCRYPTED or DECRYPTED.
-
-
default_snat_status
list block-
disabled
required - bool
When disabled is set to false, default IP masquerade rules will be applied to the nodes to prevent sNAT on cluster internal traffic.
-
-
ip_allocation_policy
list block-
cluster_ipv4_cidr_block
optional computed - string
The IP address range for the cluster pod IPs. Set to blank to have a range chosen with the default size. Set to /netmask (e.g. /14) to have a range chosen with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use.
-
cluster_secondary_range_name
optional computed - string
The name of the existing secondary range in the cluster's subnetwork to use for pod IP addresses. Alternatively, cluster_ipv4_cidr_block can be used to automatically create a GKE-managed one.
-
services_ipv4_cidr_block
optional computed - string
The IP address range of the services IPs in this cluster. Set to blank to have a range chosen with the default size. Set to /netmask (e.g. /14) to have a range chosen with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use.
-
services_secondary_range_name
optional computed - string
The name of the existing secondary range in the cluster's subnetwork to use for service ClusterIPs. Alternatively, services_ipv4_cidr_block can be used to automatically create a GKE-managed one.
-
-
maintenance_policy
list block-
daily_maintenance_window
list block-
duration
optional computed - string -
start_time
required - string
-
-
maintenance_exclusion
set block-
end_time
required - string -
exclusion_name
required - string -
start_time
required - string
-
-
recurring_window
list block-
end_time
required - string -
recurrence
required - string -
start_time
required - string
-
-
-
master_auth
list block-
client_certificate
optional computed - string
Base64 encoded public certificate used by clients to authenticate to the cluster endpoint.
-
client_key
optional computed - string
Base64 encoded private key used by clients to authenticate to the cluster endpoint.
-
cluster_ca_certificate
optional computed - string
Base64 encoded public certificate that is the root of trust for the cluster.
-
password
optional - string
The password to use for HTTP basic authentication when accessing the Kubernetes master endpoint.
-
username
optional - string
The username to use for HTTP basic authentication when accessing the Kubernetes master endpoint. If not present basic auth will be disabled.
-
client_certificate_config
list block-
issue_client_certificate
required - bool
Whether client certificate authorization is enabled for this cluster.
-
-
-
master_authorized_networks_config
list block-
cidr_blocks
set block-
cidr_block
required - string
External network that can access Kubernetes master through HTTPS. Must be specified in CIDR notation.
-
display_name
optional - string
Field for users to identify CIDR blocks.
-
-
-
network_policy
list block-
enabled
required - bool
Whether network policy is enabled on the cluster.
-
provider
optional - string
The selected network policy provider. Defaults to PROVIDER_UNSPECIFIED.
-
-
node_config
list block-
disk_size_gb
optional computed - number
Size of the disk attached to each node, specified in GB. The smallest allowed disk size is 10GB.
-
disk_type
optional computed - string
Type of the disk attached to each node.
-
guest_accelerator
optional computed - list of object
List of the type and count of accelerator cards attached to the instance.
-
count
- number -
type
- string -
image_type
optional computed - string
The image type to use for this node. Note that for a given image type, the latest version of it will be used.
-
labels
optional computed - map from string to string
The map of Kubernetes labels (key/value pairs) to be applied to each node. These will added in addition to any default label(s) that Kubernetes may apply to the node.
-
local_ssd_count
optional computed - number
The number of local SSD disks to be attached to the node.
-
machine_type
optional computed - string
The name of a Google Compute Engine machine type.
-
metadata
optional computed - map from string to string
The metadata key/value pairs assigned to instances in the cluster.
-
min_cpu_platform
optional - string
Minimum CPU platform to be used by this instance. The instance may be scheduled on the specified or newer CPU platform.
-
oauth_scopes
optional computed - set of string
The set of Google API scopes to be made available on all of the node VMs.
-
preemptible
optional - bool
Whether the nodes are created as preemptible VM instances.
-
service_account
optional computed - string
The Google Cloud Platform Service Account to be used by the node VMs.
-
tags
optional - list of string
The list of instance tags applied to all nodes.
-
taint
optional computed - list of object
List of Kubernetes taints to be applied to each node.
-
effect
- string -
key
- string -
value
- string -
shielded_instance_config
list block-
enable_integrity_monitoring
optional - bool
Defines whether the instance has integrity monitoring enabled.
-
enable_secure_boot
optional - bool
Defines whether the instance has Secure Boot enabled.
-
-
workload_metadata_config
list block-
node_metadata
required - string
NodeMetadata is the configuration for how to expose metadata to the workloads running on the node.
-
-
-
node_pool
list block-
initial_node_count
optional computed - number
The initial number of nodes for the pool. In regional or multi-zonal clusters, this is the number of nodes per zone. Changing this will force recreation of the resource.
-
instance_group_urls
optional computed - list of string
The resource URLs of the managed instance groups associated with this node pool.
-
max_pods_per_node
optional computed - number
The maximum number of pods per node in this node pool. Note that this does not work on node pools which are "route-based" - that is, node pools belonging to clusters that do not have IP Aliasing enabled.
-
name
optional computed - string
The name of the node pool. If left blank, Terraform will auto-generate a unique name.
-
name_prefix
optional computed - string
Creates a unique name for the node pool beginning with the specified prefix. Conflicts with name.
-
node_count
optional computed - number
The number of nodes per instance group. This field can be used to update the number of nodes per instance group but should not be used alongside autoscaling.
-
node_locations
optional computed - set of string
The list of zones in which the node pool's nodes should be located. Nodes must be in the region of their regional cluster or in the same region as their cluster's zone for zonal clusters. If unspecified, the cluster-level node_locations will be used.
-
version
optional computed - string
The Kubernetes version for the nodes in this pool. Note that if this field and auto_upgrade are both specified, they will fight each other for what the node version should be, so setting both is highly discouraged. While a fuzzy version can be specified, it's recommended that you specify explicit versions as Terraform will see spurious diffs when fuzzy versions are used. See the google_container_engine_versions data source's version_prefix field to approximate fuzzy versions in a Terraform-compatible way.
-
autoscaling
list block-
max_node_count
required - number
Maximum number of nodes in the NodePool. Must be >= min_node_count.
-
min_node_count
required - number
Minimum number of nodes in the NodePool. Must be >=0 and <= max_node_count.
-
-
management
list block-
auto_repair
optional - bool
Whether the nodes will be automatically repaired.
-
auto_upgrade
optional - bool
Whether the nodes will be automatically upgraded.
-
-
node_config
list block-
disk_size_gb
optional computed - number
Size of the disk attached to each node, specified in GB. The smallest allowed disk size is 10GB.
-
disk_type
optional computed - string
Type of the disk attached to each node.
-
guest_accelerator
optional computed - list of object
List of the type and count of accelerator cards attached to the instance.
-
count
- number -
type
- string -
image_type
optional computed - string
The image type to use for this node. Note that for a given image type, the latest version of it will be used.
-
labels
optional computed - map from string to string
The map of Kubernetes labels (key/value pairs) to be applied to each node. These will added in addition to any default label(s) that Kubernetes may apply to the node.
-
local_ssd_count
optional computed - number
The number of local SSD disks to be attached to the node.
-
machine_type
optional computed - string
The name of a Google Compute Engine machine type.
-
metadata
optional computed - map from string to string
The metadata key/value pairs assigned to instances in the cluster.
-
min_cpu_platform
optional - string
Minimum CPU platform to be used by this instance. The instance may be scheduled on the specified or newer CPU platform.
-
oauth_scopes
optional computed - set of string
The set of Google API scopes to be made available on all of the node VMs.
-
preemptible
optional - bool
Whether the nodes are created as preemptible VM instances.
-
service_account
optional computed - string
The Google Cloud Platform Service Account to be used by the node VMs.
-
tags
optional - list of string
The list of instance tags applied to all nodes.
-
taint
optional computed - list of object
List of Kubernetes taints to be applied to each node.
-
effect
- string -
key
- string -
value
- string -
shielded_instance_config
list block-
enable_integrity_monitoring
optional - bool
Defines whether the instance has integrity monitoring enabled.
-
enable_secure_boot
optional - bool
Defines whether the instance has Secure Boot enabled.
-
-
workload_metadata_config
list block-
node_metadata
required - string
NodeMetadata is the configuration for how to expose metadata to the workloads running on the node.
-
-
-
upgrade_settings
list block-
max_surge
required - number
The number of additional nodes that can be added to the node pool during an upgrade. Increasing max_surge raises the number of nodes that can be upgraded simultaneously. Can be set to 0 or greater.
-
max_unavailable
required - number
The number of nodes that can be simultaneously unavailable during an upgrade. Increasing max_unavailable raises the number of nodes that can be upgraded in parallel. Can be set to 0 or greater.
-
-
-
pod_security_policy_config
list block-
enabled
required - bool
Enable the PodSecurityPolicy controller for this cluster. If enabled, pods must be valid under a PodSecurityPolicy to be created.
-
-
private_cluster_config
list block-
enable_private_endpoint
required - bool
Enables the private cluster feature, creating a private endpoint on the cluster. In a private cluster, nodes only have RFC 1918 private addresses and communicate with the master's private endpoint via private networking.
-
enable_private_nodes
optional - bool
When true, the cluster's private endpoint is used as the cluster endpoint and access through the public endpoint is disabled. When false, either endpoint can be used. This field only applies to private clusters, when enable_private_nodes is true.
-
master_ipv4_cidr_block
optional - string
The IP range in CIDR notation to use for the hosted master network. This range will be used for assigning private IP addresses to the cluster master(s) and the ILB VIP. This range must not overlap with any other ranges in use within the cluster's network, and it must be a /28 subnet. See Private Cluster Limitations for more details. This field only applies to private clusters, when enable_private_nodes is true.
-
peering_name
optional computed - string
The name of the peering between this cluster and the Google owned VPC.
-
private_endpoint
optional computed - string
The internal IP address of this cluster's master endpoint.
-
public_endpoint
optional computed - string
The external IP address of this cluster's master endpoint.
-
master_global_access_config
list block-
enabled
required - bool
Whether the cluster master is accessible globally or not.
-
-
-
release_channel
list block-
channel
required - string
The selected release channel. Accepted values are: UNSPECIFIED: Not set. RAPID: Weekly upgrade cadence; Early testers and developers who requires new features. REGULAR: Multiple per month upgrade cadence; Production users who need features not yet offered in the Stable channel. STABLE: Every few months upgrade cadence; Production users who need stability above all else, and for whom frequent upgrades are too risky.
-
-
resource_usage_export_config
list block-
enable_network_egress_metering
optional - bool
Whether to enable network egress metering for this cluster. If enabled, a daemonset will be created in the cluster to meter network egress traffic.
-
enable_resource_consumption_metering
optional - bool
Whether to enable resource consumption metering on this cluster. When enabled, a table will be created in the resource export BigQuery dataset to store resource consumption data. The resulting table can be joined with the resource usage table or with BigQuery billing export. Defaults to true.
-
bigquery_destination
list block-
dataset_id
required - string
The ID of a BigQuery Dataset.
-
-
-
timeouts
single block -
vertical_pod_autoscaling
list block-
enabled
required - bool
Enables vertical pod autoscaling.
-
-
workload_identity_config
list block-
identity_namespace
required - string
Enables workload identity.
-
Explanation in Terraform Registry
-> Visit the Provision a GKE Cluster (Google Cloud) Learn tutorial to learn how to provision and interact with a GKE cluster. -> See the Using GKE with Terraform guide for more information about using GKE with Terraform. Manages a Google Kubernetes Engine (GKE) cluster. For more information see the official documentation and the API reference.
Note: All arguments and attributes, including basic auth username and passwords as well as certificate outputs will be stored in the raw state as plaintext. Read more about sensitive data in state.
Tips: Best Practices for The Other Google Kubernetes (Container) Engine Resources
In addition to the google_container_node_pool, Google Kubernetes (Container) Engine has the other resources that should be configured for security reasons. Please check some examples of those resources and precautions.
google_container_node_pool
Ensure to use Container-Optimized OS (cos) for node images
It is better to use Container-Optimized OS (cos) for node images. GKE supports several OS image types. However, COS_CONTAINERD should be used for enhanced security.