AWS Amazon MSK Cluster
This page shows how to write Terraform and CloudFormation for Amazon MSK Cluster and write them securely.
aws_msk_cluster (Terraform)
The Cluster in Amazon MSK can be configured in Terraform with the resource name aws_msk_cluster
. The following sections describe 5 examples of how to use the resource and its parameters.
Example Usage from GitHub
resource "aws_msk_cluster" "allowed" {
logging_info {
broker_logs {
cloudwatch_logs {
enabled = true
}
resource "aws_msk_cluster" "positive1" {
cluster_name = "example"
kafka_version = "2.4.1"
number_of_broker_nodes = 3
}
resource "aws_msk_cluster" "negative1" {
encryption_info {
encryption_at_rest_kms_key_arn = aws_kms_key.kms.arn
}
}
resource "aws_msk_cluster" "sensitive_data_cluster_1" {
encryption_info {
encryption_in_transit {
client_broker = "PLAINTEXT" # Noncompliant
}
}
resource "aws_msk_cluster" "sensitive_msk" {
cluster_name = "sensitive_msk"
}
resource "aws_msk_cluster" "sensitive_msk" {
cluster_name = "sensitive_msk"
Parameters
-
arn
optional computed - string -
bootstrap_brokers
optional computed - string -
bootstrap_brokers_sasl_scram
optional computed - string -
bootstrap_brokers_tls
optional computed - string -
cluster_name
required - string -
current_version
optional computed - string -
enhanced_monitoring
optional - string -
id
optional computed - string -
kafka_version
required - string -
number_of_broker_nodes
required - number -
tags
optional - map from string to string -
zookeeper_connect_string
optional computed - string -
broker_node_group_info
list block-
az_distribution
optional - string -
client_subnets
required - list of string -
ebs_volume_size
required - number -
instance_type
required - string -
security_groups
required - list of string
-
-
client_authentication
list block-
sasl
list block-
scram
optional - bool
-
-
tls
list block-
certificate_authority_arns
optional - set of string
-
-
-
configuration_info
list block -
encryption_info
list block-
encryption_at_rest_kms_key_arn
optional computed - string -
encryption_in_transit
list block-
client_broker
optional - string -
in_cluster
optional - bool
-
-
-
logging_info
list block-
broker_logs
list block-
cloudwatch_logs
list block -
firehose
list block-
delivery_stream
optional - string -
enabled
required - bool
-
-
s3
list block
-
-
-
open_monitoring
list block-
prometheus
list block-
jmx_exporter
list block-
enabled_in_broker
required - bool
-
-
node_exporter
list block-
enabled_in_broker
required - bool
-
-
-
Explanation in Terraform Registry
Manages AWS Managed Streaming for Kafka cluster
AWS::MSK::Cluster (CloudFormation)
The Cluster in MSK can be configured in CloudFormation with the resource name AWS::MSK::Cluster
. The following sections describe 10 examples of how to use the resource and its parameters.
Example Usage from GitHub
Type: AWS::MSK::Cluster
Condition: MTLSMSKCluster1
Properties:
BrokerNodeGroupInfo:
ClientSubnets:
- Fn::ImportValue:
Type: AWS::MSK::Cluster
Properties:
ClusterName: KafkaForWaterstream
KafkaVersion: !Ref KafkaVersion
NumberOfBrokerNodes: !Ref KafkaNodes
EnhancedMonitoring: DEFAULT
Type: "AWS::MSK::Cluster"
Properties:
BrokerNodeGroupInfo:
ClientSubnets: !Ref SubnetIds
InstanceType: "kafka.m5.large"
SecurityGroups:
Type: 'AWS::MSK::Cluster'
Properties:
ClusterName: spn-kafka-cluster
NumberOfBrokerNodes: 3
KafkaVersion: "2.2.1"
BrokerNodeGroupInfo:
Type: AWS::MSK::Cluster
Properties:
ClusterName: !Ref ClusterName
KafkaVersion: 2.2.1
NumberOfBrokerNodes: 2
EnhancedMonitoring: PER_BROKER
"Type": "AWS::MSK::Cluster",
"Properties": {
"ClusterName": "ClusterWithRequiredProperties",
"KafkaVersion": "2.2.1",
"NumberOfBrokerNodes": 3,
"BrokerNodeGroupInfo": {
"Type": "AWS::MSK::Cluster",
"Properties": {
"ClusterName": "ClusterWithRequiredProperties",
"KafkaVersion": "2.2.1",
"NumberOfBrokerNodes": 3,
"BrokerNodeGroupInfo": {
"Type": "AWS::MSK::Cluster",
"Properties": {
"ClusterName": "ClusterWithRequiredProperties",
"KafkaVersion": "2.2.1",
"LoggingInfo": {
"BrokerLogs": {
{ "LogicalResourceId":"MyMskCluster", "ResourceType":"AWS::MSK::Cluster" },
{ "LogicalResourceId":"MyMskStreamProcessorMyMskEvent", "ResourceType":"AWS::Lambda::EventSourceMapping" }
{ "LogicalResourceId":"MyMskCluster", "ResourceType":"AWS::MSK::Cluster" },
{ "LogicalResourceId":"MyMskStreamProcessorMyMskEvent", "ResourceType":"AWS::Lambda::EventSourceMapping" }
Parameters
-
BrokerNodeGroupInfo
required - BrokerNodeGroupInfo -
EnhancedMonitoring
optional - String -
KafkaVersion
required - String -
NumberOfBrokerNodes
required - Integer -
EncryptionInfo
optional - EncryptionInfo -
OpenMonitoring
optional - OpenMonitoring -
ClusterName
required - String -
ClientAuthentication
optional - ClientAuthentication -
LoggingInfo
optional - LoggingInfo -
Tags
optional - Json -
ConfigurationInfo
optional - ConfigurationInfo
Explanation in CloudFormation Registry
The
AWS::MSK::Cluster
resource creates an Amazon MSK cluster. For more information, see What Is Amazon MSK? in the Amazon MSK Developer Guide.
Frequently asked questions
What is AWS Amazon MSK Cluster?
AWS Amazon MSK Cluster is a resource for Amazon MSK of Amazon Web Service. Settings can be wrote in Terraform and CloudFormation.
Where can I find the example code for the AWS Amazon MSK Cluster?
For Terraform, the snyk-labs/infrastructure-as-code-goof, leonidweinbergcx/mykics and leonidweinbergcx/mykics source code examples are useful. See the Terraform Example section for further details.
For CloudFormation, the aws-samples/mirrormaker2-msk-migration, simplematter/waterstream-aws-ecs-cloudformation and michael-robbins/aws-streaming-session source code examples are useful. See the CloudFormation Example section for further details.