AWS Amazon EMR Instance Group
This page shows how to write Terraform and CloudFormation for Amazon EMR Instance Group and write them securely.
aws_emr_instance_group (Terraform)
The Instance Group in Amazon EMR can be configured in Terraform with the resource name aws_emr_instance_group
. The following sections describe 2 examples of how to use the resource and its parameters.
Example Usage from GitHub
resource "aws_emr_instance_group" "emr_instance_group" {
name = var.emr_instance_group_name
cluster_id = aws_emr_cluster.emr_cluster.id
instance_type = var.emr_instance_group_instance_type
instance_count = var.emr_instance_group_instance_count
resource "aws_emr_instance_group" "this" {
autoscaling_policy = var.autoscaling_policy
bid_price = var.bid_price
cluster_id = var.cluster_id
configurations_json = var.configurations_json
ebs_optimized = var.ebs_optimized
Parameters
-
autoscaling_policy
optional - string -
bid_price
optional - string -
cluster_id
required - string -
configurations_json
optional - string -
ebs_optimized
optional - bool -
id
optional computed - string -
instance_count
optional - number -
instance_type
required - string -
name
optional - string -
running_instance_count
optional computed - number -
status
optional computed - string -
ebs_config
set block-
iops
optional - number -
size
required - number -
type
required - string -
volumes_per_instance
optional - number
-
Explanation in Terraform Registry
Provides an Elastic MapReduce Cluster Instance Group configuration. See Amazon Elastic MapReduce Documentation for more information.
NOTE: At this time, Instance Groups cannot be destroyed through the API nor web interface. Instance Groups are destroyed when the EMR Cluster is destroyed. Terraform will resize any Instance Group to zero when destroying the resource.
AWS::EMR::InstanceGroupConfig (CloudFormation)
The InstanceGroupConfig in EMR can be configured in CloudFormation with the resource name AWS::EMR::InstanceGroupConfig
. The following sections describe 10 examples of how to use the resource and its parameters.
Example Usage from GitHub
Type: AWS::EMR::InstanceGroupConfig
Properties:
InstanceCount: 2
InstanceType: "r5.4xlarge"
InstanceRole: "TASK"
Market: "ON_DEMAND"
Type: AWS::EMR::InstanceGroupConfig
Properties:
InstanceCount: 2
InstanceType: "r5.4xlarge"
InstanceRole: "TASK"
Market: "ON_DEMAND"
Type: AWS::EMR::InstanceGroupConfig
Properties:
InstanceCount: !Ref taskInstanceCount
InstanceType: !Ref taskInstanceType
BidPrice: !Ref taskBidPrice
Market: SPOT
Type: AWS::EMR::InstanceGroupConfig
Properties:
Name: !Sub "${StageName}_emr_task"
AutoScalingPolicy:
Constraints:
MaxCapacity: !Sub "{{resolve:ssm:/me/${StageName}/emr/taskmaxcapacity:1}}"
Type: AWS::EMR::InstanceGroupConfig
Properties:
InstanceCount:
Ref: taskInstanceCount
InstanceType:
Ref: taskInstanceType
"Type": "AWS::EMR::InstanceGroupConfig",
"Properties": {
"BidPrice": "0.039",
"InstanceCount": "10",
"InstanceRole": "TASK",
"InstanceType": "m1.large",
"Type": "AWS::EMR::InstanceGroupConfig",
"Properties": {
"InstanceCount": {
"Ref": "TaskInstanceCount"
},
"InstanceType": {
"AWS::EMR::InstanceGroupConfig.Configuration": {
"Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
"Properties": {
"Classification": {
"Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
"PrimitiveType": "String",
"AWS::EMR::InstanceGroupConfig.Configuration": {
"Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
"Properties": {
"Classification": {
"Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
"PrimitiveType": "String",
"resourceType" : "AWS::EMR::InstanceGroupConfig",
"properties" : [ {
"propertyName" : "BidPrice",
"propertyType" : "String",
"required" : false
}, {
Parameters
-
AutoScalingPolicy
optional - AutoScalingPolicy -
BidPrice
optional - String -
Configurations
optional - List of Configuration -
EbsConfiguration
optional - EbsConfiguration -
InstanceCount
required - Integer -
InstanceRole
required - String -
InstanceType
required - String -
JobFlowId
required - String -
Market
optional - String -
Name
optional - String
Explanation in CloudFormation Registry
Use
InstanceGroupConfig
to define instance groups for an EMR cluster. A cluster can not use both instance groups and instance fleets. For more information, see Create a Cluster with Instance Fleets or Uniform Instance Groups in the Amazon EMR Management Guide.
Frequently asked questions
What is AWS Amazon EMR Instance Group?
AWS Amazon EMR Instance Group is a resource for Amazon EMR of Amazon Web Service. Settings can be wrote in Terraform and CloudFormation.
Where can I find the example code for the AWS Amazon EMR Instance Group?
For Terraform, the tunnavacher/terraform-aws-emr and niveklabs/aws source code examples are useful. See the Terraform Example section for further details.
For CloudFormation, the dacort/demo-code, tochandrashekhar/EMR-on-EKS and markmester/CloudformationEMR source code examples are useful. See the CloudFormation Example section for further details.