AWS Amazon EMR Instance Group

This page shows how to write Terraform and CloudFormation for Amazon EMR Instance Group and write them securely.

aws_emr_instance_group (Terraform)

The Instance Group in Amazon EMR can be configured in Terraform with the resource name aws_emr_instance_group. The following sections describe 2 examples of how to use the resource and its parameters.

Example Usage from GitHub

emr_instance_group.tf#L1
resource "aws_emr_instance_group" "emr_instance_group" {
  name          = var.emr_instance_group_name
  cluster_id    = aws_emr_cluster.emr_cluster.id
  instance_type = var.emr_instance_group_instance_type

  instance_count      = var.emr_instance_group_instance_count
main.tf#L7
resource "aws_emr_instance_group" "this" {
  autoscaling_policy  = var.autoscaling_policy
  bid_price           = var.bid_price
  cluster_id          = var.cluster_id
  configurations_json = var.configurations_json
  ebs_optimized       = var.ebs_optimized

Parameters

Explanation in Terraform Registry

Provides an Elastic MapReduce Cluster Instance Group configuration. See Amazon Elastic MapReduce Documentation for more information.

NOTE: At this time, Instance Groups cannot be destroyed through the API nor web interface. Instance Groups are destroyed when the EMR Cluster is destroyed. Terraform will resize any Instance Group to zero when destroying the resource.

AWS::EMR::InstanceGroupConfig (CloudFormation)

The InstanceGroupConfig in EMR can be configured in CloudFormation with the resource name AWS::EMR::InstanceGroupConfig. The following sections describe 10 examples of how to use the resource and its parameters.

Example Usage from GitHub

Presto_Cluster.cf.yml#L66
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: 2
      InstanceType: "r5.4xlarge"
      InstanceRole: "TASK"
      Market: "ON_DEMAND"
Presto_Cluster.cf.yml#L66
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: 2
      InstanceType: "r5.4xlarge"
      InstanceRole: "TASK"
      Market: "ON_DEMAND"
emr-complex-template.yml#L199
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: !Ref taskInstanceCount
      InstanceType: !Ref taskInstanceType
      BidPrice: !Ref taskBidPrice
      Market: SPOT
2000-emr.yml#L116
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      Name: !Sub "${StageName}_emr_task"
      AutoScalingPolicy:
        Constraints:
          MaxCapacity: !Sub "{{resolve:ssm:/me/${StageName}/emr/taskmaxcapacity:1}}"
emr.yaml#L230
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount:
        Ref: taskInstanceCount
      InstanceType:
        Ref: taskInstanceType
test-emr.json#L58
      "Type": "AWS::EMR::InstanceGroupConfig",
      "Properties": {
        "BidPrice": "0.039",
        "InstanceCount": "10",
        "InstanceRole": "TASK",
        "InstanceType": "m1.large",
emr_with_security_config.json#L396
            "Type": "AWS::EMR::InstanceGroupConfig",
            "Properties": {
                "InstanceCount": {
                    "Ref": "TaskInstanceCount"
                },
                "InstanceType": {
ElasticMapReduceInstanceGroupConfigSpecification.json#L3
    "AWS::EMR::InstanceGroupConfig.Configuration": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
      "Properties": {
        "Classification": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
          "PrimitiveType": "String",
ElasticMapReduceInstanceGroupConfigSpecification.json#L3
    "AWS::EMR::InstanceGroupConfig.Configuration": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
      "Properties": {
        "Classification": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
          "PrimitiveType": "String",
EMR.json#L261
  "resourceType" : "AWS::EMR::InstanceGroupConfig",
  "properties" : [ {
    "propertyName" : "BidPrice",
    "propertyType" : "String",
    "required" : false
  }, {

Parameters

Explanation in CloudFormation Registry

Use InstanceGroupConfig to define instance groups for an EMR cluster. A cluster can not use both instance groups and instance fleets. For more information, see Create a Cluster with Instance Fleets or Uniform Instance Groups in the Amazon EMR Management Guide.

Frequently asked questions

What is AWS Amazon EMR Instance Group?

AWS Amazon EMR Instance Group is a resource for Amazon EMR of Amazon Web Service. Settings can be wrote in Terraform and CloudFormation.

Where can I find the example code for the AWS Amazon EMR Instance Group?

For Terraform, the tunnavacher/terraform-aws-emr and niveklabs/aws source code examples are useful. See the Terraform Example section for further details.

For CloudFormation, the tochandrashekhar/EMR-on-EKS, dacort/demo-code and markmester/CloudformationEMR source code examples are useful. See the CloudFormation Example section for further details.