Top / Amazon Web Service / AWS Amazon EMR / Instance Group

AWS Amazon EMR Instance Group

This page shows how to write Terraform and CloudFormation for Amazon EMR Instance Group and write them securely.

terraform-logo

Review your .tf file for AWS best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

aws_emr_instance_group (Terraform)

The Instance Group in Amazon EMR can be configured in Terraform with the resource name aws_emr_instance_group. The following sections describe 2 examples of how to use the resource and its parameters.

Example Usage from GitHub

emr_instance_group.tf#L1
resource "aws_emr_instance_group" "emr_instance_group" {
  name          = var.emr_instance_group_name
  cluster_id    = aws_emr_cluster.emr_cluster.id
  instance_type = var.emr_instance_group_instance_type

  instance_count      = var.emr_instance_group_instance_count
main.tf#L7
resource "aws_emr_instance_group" "this" {
  autoscaling_policy  = var.autoscaling_policy
  bid_price           = var.bid_price
  cluster_id          = var.cluster_id
  configurations_json = var.configurations_json
  ebs_optimized       = var.ebs_optimized

Review your Terraform file for AWS best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

>> from Terraform Registry

Explanation in Terraform Registry

Provides an Elastic MapReduce Cluster Instance Group configuration. See Amazon Elastic MapReduce Documentation for more information.

NOTE: At this time, Instance Groups cannot be destroyed through the API nor web interface. Instance Groups are destroyed when the EMR Cluster is destroyed. Terraform will resize any Instance Group to zero when destroying the resource.

>> from Terraform Registry

AWS::EMR::InstanceGroupConfig (CloudFormation)

The InstanceGroupConfig in EMR can be configured in CloudFormation with the resource name AWS::EMR::InstanceGroupConfig. The following sections describe 10 examples of how to use the resource and its parameters.

Example Usage from GitHub

Presto_Cluster.cf.yml#L66
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: 2
      InstanceType: "r5.4xlarge"
      InstanceRole: "TASK"
      Market: "ON_DEMAND"
Presto_Cluster.cf.yml#L66
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: 2
      InstanceType: "r5.4xlarge"
      InstanceRole: "TASK"
      Market: "ON_DEMAND"
emr-complex-template.yml#L199
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount: !Ref taskInstanceCount
      InstanceType: !Ref taskInstanceType
      BidPrice: !Ref taskBidPrice
      Market: SPOT
2000-emr.yml#L116
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      Name: !Sub "${StageName}_emr_task"
      AutoScalingPolicy:
        Constraints:
          MaxCapacity: !Sub "{{resolve:ssm:/me/${StageName}/emr/taskmaxcapacity:1}}"
emr.yaml#L230
    Type: AWS::EMR::InstanceGroupConfig
    Properties:
      InstanceCount:
        Ref: taskInstanceCount
      InstanceType:
        Ref: taskInstanceType
test-emr.json#L58
      "Type": "AWS::EMR::InstanceGroupConfig",
      "Properties": {
        "BidPrice": "0.039",
        "InstanceCount": "10",
        "InstanceRole": "TASK",
        "InstanceType": "m1.large",
emr_with_security_config.json#L396
            "Type": "AWS::EMR::InstanceGroupConfig",
            "Properties": {
                "InstanceCount": {
                    "Ref": "TaskInstanceCount"
                },
                "InstanceType": {
ElasticMapReduceInstanceGroupConfigSpecification.json#L3
    "AWS::EMR::InstanceGroupConfig.Configuration": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
      "Properties": {
        "Classification": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
          "PrimitiveType": "String",
ElasticMapReduceInstanceGroupConfigSpecification.json#L3
    "AWS::EMR::InstanceGroupConfig.Configuration": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html",
      "Properties": {
        "Classification": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-configuration.html#cfn-emr-cluster-configuration-classification",
          "PrimitiveType": "String",
EMR.json#L261
  "resourceType" : "AWS::EMR::InstanceGroupConfig",
  "properties" : [ {
    "propertyName" : "BidPrice",
    "propertyType" : "String",
    "required" : false
  }, {

Parameters

>> from AWS CloudFormation Documentation

Explanation in CloudFormation Registry

Use InstanceGroupConfig to define instance groups for an EMR cluster. A cluster can not use both instance groups and instance fleets. For more information, see Create a Cluster with Instance Fleets or Uniform Instance Groups in the Amazon EMR Management Guide.

>> from AWS CloudFormation Documentation

The Other Related AWS Amazon EMR Resources

Frequently asked questions

What is AWS Amazon EMR Instance Group?

AWS Amazon EMR Instance Group is a resource for Amazon EMR of Amazon Web Service. Settings can be wrote in Terraform and CloudFormation.

Where can I find the example code for the AWS Amazon EMR Instance Group?

For Terraform, the tunnavacher/terraform-aws-emr and niveklabs/aws source code examples are useful. See the Terraform Example section for further details.

For CloudFormation, the dacort/demo-code, tochandrashekhar/EMR-on-EKS and markmester/CloudformationEMR source code examples are useful. See the CloudFormation Example section for further details.

security-icon

Automate config file reviews on your commits

Fix issues in your infrastructure as code with auto-generated patches.

Table of Contents