Azure HDInsight Kafka Cluster

This page shows how to write Terraform and Azure Resource Manager for HDInsight Kafka Cluster and write them securely.

azurerm_hdinsight_kafka_cluster (Terraform)

The Kafka Cluster in HDInsight can be configured in Terraform with the resource name azurerm_hdinsight_kafka_cluster. The following sections describe 5 examples of how to use the resource and its parameters.

Example Usage from GitHub

hdinsight_kafka_cluster_test.tf#L25
resource "azurerm_hdinsight_kafka_cluster" "standard" {
  name                = "example-hdicluster"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_version     = "4.0"
  tier                = "Standard"
hdinsight_kafka_cluster_test.tf#L25
resource "azurerm_hdinsight_kafka_cluster" "standard" {
  name                = "example-hdicluster"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_version     = "4.0"
  tier                = "Standard"
main.tf#L7
resource "azurerm_hdinsight_kafka_cluster" "this" {
  cluster_version     = var.cluster_version
  location            = var.location
  name                = var.name
  resource_group_name = var.resource_group_name
  tags                = var.tags
main.tf#L7
resource "azurerm_hdinsight_kafka_cluster" "this" {
  cluster_version     = var.cluster_version
  location            = var.location
  name                = var.name
  resource_group_name = var.resource_group_name
  tags                = var.tags
main.tf#L16
resource "azurerm_hdinsight_kafka_cluster" "this" {
  name                = var.name
  resource_group_name = var.resource_group_name
  location            = var.location
  cluster_version     = var.cluster_version
  tier                = var.tier

Review your Terraform file for Azure best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

Explanation in Terraform Registry

Manages a HDInsight Kafka Cluster.

Microsoft.HDInsight/clusters (Azure Resource Manager)

The clusters in Microsoft.HDInsight can be configured in Azure Resource Manager with the resource name Microsoft.HDInsight/clusters. The following sections describe how to use the resource and its parameters.

Example Usage from GitHub

GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
mainTemplate.json
{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "clusterName": {
GetLinuxHadoopAllClusters.json#L13
            "type": "Microsoft.HDInsight/clusters",
            "location": "West US",
            "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
            "tags": {
              "key1": "val1"
            },

Parameters

  • apiVersion required - string
  • identity optional
      • type optional - string

        The type of identity used for the cluster. The type 'SystemAssigned, UserAssigned' includes both an implicitly created identity and a set of user assigned identities.

      • userAssignedIdentities optional - undefined

        The list of user identities associated with the cluster. The user identity dictionary key references will be ARM resource ids in the form: '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}'.

  • location optional - string

    The location of the cluster.

  • name required - string

    The name of the cluster.

  • properties required
      • clusterDefinition optional
          • blueprint optional - string

            The link to the blueprint.

          • componentVersion optional - string

            The versions of different services in the cluster.

          • configurations optional - object

            The cluster configurations.

          • kind optional - string

            The type of cluster.

      • clusterVersion optional - string

        The version of the cluster.

      • computeIsolationProperties optional
          • enableComputeIsolation optional - boolean

            The flag indicates whether enable compute isolation or not.

          • hostSku optional - string

            The host sku.

      • computeProfile optional
          • roles optional array
              • autoscale optional
                  • capacity optional
                      • maxInstanceCount optional - integer

                        The maximum instance count of the cluster

                      • minInstanceCount optional - integer

                        The minimum instance count of the cluster

                  • recurrence optional
                      • schedule optional array
                          • days optional - array

                            Days of the week for a schedule-based autoscale rule

                          • timeAndCapacity optional
                              • maxInstanceCount optional - integer

                                The maximum instance count of the cluster

                              • minInstanceCount optional - integer

                                The minimum instance count of the cluster

                              • time optional - string

                                24-hour time in the form xx:xx

                      • timeZone optional - string

                        The time zone for the autoscale schedule times

              • dataDisksGroups optional array
                  • disksPerNode optional - integer

                    The number of disks per node.

              • encryptDataDisks optional - boolean

                Indicates whether encrypt the data disks.

              • hardwareProfile optional
                  • vmSize optional - string

                    The size of the VM

              • minInstanceCount optional - integer

                The minimum instance count of the cluster.

              • name optional - string

                The name of the role.

              • osProfile optional
                  • linuxOperatingSystemProfile optional
                      • password optional - string

                        The password.

                      • sshProfile optional
                          • publicKeys optional array
                              • certificateData optional - string

                                The certificate for SSH.

                      • username optional - string

                        The username.

              • scriptActions optional array
                  • name required - string

                    The name of the script action.

                  • parameters required - string

                    The parameters for the script provided.

                  • uri required - string

                    The URI to the script.

              • targetInstanceCount optional - integer

                The instance count of the cluster.

              • virtualNetworkProfile optional
                  • id optional - string

                    The ID of the virtual network.

                  • subnet optional - string

                    The name of the subnet.

              • VMGroupName optional - string

                The name of the virtual machine group.

      • diskEncryptionProperties optional
          • encryptionAlgorithm optional - string

            Algorithm identifier for encryption, default RSA-OAEP.

          • encryptionAtHost optional - boolean

            Indicates whether or not resource disk encryption is enabled.

          • keyName optional - string

            Key name that is used for enabling disk encryption.

          • keyVersion optional - string

            Specific key version that is used for enabling disk encryption.

          • msiResourceId optional - string

            Resource ID of Managed Identity that is used to access the key vault.

          • vaultUri optional - string

            Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net

      • encryptionInTransitProperties optional
          • isEncryptionInTransitEnabled optional - boolean

            Indicates whether or not inter cluster node communication is encrypted in transit.

      • kafkaRestProperties optional
          • clientGroupInfo optional
              • groupId optional - string

                The AAD security group id.

              • groupName optional - string

                The AAD security group name.

          • configurationOverride optional - string

            The configurations that need to be overriden.

      • minSupportedTlsVersion optional - string

        The minimal supported tls version.

      • networkProperties optional
          • privateLink optional - string

            Indicates whether or not private link is enabled.

          • resourceProviderConnection optional - string

            The direction for the resource provider connection.

      • osType optional - string

        The type of operating system.

      • privateLinkConfigurations optional array
          • name required - string

            The name of private link configuration.

          • properties required
              • groupId required - string

                The HDInsight private linkable sub-resource name to apply the private link configuration to. For example, 'headnode', 'gateway', 'edgenode'.

              • ipConfigurations required array
                  • name required - string

                    The name of private link IP configuration.

                  • properties optional
                      • primary optional - boolean

                        Indicates whether this IP configuration is primary for the corresponding NIC.

                      • privateIPAddress optional - string

                        The IP address.

                      • privateIPAllocationMethod optional - string

                        The method that private IP address is allocated.

                      • subnet optional
                          • id optional - string

                            The azure resource id.

      • securityProfile optional
          • aaddsResourceId optional - string

            The resource ID of the user's Azure Active Directory Domain Service.

          • clusterUsersGroupDNs optional - array

            Optional. The Distinguished Names for cluster user groups

          • directoryType optional - string

            The directory type.

          • domain optional - string

            The organization's active directory domain.

          • domainUsername optional - string

            The domain user account that will have admin privileges on the cluster.

          • domainUserPassword optional - string

            The domain admin password.

          • ldapsUrls optional - array

            The LDAPS protocol URLs to communicate with the Active Directory.

          • msiResourceId optional - string

            User assigned identity that has permissions to read and create cluster-related artifacts in the user's AADDS.

          • organizationalUnitDN optional - string

            The organizational unit within the Active Directory to place the cluster and service accounts.

      • storageProfile optional
          • storageaccounts optional array
              • container optional - string

                The container in the storage account, only to be specified for WASB storage accounts.

              • fileshare optional - string

                The file share name.

              • fileSystem optional - string

                The filesystem, only to be specified for Azure Data Lake Storage Gen 2.

              • isDefault optional - boolean

                Whether or not the storage account is the default storage account.

              • key optional - string

                The storage account access key.

              • msiResourceId optional - string

                The managed identity (MSI) that is allowed to access the storage account, only to be specified for Azure Data Lake Storage Gen 2.

              • name optional - string

                The name of the storage account.

              • resourceId optional - string

                The resource ID of storage account, only to be specified for Azure Data Lake Storage Gen 2.

              • saskey optional - string

                The shared access signature key.

      • tier optional - string

        The cluster tier.

  • tags optional - string

    The resource tags.

  • type required - string
  • zones optional - array

    The availability zones.

Frequently asked questions

What is Azure HDInsight Kafka Cluster?

Azure HDInsight Kafka Cluster is a resource for HDInsight of Microsoft Azure. Settings can be wrote in Terraform.

Where can I find the example code for the Azure HDInsight Kafka Cluster?

For Terraform, the gilyas/infracost, infracost/infracost and kevinhead/azurerm source code examples are useful. See the Terraform Example section for further details.

For Azure Resource Manager, the assing/alerts-extension, assing/alerts-extension and assing/alerts-extension source code examples are useful. See the Azure Resource Manager Example section for further details.