Top / Microsoft Azure / Azure HDInsight / Kafka Cluster

Azure HDInsight Kafka Cluster

This page shows how to write Terraform and Azure Resource Manager for HDInsight Kafka Cluster and write them securely.

terraform-logo

Review your .tf file for Azure best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

azurerm_hdinsight_kafka_cluster (Terraform)

The Kafka Cluster in HDInsight can be configured in Terraform with the resource name azurerm_hdinsight_kafka_cluster. The following sections describe 5 examples of how to use the resource and its parameters.

Example Usage from GitHub

hdinsight_kafka_cluster_test.tf#L25
resource "azurerm_hdinsight_kafka_cluster" "standard" {
  name                = "example-hdicluster"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_version     = "4.0"
  tier                = "Standard"
hdinsight_kafka_cluster_test.tf#L25
resource "azurerm_hdinsight_kafka_cluster" "standard" {
  name                = "example-hdicluster"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  cluster_version     = "4.0"
  tier                = "Standard"
main.tf#L7
resource "azurerm_hdinsight_kafka_cluster" "this" {
  cluster_version     = var.cluster_version
  location            = var.location
  name                = var.name
  resource_group_name = var.resource_group_name
  tags                = var.tags
main.tf#L7
resource "azurerm_hdinsight_kafka_cluster" "this" {
  cluster_version     = var.cluster_version
  location            = var.location
  name                = var.name
  resource_group_name = var.resource_group_name
  tags                = var.tags
main.tf#L16
resource "azurerm_hdinsight_kafka_cluster" "this" {
  name                = var.name
  resource_group_name = var.resource_group_name
  location            = var.location
  cluster_version     = var.cluster_version
  tier                = var.tier

Review your Terraform file for Azure best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

>> from Terraform Registry

Explanation in Terraform Registry

Manages a HDInsight Kafka Cluster.

>> from Terraform Registry

Microsoft.HDInsight/clusters (Azure Resource Manager)

The clusters in Microsoft.HDInsight can be configured in Azure Resource Manager with the resource name Microsoft.HDInsight/clusters. The following sections describe how to use the resource and its parameters.

Example Usage from GitHub

GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClustersInResourceGroup.json#L13
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
GetLinuxHadoopAllClusters.json#L12
                         "type": "Microsoft.HDInsight/clusters",
                         "location": "West US",
                         "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
                         "tags": { "key1": "val1" },
                         "properties": {
                             "clusterVersion": "3.5.1000.0",
mainTemplate.json
{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "clusterName": {
GetLinuxHadoopAllClusters.json#L13
            "type": "Microsoft.HDInsight/clusters",
            "location": "West US",
            "etag": "f0212a39-b827-45e0-9ffa-4f5232e58851",
            "tags": {
              "key1": "val1"
            },

Parameters

  • apiVersion required - string
  • identity optional
      • type optional - string

        The type of identity used for the cluster. The type 'SystemAssigned, UserAssigned' includes both an implicitly created identity and a set of user assigned identities.

      • userAssignedIdentities optional - undefined

        The list of user identities associated with the cluster. The user identity dictionary key references will be ARM resource ids in the form: '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{identityName}'.

  • location optional - string

    The location of the cluster.

  • name required - string

    The name of the cluster.

  • properties required
      • clusterDefinition optional
          • blueprint optional - string

            The link to the blueprint.

          • componentVersion optional - string

            The versions of different services in the cluster.

          • configurations optional - object

            The cluster configurations.

          • kind optional - string

            The type of cluster.

      • clusterVersion optional - string

        The version of the cluster.

      • computeIsolationProperties optional
          • enableComputeIsolation optional - boolean

            The flag indicates whether enable compute isolation or not.

          • hostSku optional - string

            The host sku.

      • computeProfile optional
          • roles optional array
              • autoscale optional
                  • capacity optional
                      • maxInstanceCount optional - integer

                        The maximum instance count of the cluster

                      • minInstanceCount optional - integer

                        The minimum instance count of the cluster

                  • recurrence optional
                      • schedule optional array
                          • days optional - array

                            Days of the week for a schedule-based autoscale rule

                          • timeAndCapacity optional
                              • maxInstanceCount optional - integer

                                The maximum instance count of the cluster

                              • minInstanceCount optional - integer

                                The minimum instance count of the cluster

                              • time optional - string

                                24-hour time in the form xx:xx

                      • timeZone optional - string

                        The time zone for the autoscale schedule times

              • dataDisksGroups optional array
                  • disksPerNode optional - integer

                    The number of disks per node.

              • encryptDataDisks optional - boolean

                Indicates whether encrypt the data disks.

              • hardwareProfile optional
                  • vmSize optional - string

                    The size of the VM

              • minInstanceCount optional - integer

                The minimum instance count of the cluster.

              • name optional - string

                The name of the role.

              • osProfile optional
                  • linuxOperatingSystemProfile optional
                      • password optional - string

                        The password.

                      • sshProfile optional
                          • publicKeys optional array
                              • certificateData optional - string

                                The certificate for SSH.

                      • username optional - string

                        The username.

              • scriptActions optional array
                  • name required - string

                    The name of the script action.

                  • parameters required - string

                    The parameters for the script provided.

                  • uri required - string

                    The URI to the script.

              • targetInstanceCount optional - integer

                The instance count of the cluster.

              • virtualNetworkProfile optional
                  • id optional - string

                    The ID of the virtual network.

                  • subnet optional - string

                    The name of the subnet.

              • VMGroupName optional - string

                The name of the virtual machine group.

      • diskEncryptionProperties optional
          • encryptionAlgorithm optional - string

            Algorithm identifier for encryption, default RSA-OAEP.

          • encryptionAtHost optional - boolean

            Indicates whether or not resource disk encryption is enabled.

          • keyName optional - string

            Key name that is used for enabling disk encryption.

          • keyVersion optional - string

            Specific key version that is used for enabling disk encryption.

          • msiResourceId optional - string

            Resource ID of Managed Identity that is used to access the key vault.

          • vaultUri optional - string

            Base key vault URI where the customers key is located eg. https://myvault.vault.azure.net

      • encryptionInTransitProperties optional
          • isEncryptionInTransitEnabled optional - boolean

            Indicates whether or not inter cluster node communication is encrypted in transit.

      • kafkaRestProperties optional
          • clientGroupInfo optional
              • groupId optional - string

                The AAD security group id.

              • groupName optional - string

                The AAD security group name.

          • configurationOverride optional - string

            The configurations that need to be overriden.

      • minSupportedTlsVersion optional - string

        The minimal supported tls version.

      • networkProperties optional
          • privateLink optional - string

            Indicates whether or not private link is enabled.

          • resourceProviderConnection optional - string

            The direction for the resource provider connection.

      • osType optional - string

        The type of operating system.

      • privateLinkConfigurations optional array
          • name required - string

            The name of private link configuration.

          • properties required
              • groupId required - string

                The HDInsight private linkable sub-resource name to apply the private link configuration to. For example, 'headnode', 'gateway', 'edgenode'.

              • ipConfigurations required array
                  • name required - string

                    The name of private link IP configuration.

                  • properties optional
                      • primary optional - boolean

                        Indicates whether this IP configuration is primary for the corresponding NIC.

                      • privateIPAddress optional - string

                        The IP address.

                      • privateIPAllocationMethod optional - string

                        The method that private IP address is allocated.

                      • subnet optional
                          • id optional - string

                            The azure resource id.

      • securityProfile optional
          • aaddsResourceId optional - string

            The resource ID of the user's Azure Active Directory Domain Service.

          • clusterUsersGroupDNs optional - array

            Optional. The Distinguished Names for cluster user groups

          • directoryType optional - string

            The directory type.

          • domain optional - string

            The organization's active directory domain.

          • domainUsername optional - string

            The domain user account that will have admin privileges on the cluster.

          • domainUserPassword optional - string

            The domain admin password.

          • ldapsUrls optional - array

            The LDAPS protocol URLs to communicate with the Active Directory.

          • msiResourceId optional - string

            User assigned identity that has permissions to read and create cluster-related artifacts in the user's AADDS.

          • organizationalUnitDN optional - string

            The organizational unit within the Active Directory to place the cluster and service accounts.

      • storageProfile optional
          • storageaccounts optional array
              • container optional - string

                The container in the storage account, only to be specified for WASB storage accounts.

              • fileshare optional - string

                The file share name.

              • fileSystem optional - string

                The filesystem, only to be specified for Azure Data Lake Storage Gen 2.

              • isDefault optional - boolean

                Whether or not the storage account is the default storage account.

              • key optional - string

                The storage account access key.

              • msiResourceId optional - string

                The managed identity (MSI) that is allowed to access the storage account, only to be specified for Azure Data Lake Storage Gen 2.

              • name optional - string

                The name of the storage account.

              • resourceId optional - string

                The resource ID of storage account, only to be specified for Azure Data Lake Storage Gen 2.

              • saskey optional - string

                The shared access signature key.

      • tier optional - string

        The cluster tier.

  • tags optional - string

    The resource tags.

  • type required - string
  • zones optional - array

    The availability zones.

>> from Azure Resource Manager Documentation

The Other Related Azure HDInsight Resources

Frequently asked questions

What is Azure HDInsight Kafka Cluster?

Azure HDInsight Kafka Cluster is a resource for HDInsight of Microsoft Azure. Settings can be wrote in Terraform.

Where can I find the example code for the Azure HDInsight Kafka Cluster?

For Terraform, the gilyas/infracost, infracost/infracost and kevinhead/azurerm source code examples are useful. See the Terraform Example section for further details.

For Azure Resource Manager, the assing/alerts-extension, assing/alerts-extension and assing/alerts-extension source code examples are useful. See the Azure Resource Manager Example section for further details.

security-icon

Automate config file reviews on your commits

Fix issues in your infrastructure as code with auto-generated patches.

Table of Contents