Top / Microsoft Azure / Azure Data Factory / Pipeline

Azure Data Factory Pipeline

This page shows how to write Terraform and Azure Resource Manager for Data Factory Pipeline and write them securely.

terraform-logo

Review your .tf file for Azure best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

azurerm_data_factory_pipeline (Terraform)

The Pipeline in Data Factory can be configured in Terraform with the resource name azurerm_data_factory_pipeline. The following sections describe 10 examples of how to use the resource and its parameters.

Example Usage from GitHub

azure_pipelines_with_array_parameters.tf#L15
resource "azurerm_data_factory_pipeline" "pl_pipeline_with_array_params_test_01" {
  name                = "pl_pipeline_with_array_params_test_01"
  resource_group_name = var.resource_group_name
  data_factory_name   = var.data_factory_name
  parameters          = var.required_tags
main.tf#L1
resource "azurerm_data_factory_pipeline" "dataFactory_Pipeline" {
  name                = var.pipeline_name
  resource_group_name = var.resource_group_name
  data_factory_name   = var.datafactory_name
  annotations         = var.annotations
}
datafactoryconfig.tf#L1
resource "azurerm_data_factory_pipeline" "example" {
  name                = var.name
  resource_group_name = var.resource_group_name
  data_factory_name   = var.name
  variables = {
    "bob" = "item1"
adf_pipeline.tf#L1
resource "azurerm_data_factory_pipeline" "bbr_adf_pipe" {
  name                = local.adf_pipeline
  resource_group_name = azurerm_resource_group.bbr_adf_rg.name
  data_factory_name   = azurerm_data_factory.bbr_adf.name
  variables = {
    "bob" = "item1"
main.tf#L1
resource "azurerm_data_factory_pipeline" "adfpipe" {
  name                = var.data_factory_pipeline_name
  resource_group_name = var.resource_group_name
  data_factory_name   = var.data_factory_name
pipeline.tf#L1
resource "azurerm_data_factory_pipeline" "final" {
  name                = "final_pipeline_terraform"
  resource_group_name = azurerm_resource_group.final_task.name
  data_factory_name   = azurerm_data_factory.pipeline.name
  activities_json = <<JSON
[
azure_pipelines.tf#L1
resource "azurerm_data_factory_pipeline" "pl_without_parameter_type" {
  name                = "pl_without_parameter_type"
  resource_group_name = var.resource_group_name
  data_factory_name   = var.data_factory_name
  parameters = {
    param1 = "val 1"
module.tf#L1
resource "azurerm_data_factory_pipeline" "pipeline" {
  name                = var.name
  resource_group_name = var.resource_group_name
  data_factory_name   = var.data_factory_name
  description         = try(var.description, null)
  annotations         = try(var.annotations, null)
main.tf#L7
resource "azurerm_data_factory_pipeline" "this" {
  annotations         = var.annotations
  data_factory_name   = var.data_factory_name
  description         = var.description
  name                = var.name
  parameters          = var.parameters
main.tf#L7
resource "azurerm_data_factory_pipeline" "this" {
  annotations         = var.annotations
  data_factory_name   = var.data_factory_name
  description         = var.description
  name                = var.name
  parameters          = var.parameters

Review your Terraform file for Azure best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

>> from Terraform Registry

Explanation in Terraform Registry

Manages a Pipeline inside a Azure Data Factory.

>> from Terraform Registry

Tips: Best Practices for The Other Azure Data Factory Resources

In addition to the azurerm_data_factory, Azure Data Factory has the other resources that should be configured for security reasons. Please check some examples of those resources and precautions.

risk-label

azurerm_data_factory

Ensure to disable public access

It is better to disable public access for Data Factory, which is enabled as default.

Review your Azure Data Factory settings

In addition to the above, there are other security points you should be aware of making sure that your .tf files are protected in Shisho Cloud.

Microsoft.DataFactory/factories/pipelines (Azure Resource Manager)

The factories/pipelines in Microsoft.DataFactory can be configured in Azure Resource Manager with the resource name Microsoft.DataFactory/factories/pipelines. The following sections describe how to use the resource and its parameters.

Example Usage from GitHub

An example could not be found in GitHub.

Parameters

  • apiVersion required - string
  • name required - string

    The pipeline name.

  • properties required
      • activities optional array
          • additionalProperties optional - object

            Unmatched properties from the message are deserialized this collection

          • dependsOn optional array
              • activity required - string

                Activity name.

              • additionalProperties optional - object

                Unmatched properties from the message are deserialized this collection

              • dependencyConditions required - array

                Match-Condition for the dependency.

          • description optional - string

            Activity description.

          • name required - string

            Activity name.

          • userProperties optional array
              • name required - string

                User property name.

              • value required - object

                User property value. Type: string (or Expression with resultType string).

      • annotations optional - array

        List of tags that can be used for describing the Pipeline.

      • concurrency optional - integer

        The max number of concurrent runs for the pipeline.

      • description optional - string

        The description of the pipeline.

      • folder optional
          • name optional - string

            The name of the folder that this Pipeline is in.

      • parameters optional - undefined

        Definition of all parameters for an entity.

      • policy optional
          • elapsedTimeMetric optional
              • duration optional - object

                TimeSpan value, after which an Azure Monitoring Metric is fired.

      • runDimensions optional - object

        Dimensions emitted by Pipeline.

      • variables optional - undefined

        Definition of variable for a Pipeline.

  • type required - string

>> from Azure Resource Manager Documentation

The Other Related Azure Data Factory Resources

Frequently asked questions

What is Azure Data Factory Pipeline?

Azure Data Factory Pipeline is a resource for Data Factory of Microsoft Azure. Settings can be wrote in Terraform.

Where can I find the example code for the Azure Data Factory Pipeline?

For Terraform, the ftylmz1/terraform, prashant101386/devops and peronealex/configdatafactory source code examples are useful. See the Terraform Example section for further details.

security-icon

Automate config file reviews on your commits

Fix issues in your infrastructure as code with auto-generated patches.

Table of Contents