Azure Data Factory Pipeline
This page shows how to write Terraform and Azure Resource Manager for Data Factory Pipeline and write them securely.
azurerm_data_factory_pipeline (Terraform)
The Pipeline in Data Factory can be configured in Terraform with the resource name azurerm_data_factory_pipeline
. The following sections describe 10 examples of how to use the resource and its parameters.
Example Usage from GitHub
resource "azurerm_data_factory_pipeline" "pl_pipeline_with_array_params_test_01" {
name = "pl_pipeline_with_array_params_test_01"
resource_group_name = var.resource_group_name
data_factory_name = var.data_factory_name
parameters = var.required_tags
resource "azurerm_data_factory_pipeline" "dataFactory_Pipeline" {
name = var.pipeline_name
resource_group_name = var.resource_group_name
data_factory_name = var.datafactory_name
annotations = var.annotations
}
resource "azurerm_data_factory_pipeline" "example" {
name = var.name
resource_group_name = var.resource_group_name
data_factory_name = var.name
variables = {
"bob" = "item1"
resource "azurerm_data_factory_pipeline" "bbr_adf_pipe" {
name = local.adf_pipeline
resource_group_name = azurerm_resource_group.bbr_adf_rg.name
data_factory_name = azurerm_data_factory.bbr_adf.name
variables = {
"bob" = "item1"
resource "azurerm_data_factory_pipeline" "adfpipe" {
name = var.data_factory_pipeline_name
resource_group_name = var.resource_group_name
data_factory_name = var.data_factory_name
resource "azurerm_data_factory_pipeline" "final" {
name = "final_pipeline_terraform"
resource_group_name = azurerm_resource_group.final_task.name
data_factory_name = azurerm_data_factory.pipeline.name
activities_json = <<JSON
[
resource "azurerm_data_factory_pipeline" "pl_without_parameter_type" {
name = "pl_without_parameter_type"
resource_group_name = var.resource_group_name
data_factory_name = var.data_factory_name
parameters = {
param1 = "val 1"
resource "azurerm_data_factory_pipeline" "pipeline" {
name = var.name
resource_group_name = var.resource_group_name
data_factory_name = var.data_factory_name
description = try(var.description, null)
annotations = try(var.annotations, null)
resource "azurerm_data_factory_pipeline" "this" {
annotations = var.annotations
data_factory_name = var.data_factory_name
description = var.description
name = var.name
parameters = var.parameters
resource "azurerm_data_factory_pipeline" "this" {
annotations = var.annotations
data_factory_name = var.data_factory_name
description = var.description
name = var.name
parameters = var.parameters
Parameters
-
activities_json
optional - string -
annotations
optional - list of string -
data_factory_name
required - string -
description
optional - string -
id
optional computed - string -
name
required - string -
parameters
optional - map from string to string -
resource_group_name
required - string -
variables
optional - map from string to string -
timeouts
single block
Explanation in Terraform Registry
Manages a Pipeline inside a Azure Data Factory.
Tips: Best Practices for The Other Azure Data Factory Resources
In addition to the azurerm_data_factory, Azure Data Factory has the other resources that should be configured for security reasons. Please check some examples of those resources and precautions.
azurerm_data_factory
Ensure to disable public access
It is better to disable public access for Data Factory, which is enabled as default.
Microsoft.DataFactory/factories/pipelines (Azure Resource Manager)
The factories/pipelines in Microsoft.DataFactory can be configured in Azure Resource Manager with the resource name Microsoft.DataFactory/factories/pipelines
. The following sections describe how to use the resource and its parameters.
Example Usage from GitHub
An example could not be found in GitHub.
Parameters
apiVersion
required - stringname
required - stringThe pipeline name.
properties
requiredactivities
optional arrayadditionalProperties
optional - objectUnmatched properties from the message are deserialized this collection
dependsOn
optional arrayactivity
required - stringActivity name.
additionalProperties
optional - objectUnmatched properties from the message are deserialized this collection
dependencyConditions
required - arrayMatch-Condition for the dependency.
description
optional - stringActivity description.
name
required - stringActivity name.
userProperties
optional arrayname
required - stringUser property name.
value
required - objectUser property value. Type: string (or Expression with resultType string).
annotations
optional - arrayList of tags that can be used for describing the Pipeline.
concurrency
optional - integerThe max number of concurrent runs for the pipeline.
description
optional - stringThe description of the pipeline.
folder
optionalname
optional - stringThe name of the folder that this Pipeline is in.
parameters
optional - undefinedDefinition of all parameters for an entity.
policy
optionalelapsedTimeMetric
optionalduration
optional - objectTimeSpan value, after which an Azure Monitoring Metric is fired.
runDimensions
optional - objectDimensions emitted by Pipeline.
variables
optional - undefinedDefinition of variable for a Pipeline.
type
required - string
Frequently asked questions
What is Azure Data Factory Pipeline?
Azure Data Factory Pipeline is a resource for Data Factory of Microsoft Azure. Settings can be wrote in Terraform.
Where can I find the example code for the Azure Data Factory Pipeline?
For Terraform, the ftylmz1/terraform, prashant101386/devops and peronealex/configdatafactory source code examples are useful. See the Terraform Example section for further details.