AWS Glue Crawler

This page shows how to write Terraform and CloudFormation for AWS Glue Crawler and write them securely.

aws_glue_crawler (Terraform)

The Crawler in AWS Glue can be configured in Terraform with the resource name aws_glue_crawler. The following sections describe how to use the resource and its parameters.

Example Usage from GitHub

An example could not be found in GitHub.

Review your Terraform file for AWS best practices

Shisho Cloud, our free checker to make sure your Terraform configuration follows best practices, is available (beta).

Parameters

Explanation in Terraform Registry

Manages a Glue Crawler. More information can be found in the AWS Glue Developer Guide

AWS::Glue::Crawler (CloudFormation)

The Crawler in Glue can be configured in CloudFormation with the resource name AWS::Glue::Crawler. The following sections describe 10 examples of how to use the resource and its parameters.

Example Usage from GitHub

glue-workflow-stack.yml#L351
    Type: AWS::Glue::Crawler
    DependsOn: GlueRole
    Properties:
      Name: c_aggregated
      Description: !Sub Crawl aggregated datasets at s3://${Covid19Bucket}/covid19/world-cases-deaths-aggregates/
      DatabaseName: !Ref GlueDatabaseName
glue-workflow-stack.yml#L351
    Type: AWS::Glue::Crawler
    DependsOn: GlueRole
    Properties:
      Name: c_aggregated
      Description: !Sub Crawl aggregated datasets at s3://${Covid19Bucket}/covid19/world-cases-deaths-aggregates/
      DatabaseName: !Ref GlueDatabaseName
tvdata-raw-crawler.yml#L3
    Type: AWS::Glue::Crawler
    Properties:
      Name: ${self:custom.stage}-tvdata-raw-crawler
      Role:
        Fn::GetAtt: [RatingRole, Arn]
      DatabaseName:
smart-hub-athena-glue.yml#L255
    Type: AWS::Glue::Crawler
    Properties:
      Name: smart-hub-locations-csv
      Role: !GetAtt "CrawlerRole.Arn"
      Targets:
        CatalogTargets:
glue.yml#L23
    Type: "AWS::Glue::Crawler"
    Properties:
      Name: "meter-data-business-aggregated-daily"
      Role: !Sub "service-role/${IAMRole}"
      Targets:
        S3Targets:
glue_cf_template.json#L83
            "Type": "AWS::Glue::Crawler",
            "Properties": {
                "Name": {"Fn::Sub": "${AWS::StackName}-views-crawler"},
                "Role": {"Ref": "glueRole"},
                "DatabaseName": {
                    "Ref": "viewDatabase"
aws_glue.json#L4
    "path": "/ResourceTypes/AWS::Glue::Crawler/Properties/Role/Value",
    "value": {
      "ValueType": "AWS::IAM::Role.NameOrArn"
    }
  },
  {
aws_glue.json#L4
    "path": "/ResourceTypes/AWS::Glue::Crawler/Properties/Role/Value",
    "value": {
      "ValueType": "AWS::IAM::Role.NameOrArn"
    }
  },
  {
glue.json#L204
            "Type": "AWS::Glue::Crawler",
            "Properties": {
                "Role": {
                    "Ref": "AWSGlueCuratedDatasetsCrawlerRoleName"
                },
                "DatabaseName": {
twitter_analytics.json#L102
      "Type": "AWS::Glue::Crawler",
      "Properties": {
        "Name": "raw_crawler",
        "Role": {
          "Fn::GetAtt": [
            "GlueRole",

Parameters

Explanation in CloudFormation Registry

The AWS::Glue::Crawler resource specifies an AWS Glue crawler. For more information, see Cataloging Tables with a Crawler and Crawler Structure in the AWS Glue Developer Guide.

Frequently asked questions

What is AWS Glue Crawler?

AWS Glue Crawler is a resource for Glue of Amazon Web Service. Settings can be wrote in Terraform and CloudFormation.

Where can I find the example code for the AWS Glue Crawler?

For CloudFormation, the GirijaRaniGavara/provision-codepipeline-glue-workflows-, duyhoang15/test and ozzyince/tv source code examples are useful. See the CloudFormation Example section for further details.