AWS Glue

2018/07/10 - AWS Glue - 6 updated api methods

Changes  Update glue client to latest version

CreateCrawler (updated) Link ¶
Changes (request) 
{'Targets': {'DynamoDBTargets': [{'Path': 'string'}]}}

Creates a new crawler with specified targets, role, configuration, and optional schedule. At least one crawl target must be specified, in the s3Targets field, the jdbcTargets field, or the DynamoDBTargets field.

See also: AWS API Documentation

Request Syntax

client.create_crawler(
    Name='string',
    Role='string',
    DatabaseName='string',
    Description='string',
    Targets={
        'S3Targets': [
            {
                'Path': 'string',
                'Exclusions': [
                    'string',
                ]
            },
        ],
        'JdbcTargets': [
            {
                'ConnectionName': 'string',
                'Path': 'string',
                'Exclusions': [
                    'string',
                ]
            },
        ],
        'DynamoDBTargets': [
            {
                'Path': 'string'
            },
        ]
    },
    Schedule='string',
    Classifiers=[
        'string',
    ],
    TablePrefix='string',
    SchemaChangePolicy={
        'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
        'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
    },
    Configuration='string'
)
type Name:

string

param Name:

[REQUIRED]

Name of the new crawler.

type Role:

string

param Role:

[REQUIRED]

The IAM role (or ARN of an IAM role) used by the new crawler to access customer resources.

type DatabaseName:

string

param DatabaseName:

[REQUIRED]

The AWS Glue database where results are written, such as: arn:aws:daylight:us-east-1::database/sometable/*.

type Description:

string

param Description:

A description of the new crawler.

type Targets:

dict

param Targets:

[REQUIRED]

A list of collection of targets to crawl.

  • S3Targets (list) --

    Specifies Amazon S3 targets.

    • (dict) --

      Specifies a data store in Amazon S3.

      • Path (string) --

        The path to the Amazon S3 target.

      • Exclusions (list) --

        A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

        • (string) --

  • JdbcTargets (list) --

    Specifies JDBC targets.

    • (dict) --

      Specifies a JDBC data store to crawl.

      • ConnectionName (string) --

        The name of the connection to use to connect to the JDBC target.

      • Path (string) --

        The path of the JDBC target.

      • Exclusions (list) --

        A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

        • (string) --

  • DynamoDBTargets (list) --

    Specifies DynamoDB targets.

    • (dict) --

      Specifies a DynamoDB table to crawl.

      • Path (string) --

        The name of the DynamoDB table to crawl.

type Schedule:

string

param Schedule:

A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).

type Classifiers:

list

param Classifiers:

A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.

  • (string) --

type TablePrefix:

string

param TablePrefix:

The table prefix used for catalog tables that are created.

type SchemaChangePolicy:

dict

param SchemaChangePolicy:

Policy for the crawler's update and deletion behavior.

  • UpdateBehavior (string) --

    The update behavior when the crawler finds a changed schema.

  • DeleteBehavior (string) --

    The deletion behavior when the crawler finds a deleted object.

type Configuration:

string

param Configuration:

Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

  • (dict) --

GetCrawler (updated) Link ¶
Changes (response) 
{'Crawler': {'Targets': {'DynamoDBTargets': [{'Path': 'string'}]}}}

Retrieves metadata for a specified crawler.

See also: AWS API Documentation

Request Syntax

client.get_crawler(
    Name='string'
)
type Name:

string

param Name:

[REQUIRED]

Name of the crawler to retrieve metadata for.

rtype:

dict

returns:

Response Syntax

{
    'Crawler': {
        'Name': 'string',
        'Role': 'string',
        'Targets': {
            'S3Targets': [
                {
                    'Path': 'string',
                    'Exclusions': [
                        'string',
                    ]
                },
            ],
            'JdbcTargets': [
                {
                    'ConnectionName': 'string',
                    'Path': 'string',
                    'Exclusions': [
                        'string',
                    ]
                },
            ],
            'DynamoDBTargets': [
                {
                    'Path': 'string'
                },
            ]
        },
        'DatabaseName': 'string',
        'Description': 'string',
        'Classifiers': [
            'string',
        ],
        'SchemaChangePolicy': {
            'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
            'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
        },
        'State': 'READY'|'RUNNING'|'STOPPING',
        'TablePrefix': 'string',
        'Schedule': {
            'ScheduleExpression': 'string',
            'State': 'SCHEDULED'|'NOT_SCHEDULED'|'TRANSITIONING'
        },
        'CrawlElapsedTime': 123,
        'CreationTime': datetime(2015, 1, 1),
        'LastUpdated': datetime(2015, 1, 1),
        'LastCrawl': {
            'Status': 'SUCCEEDED'|'CANCELLED'|'FAILED',
            'ErrorMessage': 'string',
            'LogGroup': 'string',
            'LogStream': 'string',
            'MessagePrefix': 'string',
            'StartTime': datetime(2015, 1, 1)
        },
        'Version': 123,
        'Configuration': 'string'
    }
}

Response Structure

  • (dict) --

    • Crawler (dict) --

      The metadata for the specified crawler.

      • Name (string) --

        The crawler name.

      • Role (string) --

        The IAM role (or ARN of an IAM role) used to access customer resources, such as data in Amazon S3.

      • Targets (dict) --

        A collection of targets to crawl.

        • S3Targets (list) --

          Specifies Amazon S3 targets.

          • (dict) --

            Specifies a data store in Amazon S3.

            • Path (string) --

              The path to the Amazon S3 target.

            • Exclusions (list) --

              A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

              • (string) --

        • JdbcTargets (list) --

          Specifies JDBC targets.

          • (dict) --

            Specifies a JDBC data store to crawl.

            • ConnectionName (string) --

              The name of the connection to use to connect to the JDBC target.

            • Path (string) --

              The path of the JDBC target.

            • Exclusions (list) --

              A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

              • (string) --

        • DynamoDBTargets (list) --

          Specifies DynamoDB targets.

          • (dict) --

            Specifies a DynamoDB table to crawl.

            • Path (string) --

              The name of the DynamoDB table to crawl.

      • DatabaseName (string) --

        The database where metadata is written by this crawler.

      • Description (string) --

        A description of the crawler.

      • Classifiers (list) --

        A list of custom classifiers associated with the crawler.

        • (string) --

      • SchemaChangePolicy (dict) --

        Sets the behavior when the crawler finds a changed or deleted object.

        • UpdateBehavior (string) --

          The update behavior when the crawler finds a changed schema.

        • DeleteBehavior (string) --

          The deletion behavior when the crawler finds a deleted object.

      • State (string) --

        Indicates whether the crawler is running, or whether a run is pending.

      • TablePrefix (string) --

        The prefix added to the names of tables that are created.

      • Schedule (dict) --

        For scheduled crawlers, the schedule when the crawler runs.

        • ScheduleExpression (string) --

          A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).

        • State (string) --

          The state of the schedule.

      • CrawlElapsedTime (integer) --

        If the crawler is running, contains the total time elapsed since the last crawl began.

      • CreationTime (datetime) --

        The time when the crawler was created.

      • LastUpdated (datetime) --

        The time the crawler was last updated.

      • LastCrawl (dict) --

        The status of the last crawl, and potentially error information if an error occurred.

        • Status (string) --

          Status of the last crawl.

        • ErrorMessage (string) --

          If an error occurred, the error information about the last crawl.

        • LogGroup (string) --

          The log group for the last crawl.

        • LogStream (string) --

          The log stream for the last crawl.

        • MessagePrefix (string) --

          The prefix for a message about this crawl.

        • StartTime (datetime) --

          The time at which the crawl started.

      • Version (integer) --

        The version of the crawler.

      • Configuration (string) --

        Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.

GetCrawlers (updated) Link ¶
Changes (response) 
{'Crawlers': {'Targets': {'DynamoDBTargets': [{'Path': 'string'}]}}}

Retrieves metadata for all crawlers defined in the customer account.

See also: AWS API Documentation

Request Syntax

client.get_crawlers(
    MaxResults=123,
    NextToken='string'
)
type MaxResults:

integer

param MaxResults:

The number of crawlers to return on each call.

type NextToken:

string

param NextToken:

A continuation token, if this is a continuation request.

rtype:

dict

returns:

Response Syntax

{
    'Crawlers': [
        {
            'Name': 'string',
            'Role': 'string',
            'Targets': {
                'S3Targets': [
                    {
                        'Path': 'string',
                        'Exclusions': [
                            'string',
                        ]
                    },
                ],
                'JdbcTargets': [
                    {
                        'ConnectionName': 'string',
                        'Path': 'string',
                        'Exclusions': [
                            'string',
                        ]
                    },
                ],
                'DynamoDBTargets': [
                    {
                        'Path': 'string'
                    },
                ]
            },
            'DatabaseName': 'string',
            'Description': 'string',
            'Classifiers': [
                'string',
            ],
            'SchemaChangePolicy': {
                'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
                'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
            },
            'State': 'READY'|'RUNNING'|'STOPPING',
            'TablePrefix': 'string',
            'Schedule': {
                'ScheduleExpression': 'string',
                'State': 'SCHEDULED'|'NOT_SCHEDULED'|'TRANSITIONING'
            },
            'CrawlElapsedTime': 123,
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'LastCrawl': {
                'Status': 'SUCCEEDED'|'CANCELLED'|'FAILED',
                'ErrorMessage': 'string',
                'LogGroup': 'string',
                'LogStream': 'string',
                'MessagePrefix': 'string',
                'StartTime': datetime(2015, 1, 1)
            },
            'Version': 123,
            'Configuration': 'string'
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • Crawlers (list) --

      A list of crawler metadata.

      • (dict) --

        Specifies a crawler program that examines a data source and uses classifiers to try to determine its schema. If successful, the crawler records metadata concerning the data source in the AWS Glue Data Catalog.

        • Name (string) --

          The crawler name.

        • Role (string) --

          The IAM role (or ARN of an IAM role) used to access customer resources, such as data in Amazon S3.

        • Targets (dict) --

          A collection of targets to crawl.

          • S3Targets (list) --

            Specifies Amazon S3 targets.

            • (dict) --

              Specifies a data store in Amazon S3.

              • Path (string) --

                The path to the Amazon S3 target.

              • Exclusions (list) --

                A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

                • (string) --

          • JdbcTargets (list) --

            Specifies JDBC targets.

            • (dict) --

              Specifies a JDBC data store to crawl.

              • ConnectionName (string) --

                The name of the connection to use to connect to the JDBC target.

              • Path (string) --

                The path of the JDBC target.

              • Exclusions (list) --

                A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

                • (string) --

          • DynamoDBTargets (list) --

            Specifies DynamoDB targets.

            • (dict) --

              Specifies a DynamoDB table to crawl.

              • Path (string) --

                The name of the DynamoDB table to crawl.

        • DatabaseName (string) --

          The database where metadata is written by this crawler.

        • Description (string) --

          A description of the crawler.

        • Classifiers (list) --

          A list of custom classifiers associated with the crawler.

          • (string) --

        • SchemaChangePolicy (dict) --

          Sets the behavior when the crawler finds a changed or deleted object.

          • UpdateBehavior (string) --

            The update behavior when the crawler finds a changed schema.

          • DeleteBehavior (string) --

            The deletion behavior when the crawler finds a deleted object.

        • State (string) --

          Indicates whether the crawler is running, or whether a run is pending.

        • TablePrefix (string) --

          The prefix added to the names of tables that are created.

        • Schedule (dict) --

          For scheduled crawlers, the schedule when the crawler runs.

          • ScheduleExpression (string) --

            A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).

          • State (string) --

            The state of the schedule.

        • CrawlElapsedTime (integer) --

          If the crawler is running, contains the total time elapsed since the last crawl began.

        • CreationTime (datetime) --

          The time when the crawler was created.

        • LastUpdated (datetime) --

          The time the crawler was last updated.

        • LastCrawl (dict) --

          The status of the last crawl, and potentially error information if an error occurred.

          • Status (string) --

            Status of the last crawl.

          • ErrorMessage (string) --

            If an error occurred, the error information about the last crawl.

          • LogGroup (string) --

            The log group for the last crawl.

          • LogStream (string) --

            The log stream for the last crawl.

          • MessagePrefix (string) --

            The prefix for a message about this crawl.

          • StartTime (datetime) --

            The time at which the crawl started.

        • Version (integer) --

          The version of the crawler.

        • Configuration (string) --

          Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.

    • NextToken (string) --

      A continuation token, if the returned list has not reached the end of those defined in this customer account.

GetMapping (updated) Link ¶
Changes (request) 
{'Location': {'DynamoDB': [{'Name': 'string',
                            'Param': 'boolean',
                            'Value': 'string'}]}}

Creates mappings.

See also: AWS API Documentation

Request Syntax

client.get_mapping(
    Source={
        'DatabaseName': 'string',
        'TableName': 'string'
    },
    Sinks=[
        {
            'DatabaseName': 'string',
            'TableName': 'string'
        },
    ],
    Location={
        'Jdbc': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ],
        'S3': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ],
        'DynamoDB': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ]
    }
)
type Source:

dict

param Source:

[REQUIRED]

Specifies the source table.

  • DatabaseName (string) -- [REQUIRED]

    The database in which the table metadata resides.

  • TableName (string) -- [REQUIRED]

    The name of the table in question.

type Sinks:

list

param Sinks:

A list of target tables.

  • (dict) --

    Specifies a table definition in the Data Catalog.

    • DatabaseName (string) -- [REQUIRED]

      The database in which the table metadata resides.

    • TableName (string) -- [REQUIRED]

      The name of the table in question.

type Location:

dict

param Location:

Parameters for the mapping.

  • Jdbc (list) --

    A JDBC location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

  • S3 (list) --

    An Amazon S3 location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

  • DynamoDB (list) --

    A DynamoDB Table location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

rtype:

dict

returns:

Response Syntax

{
    'Mapping': [
        {
            'SourceTable': 'string',
            'SourcePath': 'string',
            'SourceType': 'string',
            'TargetTable': 'string',
            'TargetPath': 'string',
            'TargetType': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • Mapping (list) --

      A list of mappings to the specified targets.

      • (dict) --

        Defines a mapping.

        • SourceTable (string) --

          The name of the source table.

        • SourcePath (string) --

          The source path.

        • SourceType (string) --

          The source type.

        • TargetTable (string) --

          The target table.

        • TargetPath (string) --

          The target path.

        • TargetType (string) --

          The target type.

GetPlan (updated) Link ¶
Changes (request) 
{'Location': {'DynamoDB': [{'Name': 'string',
                            'Param': 'boolean',
                            'Value': 'string'}]}}

Gets code to perform a specified mapping.

See also: AWS API Documentation

Request Syntax

client.get_plan(
    Mapping=[
        {
            'SourceTable': 'string',
            'SourcePath': 'string',
            'SourceType': 'string',
            'TargetTable': 'string',
            'TargetPath': 'string',
            'TargetType': 'string'
        },
    ],
    Source={
        'DatabaseName': 'string',
        'TableName': 'string'
    },
    Sinks=[
        {
            'DatabaseName': 'string',
            'TableName': 'string'
        },
    ],
    Location={
        'Jdbc': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ],
        'S3': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ],
        'DynamoDB': [
            {
                'Name': 'string',
                'Value': 'string',
                'Param': True|False
            },
        ]
    },
    Language='PYTHON'|'SCALA'
)
type Mapping:

list

param Mapping:

[REQUIRED]

The list of mappings from a source table to target tables.

  • (dict) --

    Defines a mapping.

    • SourceTable (string) --

      The name of the source table.

    • SourcePath (string) --

      The source path.

    • SourceType (string) --

      The source type.

    • TargetTable (string) --

      The target table.

    • TargetPath (string) --

      The target path.

    • TargetType (string) --

      The target type.

type Source:

dict

param Source:

[REQUIRED]

The source table.

  • DatabaseName (string) -- [REQUIRED]

    The database in which the table metadata resides.

  • TableName (string) -- [REQUIRED]

    The name of the table in question.

type Sinks:

list

param Sinks:

The target tables.

  • (dict) --

    Specifies a table definition in the Data Catalog.

    • DatabaseName (string) -- [REQUIRED]

      The database in which the table metadata resides.

    • TableName (string) -- [REQUIRED]

      The name of the table in question.

type Location:

dict

param Location:

Parameters for the mapping.

  • Jdbc (list) --

    A JDBC location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

  • S3 (list) --

    An Amazon S3 location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

  • DynamoDB (list) --

    A DynamoDB Table location.

    • (dict) --

      An argument or property of a node.

      • Name (string) -- [REQUIRED]

        The name of the argument or property.

      • Value (string) -- [REQUIRED]

        The value of the argument or property.

      • Param (boolean) --

        True if the value is used as a parameter.

type Language:

string

param Language:

The programming language of the code to perform the mapping.

rtype:

dict

returns:

Response Syntax

{
    'PythonScript': 'string',
    'ScalaCode': 'string'
}

Response Structure

  • (dict) --

    • PythonScript (string) --

      A Python script to perform the mapping.

    • ScalaCode (string) --

      Scala code to perform the mapping.

UpdateCrawler (updated) Link ¶
Changes (request) 
{'Targets': {'DynamoDBTargets': [{'Path': 'string'}]}}

Updates a crawler. If a crawler is running, you must stop it using StopCrawler before updating it.

See also: AWS API Documentation

Request Syntax

client.update_crawler(
    Name='string',
    Role='string',
    DatabaseName='string',
    Description='string',
    Targets={
        'S3Targets': [
            {
                'Path': 'string',
                'Exclusions': [
                    'string',
                ]
            },
        ],
        'JdbcTargets': [
            {
                'ConnectionName': 'string',
                'Path': 'string',
                'Exclusions': [
                    'string',
                ]
            },
        ],
        'DynamoDBTargets': [
            {
                'Path': 'string'
            },
        ]
    },
    Schedule='string',
    Classifiers=[
        'string',
    ],
    TablePrefix='string',
    SchemaChangePolicy={
        'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
        'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
    },
    Configuration='string'
)
type Name:

string

param Name:

[REQUIRED]

Name of the new crawler.

type Role:

string

param Role:

The IAM role (or ARN of an IAM role) used by the new crawler to access customer resources.

type DatabaseName:

string

param DatabaseName:

The AWS Glue database where results are stored, such as: arn:aws:daylight:us-east-1::database/sometable/*.

type Description:

string

param Description:

A description of the new crawler.

type Targets:

dict

param Targets:

A list of targets to crawl.

  • S3Targets (list) --

    Specifies Amazon S3 targets.

    • (dict) --

      Specifies a data store in Amazon S3.

      • Path (string) --

        The path to the Amazon S3 target.

      • Exclusions (list) --

        A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

        • (string) --

  • JdbcTargets (list) --

    Specifies JDBC targets.

    • (dict) --

      Specifies a JDBC data store to crawl.

      • ConnectionName (string) --

        The name of the connection to use to connect to the JDBC target.

      • Path (string) --

        The path of the JDBC target.

      • Exclusions (list) --

        A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.

        • (string) --

  • DynamoDBTargets (list) --

    Specifies DynamoDB targets.

    • (dict) --

      Specifies a DynamoDB table to crawl.

      • Path (string) --

        The name of the DynamoDB table to crawl.

type Schedule:

string

param Schedule:

A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).

type Classifiers:

list

param Classifiers:

A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.

  • (string) --

type TablePrefix:

string

param TablePrefix:

The table prefix used for catalog tables that are created.

type SchemaChangePolicy:

dict

param SchemaChangePolicy:

Policy for the crawler's update and deletion behavior.

  • UpdateBehavior (string) --

    The update behavior when the crawler finds a changed schema.

  • DeleteBehavior (string) --

    The deletion behavior when the crawler finds a deleted object.

type Configuration:

string

param Configuration:

Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

  • (dict) --