AWS Glue

2023/05/25 - AWS Glue - 12 updated api methods

Changes  Added ability to create data quality rulesets for shared, cross-account Glue Data Catalog tables. Added support for dataset comparison rules through a new parameter called AdditionalDataSources. Enhanced the data quality results with a map containing profiled metric values.

BatchGetDataQualityResult (updated) Link ¶
Changes (response)
{'Results': {'RuleResults': {'EvaluatedMetrics': {'string': 'double'}}}}

Retrieves a list of data quality results for the specified result IDs.

See also: AWS API Documentation

Request Syntax

client.batch_get_data_quality_result(
    ResultIds=[
        'string',
    ]
)
type ResultIds:

list

param ResultIds:

[REQUIRED]

A list of unique result IDs for the data quality results.

  • (string) --

rtype:

dict

returns:

Response Syntax

{
    'Results': [
        {
            'ResultId': 'string',
            'Score': 123.0,
            'DataSource': {
                'GlueTable': {
                    'DatabaseName': 'string',
                    'TableName': 'string',
                    'CatalogId': 'string',
                    'ConnectionName': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    }
                }
            },
            'RulesetName': 'string',
            'EvaluationContext': 'string',
            'StartedOn': datetime(2015, 1, 1),
            'CompletedOn': datetime(2015, 1, 1),
            'JobName': 'string',
            'JobRunId': 'string',
            'RulesetEvaluationRunId': 'string',
            'RuleResults': [
                {
                    'Name': 'string',
                    'Description': 'string',
                    'EvaluationMessage': 'string',
                    'Result': 'PASS'|'FAIL'|'ERROR',
                    'EvaluatedMetrics': {
                        'string': 123.0
                    }
                },
            ]
        },
    ],
    'ResultsNotFound': [
        'string',
    ]
}

Response Structure

  • (dict) --

    • Results (list) --

      A list of DataQualityResult objects representing the data quality results.

      • (dict) --

        Describes a data quality result.

        • ResultId (string) --

          A unique result ID for the data quality result.

        • Score (float) --

          An aggregate data quality score. Represents the ratio of rules that passed to the total number of rules.

        • DataSource (dict) --

          The table associated with the data quality result, if any.

          • GlueTable (dict) --

            An Glue table.

            • DatabaseName (string) --

              A database name in the Glue Data Catalog.

            • TableName (string) --

              A table name in the Glue Data Catalog.

            • CatalogId (string) --

              A unique identifier for the Glue Data Catalog.

            • ConnectionName (string) --

              The name of the connection to the Glue Data Catalog.

            • AdditionalOptions (dict) --

              Additional options for the table. Currently there are two keys supported:

              • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

              • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

              • (string) --

                • (string) --

        • RulesetName (string) --

          The name of the ruleset associated with the data quality result.

        • EvaluationContext (string) --

          In the context of a job in Glue Studio, each node in the canvas is typically assigned some sort of name and data quality nodes will have names. In the case of multiple nodes, the evaluationContext can differentiate the nodes.

        • StartedOn (datetime) --

          The date and time when this data quality run started.

        • CompletedOn (datetime) --

          The date and time when this data quality run completed.

        • JobName (string) --

          The job name associated with the data quality result, if any.

        • JobRunId (string) --

          The job run ID associated with the data quality result, if any.

        • RulesetEvaluationRunId (string) --

          The unique run ID for the ruleset evaluation for this data quality result.

        • RuleResults (list) --

          A list of DataQualityRuleResult objects representing the results for each rule.

          • (dict) --

            Describes the result of the evaluation of a data quality rule.

            • Name (string) --

              The name of the data quality rule.

            • Description (string) --

              A description of the data quality rule.

            • EvaluationMessage (string) --

              An evaluation message.

            • Result (string) --

              A pass or fail status for the rule.

            • EvaluatedMetrics (dict) --

              A map of metrics associated with the evaluation of the rule.

              • (string) --

                • (float) --

    • ResultsNotFound (list) --

      A list of result IDs for which results were not found.

      • (string) --

BatchGetJobs (updated) Link ¶
Changes (response)
{'Jobs': {'CodeGenConfigurationNodes': {'EvaluateDataQualityMultiFrame': {'AdditionalDataSources': {'string': 'string'},
                                                                          'AdditionalOptions': {'performanceTuning.caching': 'string'},
                                                                          'Inputs': ['string'],
                                                                          'Name': 'string',
                                                                          'PublishingOptions': {'CloudWatchMetricsEnabled': 'boolean',
                                                                                                'EvaluationContext': 'string',
                                                                                                'ResultsPublishingEnabled': 'boolean',
                                                                                                'ResultsS3Prefix': 'string'},
                                                                          'Ruleset': 'string',
                                                                          'StopJobOnFailureOptions': {'StopJobOnFailureTiming': 'Immediate '
                                                                                                                                '| '
                                                                                                                                'AfterDataLoad'}}}}}

Returns a list of resource metadata for a given list of job names. After calling the ListJobs operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.

See also: AWS API Documentation

Request Syntax

client.batch_get_jobs(
    JobNames=[
        'string',
    ]
)
type JobNames:

list

param JobNames:

[REQUIRED]

A list of job names, which might be the names returned from the ListJobs operation.

  • (string) --

rtype:

dict

returns:

Response Syntax

{
    'Jobs': [
        {
            'Name': 'string',
            'Description': 'string',
            'LogUri': 'string',
            'Role': 'string',
            'CreatedOn': datetime(2015, 1, 1),
            'LastModifiedOn': datetime(2015, 1, 1),
            'ExecutionProperty': {
                'MaxConcurrentRuns': 123
            },
            'Command': {
                'Name': 'string',
                'ScriptLocation': 'string',
                'PythonVersion': 'string'
            },
            'DefaultArguments': {
                'string': 'string'
            },
            'NonOverridableArguments': {
                'string': 'string'
            },
            'Connections': {
                'Connections': [
                    'string',
                ]
            },
            'MaxRetries': 123,
            'AllocatedCapacity': 123,
            'Timeout': 123,
            'MaxCapacity': 123.0,
            'WorkerType': 'Standard'|'G.1X'|'G.2X'|'G.025X'|'G.4X'|'G.8X',
            'NumberOfWorkers': 123,
            'SecurityConfiguration': 'string',
            'NotificationProperty': {
                'NotifyDelayAfter': 123
            },
            'GlueVersion': 'string',
            'CodeGenConfigurationNodes': {
                'string': {
                    'AthenaConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'ConnectionTable': 'string',
                        'SchemaName': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'JDBCConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'FilterPredicate': 'string',
                            'PartitionColumn': 'string',
                            'LowerBound': 123,
                            'UpperBound': 123,
                            'NumPartitions': 123,
                            'JobBookmarkKeys': [
                                'string',
                            ],
                            'JobBookmarkKeysSortOrder': 'string',
                            'DataTypeMapping': {
                                'string': 'DATE'|'STRING'|'TIMESTAMP'|'INT'|'FLOAT'|'LONG'|'BIGDECIMAL'|'BYTE'|'SHORT'|'DOUBLE'
                            }
                        },
                        'ConnectionTable': 'string',
                        'Query': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'RedshiftSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'RedshiftTmpDir': 'string',
                        'TmpDirIAMRole': 'string'
                    },
                    'S3CatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'PartitionPredicate': 'string',
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123
                        }
                    },
                    'S3CsvSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'gzip'|'bzip2',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'Separator': 'comma'|'ctrla'|'pipe'|'semicolon'|'tab',
                        'Escaper': 'string',
                        'QuoteChar': 'quote'|'quillemet'|'single_quote'|'disabled',
                        'Multiline': True|False,
                        'WithHeader': True|False,
                        'WriteHeader': True|False,
                        'SkipFirst': True|False,
                        'OptimizePerformance': True|False,
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3JsonSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'gzip'|'bzip2',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'JsonPath': 'string',
                        'Multiline': True|False,
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3ParquetSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'RelationalCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'DynamoDBCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'JDBCConnectorTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ConnectionName': 'string',
                        'ConnectionTable': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkConnectorTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'RedshiftTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string',
                        'RedshiftTmpDir': 'string',
                        'TmpDirIAMRole': 'string',
                        'UpsertRedshiftOptions': {
                            'TableLocation': 'string',
                            'ConnectionName': 'string',
                            'UpsertKeys': [
                                'string',
                            ]
                        }
                    },
                    'S3CatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3GlueParquetTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'S3DirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'string',
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'ApplyMapping': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Mapping': [
                            {
                                'ToKey': 'string',
                                'FromPath': [
                                    'string',
                                ],
                                'FromType': 'string',
                                'ToType': 'string',
                                'Dropped': True|False,
                                'Children': {'... recursive ...'}
                            },
                        ]
                    },
                    'SelectFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'DropFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'RenameField': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'SourcePath': [
                            'string',
                        ],
                        'TargetPath': [
                            'string',
                        ]
                    },
                    'Spigot': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Path': 'string',
                        'Topk': 123,
                        'Prob': 123.0
                    },
                    'Join': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'JoinType': 'equijoin'|'left'|'right'|'outer'|'leftsemi'|'leftanti',
                        'Columns': [
                            {
                                'From': 'string',
                                'Keys': [
                                    [
                                        'string',
                                    ],
                                ]
                            },
                        ]
                    },
                    'SplitFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'SelectFromCollection': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Index': 123
                    },
                    'FillMissingValues': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ImputedPath': 'string',
                        'FilledPath': 'string'
                    },
                    'Filter': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'LogicalOperator': 'AND'|'OR',
                        'Filters': [
                            {
                                'Operation': 'EQ'|'LT'|'GT'|'LTE'|'GTE'|'REGEX'|'ISNULL',
                                'Negated': True|False,
                                'Values': [
                                    {
                                        'Type': 'COLUMNEXTRACTED'|'CONSTANT',
                                        'Value': [
                                            'string',
                                        ]
                                    },
                                ]
                            },
                        ]
                    },
                    'CustomCode': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Code': 'string',
                        'ClassName': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkSQL': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'SqlQuery': 'string',
                        'SqlAliases': [
                            {
                                'From': 'string',
                                'Alias': 'string'
                            },
                        ],
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'DirectKinesisSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'StreamingOptions': {
                            'EndpointUrl': 'string',
                            'StreamName': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                            'MaxFetchTimeInMs': 123,
                            'MaxFetchRecordsPerShard': 123,
                            'MaxRecordPerRead': 123,
                            'AddIdleTimeBetweenReads': True|False,
                            'IdleTimeBetweenReadsInMs': 123,
                            'DescribeShardInterval': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxRetryIntervalMs': 123,
                            'AvoidEmptyBatches': True|False,
                            'StreamArn': 'string',
                            'RoleArn': 'string',
                            'RoleSessionName': 'string',
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'DirectKafkaSource': {
                        'Name': 'string',
                        'StreamingOptions': {
                            'BootstrapServers': 'string',
                            'SecurityProtocol': 'string',
                            'ConnectionName': 'string',
                            'TopicName': 'string',
                            'Assign': 'string',
                            'SubscribePattern': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingOffsets': 'string',
                            'EndingOffsets': 'string',
                            'PollTimeoutMs': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxOffsetsPerTrigger': 123,
                            'MinPartitions': 123,
                            'IncludeHeaders': True|False,
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'CatalogKinesisSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'Table': 'string',
                        'Database': 'string',
                        'StreamingOptions': {
                            'EndpointUrl': 'string',
                            'StreamName': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                            'MaxFetchTimeInMs': 123,
                            'MaxFetchRecordsPerShard': 123,
                            'MaxRecordPerRead': 123,
                            'AddIdleTimeBetweenReads': True|False,
                            'IdleTimeBetweenReadsInMs': 123,
                            'DescribeShardInterval': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxRetryIntervalMs': 123,
                            'AvoidEmptyBatches': True|False,
                            'StreamArn': 'string',
                            'RoleArn': 'string',
                            'RoleSessionName': 'string',
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'CatalogKafkaSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'Table': 'string',
                        'Database': 'string',
                        'StreamingOptions': {
                            'BootstrapServers': 'string',
                            'SecurityProtocol': 'string',
                            'ConnectionName': 'string',
                            'TopicName': 'string',
                            'Assign': 'string',
                            'SubscribePattern': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingOffsets': 'string',
                            'EndingOffsets': 'string',
                            'PollTimeoutMs': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxOffsetsPerTrigger': 123,
                            'MinPartitions': 123,
                            'IncludeHeaders': True|False,
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'DropNullFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'NullCheckBoxList': {
                            'IsEmpty': True|False,
                            'IsNullString': True|False,
                            'IsNegOne': True|False
                        },
                        'NullTextList': [
                            {
                                'Value': 'string',
                                'Datatype': {
                                    'Id': 'string',
                                    'Label': 'string'
                                }
                            },
                        ]
                    },
                    'Merge': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Source': 'string',
                        'PrimaryKeys': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'Union': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'UnionType': 'ALL'|'DISTINCT'
                    },
                    'PIIDetection': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PiiType': 'RowAudit'|'RowMasking'|'ColumnAudit'|'ColumnMasking',
                        'EntityTypesToDetect': [
                            'string',
                        ],
                        'OutputColumnName': 'string',
                        'SampleFraction': 123.0,
                        'ThresholdFraction': 123.0,
                        'MaskValue': 'string'
                    },
                    'Aggregate': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Groups': [
                            [
                                'string',
                            ],
                        ],
                        'Aggs': [
                            {
                                'Column': [
                                    'string',
                                ],
                                'AggFunc': 'avg'|'countDistinct'|'count'|'first'|'last'|'kurtosis'|'max'|'min'|'skewness'|'stddev_samp'|'stddev_pop'|'sum'|'sumDistinct'|'var_samp'|'var_pop'
                            },
                        ]
                    },
                    'DropDuplicates': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Columns': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'GovernedCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'GovernedCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'PartitionPredicate': 'string',
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123
                        }
                    },
                    'MicrosoftSQLServerCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MySQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'OracleSQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'PostgreSQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MicrosoftSQLServerCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MySQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'OracleSQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'PostgreSQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'DynamicTransform': {
                        'Name': 'string',
                        'TransformName': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Parameters': [
                            {
                                'Name': 'string',
                                'Type': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                                'ValidationRule': 'string',
                                'ValidationMessage': 'string',
                                'Value': [
                                    'string',
                                ],
                                'ListType': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                                'IsOptional': True|False
                            },
                        ],
                        'FunctionName': 'string',
                        'Path': 'string',
                        'Version': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'EvaluateDataQuality': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Ruleset': 'string',
                        'Output': 'PrimaryInput'|'EvaluationResults',
                        'PublishingOptions': {
                            'EvaluationContext': 'string',
                            'ResultsS3Prefix': 'string',
                            'CloudWatchMetricsEnabled': True|False,
                            'ResultsPublishingEnabled': True|False
                        },
                        'StopJobOnFailureOptions': {
                            'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                        }
                    },
                    'S3CatalogHudiSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogHudiSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3HudiSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3HudiCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3HudiDirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Path': 'string',
                        'Compression': 'gzip'|'lzo'|'uncompressed'|'snappy',
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'DirectJDBCSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'ConnectionName': 'string',
                        'ConnectionType': 'sqlserver'|'mysql'|'oracle'|'postgresql'|'redshift',
                        'RedshiftTmpDir': 'string'
                    },
                    'S3CatalogDeltaSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogDeltaSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3DeltaSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3DeltaCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3DeltaDirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'uncompressed'|'snappy',
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'AmazonRedshiftSource': {
                        'Name': 'string',
                        'Data': {
                            'AccessType': 'string',
                            'SourceType': 'string',
                            'Connection': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Schema': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Table': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogDatabase': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogTable': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogRedshiftSchema': 'string',
                            'CatalogRedshiftTable': 'string',
                            'TempDir': 'string',
                            'IamRole': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'AdvancedOptions': [
                                {
                                    'Key': 'string',
                                    'Value': 'string'
                                },
                            ],
                            'SampleQuery': 'string',
                            'PreAction': 'string',
                            'PostAction': 'string',
                            'Action': 'string',
                            'TablePrefix': 'string',
                            'Upsert': True|False,
                            'MergeAction': 'string',
                            'MergeWhenMatched': 'string',
                            'MergeWhenNotMatched': 'string',
                            'MergeClause': 'string',
                            'CrawlerConnection': 'string',
                            'TableSchema': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ],
                            'StagingTable': 'string',
                            'SelectedColumns': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ]
                        }
                    },
                    'AmazonRedshiftTarget': {
                        'Name': 'string',
                        'Data': {
                            'AccessType': 'string',
                            'SourceType': 'string',
                            'Connection': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Schema': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Table': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogDatabase': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogTable': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogRedshiftSchema': 'string',
                            'CatalogRedshiftTable': 'string',
                            'TempDir': 'string',
                            'IamRole': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'AdvancedOptions': [
                                {
                                    'Key': 'string',
                                    'Value': 'string'
                                },
                            ],
                            'SampleQuery': 'string',
                            'PreAction': 'string',
                            'PostAction': 'string',
                            'Action': 'string',
                            'TablePrefix': 'string',
                            'Upsert': True|False,
                            'MergeAction': 'string',
                            'MergeWhenMatched': 'string',
                            'MergeWhenNotMatched': 'string',
                            'MergeClause': 'string',
                            'CrawlerConnection': 'string',
                            'TableSchema': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ],
                            'StagingTable': 'string',
                            'SelectedColumns': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ]
                        },
                        'Inputs': [
                            'string',
                        ]
                    },
                    'EvaluateDataQualityMultiFrame': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'AdditionalDataSources': {
                            'string': 'string'
                        },
                        'Ruleset': 'string',
                        'PublishingOptions': {
                            'EvaluationContext': 'string',
                            'ResultsS3Prefix': 'string',
                            'CloudWatchMetricsEnabled': True|False,
                            'ResultsPublishingEnabled': True|False
                        },
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'StopJobOnFailureOptions': {
                            'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                        }
                    }
                }
            },
            'ExecutionClass': 'FLEX'|'STANDARD',
            'SourceControlDetails': {
                'Provider': 'GITHUB'|'AWS_CODE_COMMIT',
                'Repository': 'string',
                'Owner': 'string',
                'Branch': 'string',
                'Folder': 'string',
                'LastCommitId': 'string',
                'AuthStrategy': 'PERSONAL_ACCESS_TOKEN'|'AWS_SECRETS_MANAGER',
                'AuthToken': 'string'
            }
        },
    ],
    'JobsNotFound': [
        'string',
    ]
}


**Response Structure**
::

    # This section is too large to render.
    # Please see the AWS API Documentation linked below.

`AWS API Documentation <https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/BatchGetJobs>`_
CreateDataQualityRuleset (updated) Link ¶
Changes (request)
{'TargetTable': {'CatalogId': 'string'}}

Creates a data quality ruleset with DQDL rules applied to a specified Glue table.

You create the ruleset using the Data Quality Definition Language (DQDL). For more information, see the Glue developer guide.

See also: AWS API Documentation

Request Syntax

client.create_data_quality_ruleset(
    Name='string',
    Description='string',
    Ruleset='string',
    Tags={
        'string': 'string'
    },
    TargetTable={
        'TableName': 'string',
        'DatabaseName': 'string',
        'CatalogId': 'string'
    },
    ClientToken='string'
)
type Name:

string

param Name:

[REQUIRED]

A unique name for the data quality ruleset.

type Description:

string

param Description:

A description of the data quality ruleset.

type Ruleset:

string

param Ruleset:

[REQUIRED]

A Data Quality Definition Language (DQDL) ruleset. For more information, see the Glue developer guide.

type Tags:

dict

param Tags:

A list of tags applied to the data quality ruleset.

  • (string) --

    • (string) --

type TargetTable:

dict

param TargetTable:

A target table associated with the data quality ruleset.

  • TableName (string) -- [REQUIRED]

    The name of the Glue table.

  • DatabaseName (string) -- [REQUIRED]

    The name of the database where the Glue table exists.

  • CatalogId (string) --

    The catalog id where the Glue table exists.

type ClientToken:

string

param ClientToken:

Used for idempotency and is recommended to be set to a random ID (such as a UUID) to avoid creating or starting multiple instances of the same resource.

rtype:

dict

returns:

Response Syntax

{
    'Name': 'string'
}

Response Structure

  • (dict) --

    • Name (string) --

      A unique name for the data quality ruleset.

CreateJob (updated) Link ¶
Changes (request)
{'CodeGenConfigurationNodes': {'EvaluateDataQualityMultiFrame': {'AdditionalDataSources': {'string': 'string'},
                                                                 'AdditionalOptions': {'performanceTuning.caching': 'string'},
                                                                 'Inputs': ['string'],
                                                                 'Name': 'string',
                                                                 'PublishingOptions': {'CloudWatchMetricsEnabled': 'boolean',
                                                                                       'EvaluationContext': 'string',
                                                                                       'ResultsPublishingEnabled': 'boolean',
                                                                                       'ResultsS3Prefix': 'string'},
                                                                 'Ruleset': 'string',
                                                                 'StopJobOnFailureOptions': {'StopJobOnFailureTiming': 'Immediate '
                                                                                                                       '| '
                                                                                                                       'AfterDataLoad'}}}}

Creates a new job definition.

See also: AWS API Documentation

Request Syntax

client.create_job(
    Name='string',
    Description='string',
    LogUri='string',
    Role='string',
    ExecutionProperty={
        'MaxConcurrentRuns': 123
    },
    Command={
        'Name': 'string',
        'ScriptLocation': 'string',
        'PythonVersion': 'string'
    },
    DefaultArguments={
        'string': 'string'
    },
    NonOverridableArguments={
        'string': 'string'
    },
    Connections={
        'Connections': [
            'string',
        ]
    },
    MaxRetries=123,
    AllocatedCapacity=123,
    Timeout=123,
    MaxCapacity=123.0,
    SecurityConfiguration='string',
    Tags={
        'string': 'string'
    },
    NotificationProperty={
        'NotifyDelayAfter': 123
    },
    GlueVersion='string',
    NumberOfWorkers=123,
    WorkerType='Standard'|'G.1X'|'G.2X'|'G.025X'|'G.4X'|'G.8X',
    CodeGenConfigurationNodes={
        'string': {
            'AthenaConnectorSource': {
                'Name': 'string',
                'ConnectionName': 'string',
                'ConnectorName': 'string',
                'ConnectionType': 'string',
                'ConnectionTable': 'string',
                'SchemaName': 'string',
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'JDBCConnectorSource': {
                'Name': 'string',
                'ConnectionName': 'string',
                'ConnectorName': 'string',
                'ConnectionType': 'string',
                'AdditionalOptions': {
                    'FilterPredicate': 'string',
                    'PartitionColumn': 'string',
                    'LowerBound': 123,
                    'UpperBound': 123,
                    'NumPartitions': 123,
                    'JobBookmarkKeys': [
                        'string',
                    ],
                    'JobBookmarkKeysSortOrder': 'string',
                    'DataTypeMapping': {
                        'string': 'DATE'|'STRING'|'TIMESTAMP'|'INT'|'FLOAT'|'LONG'|'BIGDECIMAL'|'BYTE'|'SHORT'|'DOUBLE'
                    }
                },
                'ConnectionTable': 'string',
                'Query': 'string',
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'SparkConnectorSource': {
                'Name': 'string',
                'ConnectionName': 'string',
                'ConnectorName': 'string',
                'ConnectionType': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'CatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'RedshiftSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'RedshiftTmpDir': 'string',
                'TmpDirIAMRole': 'string'
            },
            'S3CatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'PartitionPredicate': 'string',
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123
                }
            },
            'S3CsvSource': {
                'Name': 'string',
                'Paths': [
                    'string',
                ],
                'CompressionType': 'gzip'|'bzip2',
                'Exclusions': [
                    'string',
                ],
                'GroupSize': 'string',
                'GroupFiles': 'string',
                'Recurse': True|False,
                'MaxBand': 123,
                'MaxFilesInBand': 123,
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123,
                    'EnableSamplePath': True|False,
                    'SamplePath': 'string'
                },
                'Separator': 'comma'|'ctrla'|'pipe'|'semicolon'|'tab',
                'Escaper': 'string',
                'QuoteChar': 'quote'|'quillemet'|'single_quote'|'disabled',
                'Multiline': True|False,
                'WithHeader': True|False,
                'WriteHeader': True|False,
                'SkipFirst': True|False,
                'OptimizePerformance': True|False,
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3JsonSource': {
                'Name': 'string',
                'Paths': [
                    'string',
                ],
                'CompressionType': 'gzip'|'bzip2',
                'Exclusions': [
                    'string',
                ],
                'GroupSize': 'string',
                'GroupFiles': 'string',
                'Recurse': True|False,
                'MaxBand': 123,
                'MaxFilesInBand': 123,
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123,
                    'EnableSamplePath': True|False,
                    'SamplePath': 'string'
                },
                'JsonPath': 'string',
                'Multiline': True|False,
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3ParquetSource': {
                'Name': 'string',
                'Paths': [
                    'string',
                ],
                'CompressionType': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                'Exclusions': [
                    'string',
                ],
                'GroupSize': 'string',
                'GroupFiles': 'string',
                'Recurse': True|False,
                'MaxBand': 123,
                'MaxFilesInBand': 123,
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123,
                    'EnableSamplePath': True|False,
                    'SamplePath': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'RelationalCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'DynamoDBCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'JDBCConnectorTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'ConnectionName': 'string',
                'ConnectionTable': 'string',
                'ConnectorName': 'string',
                'ConnectionType': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'SparkConnectorTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'ConnectionName': 'string',
                'ConnectorName': 'string',
                'ConnectionType': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'CatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string'
            },
            'RedshiftTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string',
                'RedshiftTmpDir': 'string',
                'TmpDirIAMRole': 'string',
                'UpsertRedshiftOptions': {
                    'TableLocation': 'string',
                    'ConnectionName': 'string',
                    'UpsertKeys': [
                        'string',
                    ]
                }
            },
            'S3CatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Table': 'string',
                'Database': 'string',
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                }
            },
            'S3GlueParquetTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Path': 'string',
                'Compression': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                    'Table': 'string',
                    'Database': 'string'
                }
            },
            'S3DirectTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Path': 'string',
                'Compression': 'string',
                'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                    'Table': 'string',
                    'Database': 'string'
                }
            },
            'ApplyMapping': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Mapping': [
                    {
                        'ToKey': 'string',
                        'FromPath': [
                            'string',
                        ],
                        'FromType': 'string',
                        'ToType': 'string',
                        'Dropped': True|False,
                        'Children': {'... recursive ...'}
                    },
                ]
            },
            'SelectFields': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Paths': [
                    [
                        'string',
                    ],
                ]
            },
            'DropFields': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Paths': [
                    [
                        'string',
                    ],
                ]
            },
            'RenameField': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'SourcePath': [
                    'string',
                ],
                'TargetPath': [
                    'string',
                ]
            },
            'Spigot': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Path': 'string',
                'Topk': 123,
                'Prob': 123.0
            },
            'Join': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'JoinType': 'equijoin'|'left'|'right'|'outer'|'leftsemi'|'leftanti',
                'Columns': [
                    {
                        'From': 'string',
                        'Keys': [
                            [
                                'string',
                            ],
                        ]
                    },
                ]
            },
            'SplitFields': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Paths': [
                    [
                        'string',
                    ],
                ]
            },
            'SelectFromCollection': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Index': 123
            },
            'FillMissingValues': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'ImputedPath': 'string',
                'FilledPath': 'string'
            },
            'Filter': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'LogicalOperator': 'AND'|'OR',
                'Filters': [
                    {
                        'Operation': 'EQ'|'LT'|'GT'|'LTE'|'GTE'|'REGEX'|'ISNULL',
                        'Negated': True|False,
                        'Values': [
                            {
                                'Type': 'COLUMNEXTRACTED'|'CONSTANT',
                                'Value': [
                                    'string',
                                ]
                            },
                        ]
                    },
                ]
            },
            'CustomCode': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Code': 'string',
                'ClassName': 'string',
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'SparkSQL': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'SqlQuery': 'string',
                'SqlAliases': [
                    {
                        'From': 'string',
                        'Alias': 'string'
                    },
                ],
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'DirectKinesisSource': {
                'Name': 'string',
                'WindowSize': 123,
                'DetectSchema': True|False,
                'StreamingOptions': {
                    'EndpointUrl': 'string',
                    'StreamName': 'string',
                    'Classification': 'string',
                    'Delimiter': 'string',
                    'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                    'MaxFetchTimeInMs': 123,
                    'MaxFetchRecordsPerShard': 123,
                    'MaxRecordPerRead': 123,
                    'AddIdleTimeBetweenReads': True|False,
                    'IdleTimeBetweenReadsInMs': 123,
                    'DescribeShardInterval': 123,
                    'NumRetries': 123,
                    'RetryIntervalMs': 123,
                    'MaxRetryIntervalMs': 123,
                    'AvoidEmptyBatches': True|False,
                    'StreamArn': 'string',
                    'RoleArn': 'string',
                    'RoleSessionName': 'string',
                    'AddRecordTimestamp': 'string',
                    'EmitConsumerLagMetrics': 'string'
                },
                'DataPreviewOptions': {
                    'PollingTime': 123,
                    'RecordPollingLimit': 123
                }
            },
            'DirectKafkaSource': {
                'Name': 'string',
                'StreamingOptions': {
                    'BootstrapServers': 'string',
                    'SecurityProtocol': 'string',
                    'ConnectionName': 'string',
                    'TopicName': 'string',
                    'Assign': 'string',
                    'SubscribePattern': 'string',
                    'Classification': 'string',
                    'Delimiter': 'string',
                    'StartingOffsets': 'string',
                    'EndingOffsets': 'string',
                    'PollTimeoutMs': 123,
                    'NumRetries': 123,
                    'RetryIntervalMs': 123,
                    'MaxOffsetsPerTrigger': 123,
                    'MinPartitions': 123,
                    'IncludeHeaders': True|False,
                    'AddRecordTimestamp': 'string',
                    'EmitConsumerLagMetrics': 'string'
                },
                'WindowSize': 123,
                'DetectSchema': True|False,
                'DataPreviewOptions': {
                    'PollingTime': 123,
                    'RecordPollingLimit': 123
                }
            },
            'CatalogKinesisSource': {
                'Name': 'string',
                'WindowSize': 123,
                'DetectSchema': True|False,
                'Table': 'string',
                'Database': 'string',
                'StreamingOptions': {
                    'EndpointUrl': 'string',
                    'StreamName': 'string',
                    'Classification': 'string',
                    'Delimiter': 'string',
                    'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                    'MaxFetchTimeInMs': 123,
                    'MaxFetchRecordsPerShard': 123,
                    'MaxRecordPerRead': 123,
                    'AddIdleTimeBetweenReads': True|False,
                    'IdleTimeBetweenReadsInMs': 123,
                    'DescribeShardInterval': 123,
                    'NumRetries': 123,
                    'RetryIntervalMs': 123,
                    'MaxRetryIntervalMs': 123,
                    'AvoidEmptyBatches': True|False,
                    'StreamArn': 'string',
                    'RoleArn': 'string',
                    'RoleSessionName': 'string',
                    'AddRecordTimestamp': 'string',
                    'EmitConsumerLagMetrics': 'string'
                },
                'DataPreviewOptions': {
                    'PollingTime': 123,
                    'RecordPollingLimit': 123
                }
            },
            'CatalogKafkaSource': {
                'Name': 'string',
                'WindowSize': 123,
                'DetectSchema': True|False,
                'Table': 'string',
                'Database': 'string',
                'StreamingOptions': {
                    'BootstrapServers': 'string',
                    'SecurityProtocol': 'string',
                    'ConnectionName': 'string',
                    'TopicName': 'string',
                    'Assign': 'string',
                    'SubscribePattern': 'string',
                    'Classification': 'string',
                    'Delimiter': 'string',
                    'StartingOffsets': 'string',
                    'EndingOffsets': 'string',
                    'PollTimeoutMs': 123,
                    'NumRetries': 123,
                    'RetryIntervalMs': 123,
                    'MaxOffsetsPerTrigger': 123,
                    'MinPartitions': 123,
                    'IncludeHeaders': True|False,
                    'AddRecordTimestamp': 'string',
                    'EmitConsumerLagMetrics': 'string'
                },
                'DataPreviewOptions': {
                    'PollingTime': 123,
                    'RecordPollingLimit': 123
                }
            },
            'DropNullFields': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'NullCheckBoxList': {
                    'IsEmpty': True|False,
                    'IsNullString': True|False,
                    'IsNegOne': True|False
                },
                'NullTextList': [
                    {
                        'Value': 'string',
                        'Datatype': {
                            'Id': 'string',
                            'Label': 'string'
                        }
                    },
                ]
            },
            'Merge': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Source': 'string',
                'PrimaryKeys': [
                    [
                        'string',
                    ],
                ]
            },
            'Union': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'UnionType': 'ALL'|'DISTINCT'
            },
            'PIIDetection': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PiiType': 'RowAudit'|'RowMasking'|'ColumnAudit'|'ColumnMasking',
                'EntityTypesToDetect': [
                    'string',
                ],
                'OutputColumnName': 'string',
                'SampleFraction': 123.0,
                'ThresholdFraction': 123.0,
                'MaskValue': 'string'
            },
            'Aggregate': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Groups': [
                    [
                        'string',
                    ],
                ],
                'Aggs': [
                    {
                        'Column': [
                            'string',
                        ],
                        'AggFunc': 'avg'|'countDistinct'|'count'|'first'|'last'|'kurtosis'|'max'|'min'|'skewness'|'stddev_samp'|'stddev_pop'|'sum'|'sumDistinct'|'var_samp'|'var_pop'
                    },
                ]
            },
            'DropDuplicates': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Columns': [
                    [
                        'string',
                    ],
                ]
            },
            'GovernedCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Table': 'string',
                'Database': 'string',
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                }
            },
            'GovernedCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'PartitionPredicate': 'string',
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123
                }
            },
            'MicrosoftSQLServerCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'MySQLCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'OracleSQLCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'PostgreSQLCatalogSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string'
            },
            'MicrosoftSQLServerCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string'
            },
            'MySQLCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string'
            },
            'OracleSQLCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string'
            },
            'PostgreSQLCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Database': 'string',
                'Table': 'string'
            },
            'DynamicTransform': {
                'Name': 'string',
                'TransformName': 'string',
                'Inputs': [
                    'string',
                ],
                'Parameters': [
                    {
                        'Name': 'string',
                        'Type': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                        'ValidationRule': 'string',
                        'ValidationMessage': 'string',
                        'Value': [
                            'string',
                        ],
                        'ListType': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                        'IsOptional': True|False
                    },
                ],
                'FunctionName': 'string',
                'Path': 'string',
                'Version': 'string',
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'EvaluateDataQuality': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Ruleset': 'string',
                'Output': 'PrimaryInput'|'EvaluationResults',
                'PublishingOptions': {
                    'EvaluationContext': 'string',
                    'ResultsS3Prefix': 'string',
                    'CloudWatchMetricsEnabled': True|False,
                    'ResultsPublishingEnabled': True|False
                },
                'StopJobOnFailureOptions': {
                    'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                }
            },
            'S3CatalogHudiSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'AdditionalHudiOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'CatalogHudiSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'AdditionalHudiOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3HudiSource': {
                'Name': 'string',
                'Paths': [
                    'string',
                ],
                'AdditionalHudiOptions': {
                    'string': 'string'
                },
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123,
                    'EnableSamplePath': True|False,
                    'SamplePath': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3HudiCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Table': 'string',
                'Database': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                }
            },
            'S3HudiDirectTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'Path': 'string',
                'Compression': 'gzip'|'lzo'|'uncompressed'|'snappy',
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                    'Table': 'string',
                    'Database': 'string'
                }
            },
            'DirectJDBCSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'ConnectionName': 'string',
                'ConnectionType': 'sqlserver'|'mysql'|'oracle'|'postgresql'|'redshift',
                'RedshiftTmpDir': 'string'
            },
            'S3CatalogDeltaSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'AdditionalDeltaOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'CatalogDeltaSource': {
                'Name': 'string',
                'Database': 'string',
                'Table': 'string',
                'AdditionalDeltaOptions': {
                    'string': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3DeltaSource': {
                'Name': 'string',
                'Paths': [
                    'string',
                ],
                'AdditionalDeltaOptions': {
                    'string': 'string'
                },
                'AdditionalOptions': {
                    'BoundedSize': 123,
                    'BoundedFiles': 123,
                    'EnableSamplePath': True|False,
                    'SamplePath': 'string'
                },
                'OutputSchemas': [
                    {
                        'Columns': [
                            {
                                'Name': 'string',
                                'Type': 'string'
                            },
                        ]
                    },
                ]
            },
            'S3DeltaCatalogTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Table': 'string',
                'Database': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                }
            },
            'S3DeltaDirectTarget': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'PartitionKeys': [
                    [
                        'string',
                    ],
                ],
                'Path': 'string',
                'Compression': 'uncompressed'|'snappy',
                'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                'AdditionalOptions': {
                    'string': 'string'
                },
                'SchemaChangePolicy': {
                    'EnableUpdateCatalog': True|False,
                    'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                    'Table': 'string',
                    'Database': 'string'
                }
            },
            'AmazonRedshiftSource': {
                'Name': 'string',
                'Data': {
                    'AccessType': 'string',
                    'SourceType': 'string',
                    'Connection': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'Schema': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'Table': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogDatabase': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogTable': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogRedshiftSchema': 'string',
                    'CatalogRedshiftTable': 'string',
                    'TempDir': 'string',
                    'IamRole': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'AdvancedOptions': [
                        {
                            'Key': 'string',
                            'Value': 'string'
                        },
                    ],
                    'SampleQuery': 'string',
                    'PreAction': 'string',
                    'PostAction': 'string',
                    'Action': 'string',
                    'TablePrefix': 'string',
                    'Upsert': True|False,
                    'MergeAction': 'string',
                    'MergeWhenMatched': 'string',
                    'MergeWhenNotMatched': 'string',
                    'MergeClause': 'string',
                    'CrawlerConnection': 'string',
                    'TableSchema': [
                        {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                    ],
                    'StagingTable': 'string',
                    'SelectedColumns': [
                        {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                    ]
                }
            },
            'AmazonRedshiftTarget': {
                'Name': 'string',
                'Data': {
                    'AccessType': 'string',
                    'SourceType': 'string',
                    'Connection': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'Schema': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'Table': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogDatabase': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogTable': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'CatalogRedshiftSchema': 'string',
                    'CatalogRedshiftTable': 'string',
                    'TempDir': 'string',
                    'IamRole': {
                        'Value': 'string',
                        'Label': 'string',
                        'Description': 'string'
                    },
                    'AdvancedOptions': [
                        {
                            'Key': 'string',
                            'Value': 'string'
                        },
                    ],
                    'SampleQuery': 'string',
                    'PreAction': 'string',
                    'PostAction': 'string',
                    'Action': 'string',
                    'TablePrefix': 'string',
                    'Upsert': True|False,
                    'MergeAction': 'string',
                    'MergeWhenMatched': 'string',
                    'MergeWhenNotMatched': 'string',
                    'MergeClause': 'string',
                    'CrawlerConnection': 'string',
                    'TableSchema': [
                        {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                    ],
                    'StagingTable': 'string',
                    'SelectedColumns': [
                        {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                    ]
                },
                'Inputs': [
                    'string',
                ]
            },
            'EvaluateDataQualityMultiFrame': {
                'Name': 'string',
                'Inputs': [
                    'string',
                ],
                'AdditionalDataSources': {
                    'string': 'string'
                },
                'Ruleset': 'string',
                'PublishingOptions': {
                    'EvaluationContext': 'string',
                    'ResultsS3Prefix': 'string',
                    'CloudWatchMetricsEnabled': True|False,
                    'ResultsPublishingEnabled': True|False
                },
                'AdditionalOptions': {
                    'string': 'string'
                },
                'StopJobOnFailureOptions': {
                    'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                }
            }
        }
    },
    ExecutionClass='FLEX'|'STANDARD',
    SourceControlDetails={
        'Provider': 'GITHUB'|'AWS_CODE_COMMIT',
        'Repository': 'string',
        'Owner': 'string',
        'Branch': 'string',
        'Folder': 'string',
        'LastCommitId': 'string',
        'AuthStrategy': 'PERSONAL_ACCESS_TOKEN'|'AWS_SECRETS_MANAGER',
        'AuthToken': 'string'
    }
)


  **Parameters**
  ::

      # This section is too large to render.
      # Please see the AWS API Documentation linked below.

  `AWS API Documentation <https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/CreateJob>`_
rtype:

dict

returns:

Response Syntax

{
    'Name': 'string'
}

Response Structure

  • (dict) --

    • Name (string) --

      The unique name that was provided for this job definition.

GetDataQualityResult (updated) Link ¶
Changes (response)
{'RuleResults': {'EvaluatedMetrics': {'string': 'double'}}}

Retrieves the result of a data quality rule evaluation.

See also: AWS API Documentation

Request Syntax

client.get_data_quality_result(
    ResultId='string'
)
type ResultId:

string

param ResultId:

[REQUIRED]

A unique result ID for the data quality result.

rtype:

dict

returns:

Response Syntax

{
    'ResultId': 'string',
    'Score': 123.0,
    'DataSource': {
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        }
    },
    'RulesetName': 'string',
    'EvaluationContext': 'string',
    'StartedOn': datetime(2015, 1, 1),
    'CompletedOn': datetime(2015, 1, 1),
    'JobName': 'string',
    'JobRunId': 'string',
    'RulesetEvaluationRunId': 'string',
    'RuleResults': [
        {
            'Name': 'string',
            'Description': 'string',
            'EvaluationMessage': 'string',
            'Result': 'PASS'|'FAIL'|'ERROR',
            'EvaluatedMetrics': {
                'string': 123.0
            }
        },
    ]
}

Response Structure

  • (dict) --

    • ResultId (string) --

      A unique result ID for the data quality result.

    • Score (float) --

      An aggregate data quality score. Represents the ratio of rules that passed to the total number of rules.

    • DataSource (dict) --

      The table associated with the data quality result, if any.

      • GlueTable (dict) --

        An Glue table.

        • DatabaseName (string) --

          A database name in the Glue Data Catalog.

        • TableName (string) --

          A table name in the Glue Data Catalog.

        • CatalogId (string) --

          A unique identifier for the Glue Data Catalog.

        • ConnectionName (string) --

          The name of the connection to the Glue Data Catalog.

        • AdditionalOptions (dict) --

          Additional options for the table. Currently there are two keys supported:

          • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

          • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

          • (string) --

            • (string) --

    • RulesetName (string) --

      The name of the ruleset associated with the data quality result.

    • EvaluationContext (string) --

      In the context of a job in Glue Studio, each node in the canvas is typically assigned some sort of name and data quality nodes will have names. In the case of multiple nodes, the evaluationContext can differentiate the nodes.

    • StartedOn (datetime) --

      The date and time when the run for this data quality result started.

    • CompletedOn (datetime) --

      The date and time when the run for this data quality result was completed.

    • JobName (string) --

      The job name associated with the data quality result, if any.

    • JobRunId (string) --

      The job run ID associated with the data quality result, if any.

    • RulesetEvaluationRunId (string) --

      The unique run ID associated with the ruleset evaluation.

    • RuleResults (list) --

      A list of DataQualityRuleResult objects representing the results for each rule.

      • (dict) --

        Describes the result of the evaluation of a data quality rule.

        • Name (string) --

          The name of the data quality rule.

        • Description (string) --

          A description of the data quality rule.

        • EvaluationMessage (string) --

          An evaluation message.

        • Result (string) --

          A pass or fail status for the rule.

        • EvaluatedMetrics (dict) --

          A map of metrics associated with the evaluation of the rule.

          • (string) --

            • (float) --

GetDataQualityRuleset (updated) Link ¶
Changes (response)
{'TargetTable': {'CatalogId': 'string'}}

Returns an existing ruleset by identifier or name.

See also: AWS API Documentation

Request Syntax

client.get_data_quality_ruleset(
    Name='string'
)
type Name:

string

param Name:

[REQUIRED]

The name of the ruleset.

rtype:

dict

returns:

Response Syntax

{
    'Name': 'string',
    'Description': 'string',
    'Ruleset': 'string',
    'TargetTable': {
        'TableName': 'string',
        'DatabaseName': 'string',
        'CatalogId': 'string'
    },
    'CreatedOn': datetime(2015, 1, 1),
    'LastModifiedOn': datetime(2015, 1, 1),
    'RecommendationRunId': 'string'
}

Response Structure

  • (dict) --

    • Name (string) --

      The name of the ruleset.

    • Description (string) --

      A description of the ruleset.

    • Ruleset (string) --

      A Data Quality Definition Language (DQDL) ruleset. For more information, see the Glue developer guide.

    • TargetTable (dict) --

      The name and database name of the target table.

      • TableName (string) --

        The name of the Glue table.

      • DatabaseName (string) --

        The name of the database where the Glue table exists.

      • CatalogId (string) --

        The catalog id where the Glue table exists.

    • CreatedOn (datetime) --

      A timestamp. The time and date that this data quality ruleset was created.

    • LastModifiedOn (datetime) --

      A timestamp. The last point in time when this data quality ruleset was modified.

    • RecommendationRunId (string) --

      When a ruleset was created from a recommendation run, this run ID is generated to link the two together.

GetDataQualityRulesetEvaluationRun (updated) Link ¶
Changes (response)
{'AdditionalDataSources': {'string': {'GlueTable': {'AdditionalOptions': {'string': 'string'},
                                                    'CatalogId': 'string',
                                                    'ConnectionName': 'string',
                                                    'DatabaseName': 'string',
                                                    'TableName': 'string'}}}}

Retrieves a specific run where a ruleset is evaluated against a data source.

See also: AWS API Documentation

Request Syntax

client.get_data_quality_ruleset_evaluation_run(
    RunId='string'
)
type RunId:

string

param RunId:

[REQUIRED]

The unique run identifier associated with this run.

rtype:

dict

returns:

Response Syntax

{
    'RunId': 'string',
    'DataSource': {
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        }
    },
    'Role': 'string',
    'NumberOfWorkers': 123,
    'Timeout': 123,
    'AdditionalRunOptions': {
        'CloudWatchMetricsEnabled': True|False,
        'ResultsS3Prefix': 'string'
    },
    'Status': 'STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT',
    'ErrorString': 'string',
    'StartedOn': datetime(2015, 1, 1),
    'LastModifiedOn': datetime(2015, 1, 1),
    'CompletedOn': datetime(2015, 1, 1),
    'ExecutionTime': 123,
    'RulesetNames': [
        'string',
    ],
    'ResultIds': [
        'string',
    ],
    'AdditionalDataSources': {
        'string': {
            'GlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                }
            }
        }
    }
}

Response Structure

  • (dict) --

    • RunId (string) --

      The unique run identifier associated with this run.

    • DataSource (dict) --

      The data source (an Glue table) associated with this evaluation run.

      • GlueTable (dict) --

        An Glue table.

        • DatabaseName (string) --

          A database name in the Glue Data Catalog.

        • TableName (string) --

          A table name in the Glue Data Catalog.

        • CatalogId (string) --

          A unique identifier for the Glue Data Catalog.

        • ConnectionName (string) --

          The name of the connection to the Glue Data Catalog.

        • AdditionalOptions (dict) --

          Additional options for the table. Currently there are two keys supported:

          • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

          • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

          • (string) --

            • (string) --

    • Role (string) --

      An IAM role supplied to encrypt the results of the run.

    • NumberOfWorkers (integer) --

      The number of G.1X workers to be used in the run. The default is 5.

    • Timeout (integer) --

      The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

    • AdditionalRunOptions (dict) --

      Additional run options you can specify for an evaluation run.

      • CloudWatchMetricsEnabled (boolean) --

        Whether or not to enable CloudWatch metrics.

      • ResultsS3Prefix (string) --

        Prefix for Amazon S3 to store results.

    • Status (string) --

      The status for this run.

    • ErrorString (string) --

      The error strings that are associated with the run.

    • StartedOn (datetime) --

      The date and time when this run started.

    • LastModifiedOn (datetime) --

      A timestamp. The last point in time when this data quality rule recommendation run was modified.

    • CompletedOn (datetime) --

      The date and time when this run was completed.

    • ExecutionTime (integer) --

      The amount of time (in seconds) that the run consumed resources.

    • RulesetNames (list) --

      A list of ruleset names for the run.

      • (string) --

    • ResultIds (list) --

      A list of result IDs for the data quality results for the run.

      • (string) --

    • AdditionalDataSources (dict) --

      A map of reference strings to additional data sources you can specify for an evaluation run.

      • (string) --

        • (dict) --

          A data source (an Glue table) for which you want data quality results.

          • GlueTable (dict) --

            An Glue table.

            • DatabaseName (string) --

              A database name in the Glue Data Catalog.

            • TableName (string) --

              A table name in the Glue Data Catalog.

            • CatalogId (string) --

              A unique identifier for the Glue Data Catalog.

            • ConnectionName (string) --

              The name of the connection to the Glue Data Catalog.

            • AdditionalOptions (dict) --

              Additional options for the table. Currently there are two keys supported:

              • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

              • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

              • (string) --

                • (string) --

GetJob (updated) Link ¶
Changes (response)
{'Job': {'CodeGenConfigurationNodes': {'EvaluateDataQualityMultiFrame': {'AdditionalDataSources': {'string': 'string'},
                                                                         'AdditionalOptions': {'performanceTuning.caching': 'string'},
                                                                         'Inputs': ['string'],
                                                                         'Name': 'string',
                                                                         'PublishingOptions': {'CloudWatchMetricsEnabled': 'boolean',
                                                                                               'EvaluationContext': 'string',
                                                                                               'ResultsPublishingEnabled': 'boolean',
                                                                                               'ResultsS3Prefix': 'string'},
                                                                         'Ruleset': 'string',
                                                                         'StopJobOnFailureOptions': {'StopJobOnFailureTiming': 'Immediate '
                                                                                                                               '| '
                                                                                                                               'AfterDataLoad'}}}}}

Retrieves an existing job definition.

See also: AWS API Documentation

Request Syntax

client.get_job(
    JobName='string'
)
type JobName:

string

param JobName:

[REQUIRED]

The name of the job definition to retrieve.

rtype:

dict

returns:

Response Syntax

{
    'Job': {
        'Name': 'string',
        'Description': 'string',
        'LogUri': 'string',
        'Role': 'string',
        'CreatedOn': datetime(2015, 1, 1),
        'LastModifiedOn': datetime(2015, 1, 1),
        'ExecutionProperty': {
            'MaxConcurrentRuns': 123
        },
        'Command': {
            'Name': 'string',
            'ScriptLocation': 'string',
            'PythonVersion': 'string'
        },
        'DefaultArguments': {
            'string': 'string'
        },
        'NonOverridableArguments': {
            'string': 'string'
        },
        'Connections': {
            'Connections': [
                'string',
            ]
        },
        'MaxRetries': 123,
        'AllocatedCapacity': 123,
        'Timeout': 123,
        'MaxCapacity': 123.0,
        'WorkerType': 'Standard'|'G.1X'|'G.2X'|'G.025X'|'G.4X'|'G.8X',
        'NumberOfWorkers': 123,
        'SecurityConfiguration': 'string',
        'NotificationProperty': {
            'NotifyDelayAfter': 123
        },
        'GlueVersion': 'string',
        'CodeGenConfigurationNodes': {
            'string': {
                'AthenaConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'ConnectionTable': 'string',
                    'SchemaName': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'JDBCConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'FilterPredicate': 'string',
                        'PartitionColumn': 'string',
                        'LowerBound': 123,
                        'UpperBound': 123,
                        'NumPartitions': 123,
                        'JobBookmarkKeys': [
                            'string',
                        ],
                        'JobBookmarkKeysSortOrder': 'string',
                        'DataTypeMapping': {
                            'string': 'DATE'|'STRING'|'TIMESTAMP'|'INT'|'FLOAT'|'LONG'|'BIGDECIMAL'|'BYTE'|'SHORT'|'DOUBLE'
                        }
                    },
                    'ConnectionTable': 'string',
                    'Query': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'RedshiftSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'RedshiftTmpDir': 'string',
                    'TmpDirIAMRole': 'string'
                },
                'S3CatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'PartitionPredicate': 'string',
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123
                    }
                },
                'S3CsvSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'gzip'|'bzip2',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'Separator': 'comma'|'ctrla'|'pipe'|'semicolon'|'tab',
                    'Escaper': 'string',
                    'QuoteChar': 'quote'|'quillemet'|'single_quote'|'disabled',
                    'Multiline': True|False,
                    'WithHeader': True|False,
                    'WriteHeader': True|False,
                    'SkipFirst': True|False,
                    'OptimizePerformance': True|False,
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3JsonSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'gzip'|'bzip2',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'JsonPath': 'string',
                    'Multiline': True|False,
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3ParquetSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'RelationalCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'DynamoDBCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'JDBCConnectorTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ConnectionName': 'string',
                    'ConnectionTable': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkConnectorTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'RedshiftTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string',
                    'RedshiftTmpDir': 'string',
                    'TmpDirIAMRole': 'string',
                    'UpsertRedshiftOptions': {
                        'TableLocation': 'string',
                        'ConnectionName': 'string',
                        'UpsertKeys': [
                            'string',
                        ]
                    }
                },
                'S3CatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3GlueParquetTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'S3DirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'string',
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'ApplyMapping': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Mapping': [
                        {
                            'ToKey': 'string',
                            'FromPath': [
                                'string',
                            ],
                            'FromType': 'string',
                            'ToType': 'string',
                            'Dropped': True|False,
                            'Children': {'... recursive ...'}
                        },
                    ]
                },
                'SelectFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'DropFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'RenameField': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'SourcePath': [
                        'string',
                    ],
                    'TargetPath': [
                        'string',
                    ]
                },
                'Spigot': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Path': 'string',
                    'Topk': 123,
                    'Prob': 123.0
                },
                'Join': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'JoinType': 'equijoin'|'left'|'right'|'outer'|'leftsemi'|'leftanti',
                    'Columns': [
                        {
                            'From': 'string',
                            'Keys': [
                                [
                                    'string',
                                ],
                            ]
                        },
                    ]
                },
                'SplitFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'SelectFromCollection': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Index': 123
                },
                'FillMissingValues': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ImputedPath': 'string',
                    'FilledPath': 'string'
                },
                'Filter': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'LogicalOperator': 'AND'|'OR',
                    'Filters': [
                        {
                            'Operation': 'EQ'|'LT'|'GT'|'LTE'|'GTE'|'REGEX'|'ISNULL',
                            'Negated': True|False,
                            'Values': [
                                {
                                    'Type': 'COLUMNEXTRACTED'|'CONSTANT',
                                    'Value': [
                                        'string',
                                    ]
                                },
                            ]
                        },
                    ]
                },
                'CustomCode': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Code': 'string',
                    'ClassName': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkSQL': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'SqlQuery': 'string',
                    'SqlAliases': [
                        {
                            'From': 'string',
                            'Alias': 'string'
                        },
                    ],
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'DirectKinesisSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'StreamingOptions': {
                        'EndpointUrl': 'string',
                        'StreamName': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                        'MaxFetchTimeInMs': 123,
                        'MaxFetchRecordsPerShard': 123,
                        'MaxRecordPerRead': 123,
                        'AddIdleTimeBetweenReads': True|False,
                        'IdleTimeBetweenReadsInMs': 123,
                        'DescribeShardInterval': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxRetryIntervalMs': 123,
                        'AvoidEmptyBatches': True|False,
                        'StreamArn': 'string',
                        'RoleArn': 'string',
                        'RoleSessionName': 'string',
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'DirectKafkaSource': {
                    'Name': 'string',
                    'StreamingOptions': {
                        'BootstrapServers': 'string',
                        'SecurityProtocol': 'string',
                        'ConnectionName': 'string',
                        'TopicName': 'string',
                        'Assign': 'string',
                        'SubscribePattern': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingOffsets': 'string',
                        'EndingOffsets': 'string',
                        'PollTimeoutMs': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxOffsetsPerTrigger': 123,
                        'MinPartitions': 123,
                        'IncludeHeaders': True|False,
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'CatalogKinesisSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'Table': 'string',
                    'Database': 'string',
                    'StreamingOptions': {
                        'EndpointUrl': 'string',
                        'StreamName': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                        'MaxFetchTimeInMs': 123,
                        'MaxFetchRecordsPerShard': 123,
                        'MaxRecordPerRead': 123,
                        'AddIdleTimeBetweenReads': True|False,
                        'IdleTimeBetweenReadsInMs': 123,
                        'DescribeShardInterval': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxRetryIntervalMs': 123,
                        'AvoidEmptyBatches': True|False,
                        'StreamArn': 'string',
                        'RoleArn': 'string',
                        'RoleSessionName': 'string',
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'CatalogKafkaSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'Table': 'string',
                    'Database': 'string',
                    'StreamingOptions': {
                        'BootstrapServers': 'string',
                        'SecurityProtocol': 'string',
                        'ConnectionName': 'string',
                        'TopicName': 'string',
                        'Assign': 'string',
                        'SubscribePattern': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingOffsets': 'string',
                        'EndingOffsets': 'string',
                        'PollTimeoutMs': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxOffsetsPerTrigger': 123,
                        'MinPartitions': 123,
                        'IncludeHeaders': True|False,
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'DropNullFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'NullCheckBoxList': {
                        'IsEmpty': True|False,
                        'IsNullString': True|False,
                        'IsNegOne': True|False
                    },
                    'NullTextList': [
                        {
                            'Value': 'string',
                            'Datatype': {
                                'Id': 'string',
                                'Label': 'string'
                            }
                        },
                    ]
                },
                'Merge': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Source': 'string',
                    'PrimaryKeys': [
                        [
                            'string',
                        ],
                    ]
                },
                'Union': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'UnionType': 'ALL'|'DISTINCT'
                },
                'PIIDetection': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PiiType': 'RowAudit'|'RowMasking'|'ColumnAudit'|'ColumnMasking',
                    'EntityTypesToDetect': [
                        'string',
                    ],
                    'OutputColumnName': 'string',
                    'SampleFraction': 123.0,
                    'ThresholdFraction': 123.0,
                    'MaskValue': 'string'
                },
                'Aggregate': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Groups': [
                        [
                            'string',
                        ],
                    ],
                    'Aggs': [
                        {
                            'Column': [
                                'string',
                            ],
                            'AggFunc': 'avg'|'countDistinct'|'count'|'first'|'last'|'kurtosis'|'max'|'min'|'skewness'|'stddev_samp'|'stddev_pop'|'sum'|'sumDistinct'|'var_samp'|'var_pop'
                        },
                    ]
                },
                'DropDuplicates': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Columns': [
                        [
                            'string',
                        ],
                    ]
                },
                'GovernedCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'GovernedCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'PartitionPredicate': 'string',
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123
                    }
                },
                'MicrosoftSQLServerCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'MySQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'OracleSQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'PostgreSQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'MicrosoftSQLServerCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'MySQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'OracleSQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'PostgreSQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'DynamicTransform': {
                    'Name': 'string',
                    'TransformName': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Parameters': [
                        {
                            'Name': 'string',
                            'Type': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                            'ValidationRule': 'string',
                            'ValidationMessage': 'string',
                            'Value': [
                                'string',
                            ],
                            'ListType': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                            'IsOptional': True|False
                        },
                    ],
                    'FunctionName': 'string',
                    'Path': 'string',
                    'Version': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'EvaluateDataQuality': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Ruleset': 'string',
                    'Output': 'PrimaryInput'|'EvaluationResults',
                    'PublishingOptions': {
                        'EvaluationContext': 'string',
                        'ResultsS3Prefix': 'string',
                        'CloudWatchMetricsEnabled': True|False,
                        'ResultsPublishingEnabled': True|False
                    },
                    'StopJobOnFailureOptions': {
                        'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                    }
                },
                'S3CatalogHudiSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogHudiSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3HudiSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3HudiCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3HudiDirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Path': 'string',
                    'Compression': 'gzip'|'lzo'|'uncompressed'|'snappy',
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'DirectJDBCSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'ConnectionName': 'string',
                    'ConnectionType': 'sqlserver'|'mysql'|'oracle'|'postgresql'|'redshift',
                    'RedshiftTmpDir': 'string'
                },
                'S3CatalogDeltaSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogDeltaSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3DeltaSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3DeltaCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3DeltaDirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'uncompressed'|'snappy',
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'AmazonRedshiftSource': {
                    'Name': 'string',
                    'Data': {
                        'AccessType': 'string',
                        'SourceType': 'string',
                        'Connection': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Schema': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Table': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogDatabase': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogTable': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogRedshiftSchema': 'string',
                        'CatalogRedshiftTable': 'string',
                        'TempDir': 'string',
                        'IamRole': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'AdvancedOptions': [
                            {
                                'Key': 'string',
                                'Value': 'string'
                            },
                        ],
                        'SampleQuery': 'string',
                        'PreAction': 'string',
                        'PostAction': 'string',
                        'Action': 'string',
                        'TablePrefix': 'string',
                        'Upsert': True|False,
                        'MergeAction': 'string',
                        'MergeWhenMatched': 'string',
                        'MergeWhenNotMatched': 'string',
                        'MergeClause': 'string',
                        'CrawlerConnection': 'string',
                        'TableSchema': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ],
                        'StagingTable': 'string',
                        'SelectedColumns': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ]
                    }
                },
                'AmazonRedshiftTarget': {
                    'Name': 'string',
                    'Data': {
                        'AccessType': 'string',
                        'SourceType': 'string',
                        'Connection': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Schema': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Table': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogDatabase': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogTable': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogRedshiftSchema': 'string',
                        'CatalogRedshiftTable': 'string',
                        'TempDir': 'string',
                        'IamRole': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'AdvancedOptions': [
                            {
                                'Key': 'string',
                                'Value': 'string'
                            },
                        ],
                        'SampleQuery': 'string',
                        'PreAction': 'string',
                        'PostAction': 'string',
                        'Action': 'string',
                        'TablePrefix': 'string',
                        'Upsert': True|False,
                        'MergeAction': 'string',
                        'MergeWhenMatched': 'string',
                        'MergeWhenNotMatched': 'string',
                        'MergeClause': 'string',
                        'CrawlerConnection': 'string',
                        'TableSchema': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ],
                        'StagingTable': 'string',
                        'SelectedColumns': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ]
                    },
                    'Inputs': [
                        'string',
                    ]
                },
                'EvaluateDataQualityMultiFrame': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'AdditionalDataSources': {
                        'string': 'string'
                    },
                    'Ruleset': 'string',
                    'PublishingOptions': {
                        'EvaluationContext': 'string',
                        'ResultsS3Prefix': 'string',
                        'CloudWatchMetricsEnabled': True|False,
                        'ResultsPublishingEnabled': True|False
                    },
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'StopJobOnFailureOptions': {
                        'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                    }
                }
            }
        },
        'ExecutionClass': 'FLEX'|'STANDARD',
        'SourceControlDetails': {
            'Provider': 'GITHUB'|'AWS_CODE_COMMIT',
            'Repository': 'string',
            'Owner': 'string',
            'Branch': 'string',
            'Folder': 'string',
            'LastCommitId': 'string',
            'AuthStrategy': 'PERSONAL_ACCESS_TOKEN'|'AWS_SECRETS_MANAGER',
            'AuthToken': 'string'
        }
    }
}


**Response Structure**
::

    # This section is too large to render.
    # Please see the AWS API Documentation linked below.

`AWS API Documentation <https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/GetJob>`_
GetJobs (updated) Link ¶
Changes (response)
{'Jobs': {'CodeGenConfigurationNodes': {'EvaluateDataQualityMultiFrame': {'AdditionalDataSources': {'string': 'string'},
                                                                          'AdditionalOptions': {'performanceTuning.caching': 'string'},
                                                                          'Inputs': ['string'],
                                                                          'Name': 'string',
                                                                          'PublishingOptions': {'CloudWatchMetricsEnabled': 'boolean',
                                                                                                'EvaluationContext': 'string',
                                                                                                'ResultsPublishingEnabled': 'boolean',
                                                                                                'ResultsS3Prefix': 'string'},
                                                                          'Ruleset': 'string',
                                                                          'StopJobOnFailureOptions': {'StopJobOnFailureTiming': 'Immediate '
                                                                                                                                '| '
                                                                                                                                'AfterDataLoad'}}}}}

Retrieves all current job definitions.

See also: AWS API Documentation

Request Syntax

client.get_jobs(
    NextToken='string',
    MaxResults=123
)
type NextToken:

string

param NextToken:

A continuation token, if this is a continuation call.

type MaxResults:

integer

param MaxResults:

The maximum size of the response.

rtype:

dict

returns:

Response Syntax

{
    'Jobs': [
        {
            'Name': 'string',
            'Description': 'string',
            'LogUri': 'string',
            'Role': 'string',
            'CreatedOn': datetime(2015, 1, 1),
            'LastModifiedOn': datetime(2015, 1, 1),
            'ExecutionProperty': {
                'MaxConcurrentRuns': 123
            },
            'Command': {
                'Name': 'string',
                'ScriptLocation': 'string',
                'PythonVersion': 'string'
            },
            'DefaultArguments': {
                'string': 'string'
            },
            'NonOverridableArguments': {
                'string': 'string'
            },
            'Connections': {
                'Connections': [
                    'string',
                ]
            },
            'MaxRetries': 123,
            'AllocatedCapacity': 123,
            'Timeout': 123,
            'MaxCapacity': 123.0,
            'WorkerType': 'Standard'|'G.1X'|'G.2X'|'G.025X'|'G.4X'|'G.8X',
            'NumberOfWorkers': 123,
            'SecurityConfiguration': 'string',
            'NotificationProperty': {
                'NotifyDelayAfter': 123
            },
            'GlueVersion': 'string',
            'CodeGenConfigurationNodes': {
                'string': {
                    'AthenaConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'ConnectionTable': 'string',
                        'SchemaName': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'JDBCConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'FilterPredicate': 'string',
                            'PartitionColumn': 'string',
                            'LowerBound': 123,
                            'UpperBound': 123,
                            'NumPartitions': 123,
                            'JobBookmarkKeys': [
                                'string',
                            ],
                            'JobBookmarkKeysSortOrder': 'string',
                            'DataTypeMapping': {
                                'string': 'DATE'|'STRING'|'TIMESTAMP'|'INT'|'FLOAT'|'LONG'|'BIGDECIMAL'|'BYTE'|'SHORT'|'DOUBLE'
                            }
                        },
                        'ConnectionTable': 'string',
                        'Query': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkConnectorSource': {
                        'Name': 'string',
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'RedshiftSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'RedshiftTmpDir': 'string',
                        'TmpDirIAMRole': 'string'
                    },
                    'S3CatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'PartitionPredicate': 'string',
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123
                        }
                    },
                    'S3CsvSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'gzip'|'bzip2',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'Separator': 'comma'|'ctrla'|'pipe'|'semicolon'|'tab',
                        'Escaper': 'string',
                        'QuoteChar': 'quote'|'quillemet'|'single_quote'|'disabled',
                        'Multiline': True|False,
                        'WithHeader': True|False,
                        'WriteHeader': True|False,
                        'SkipFirst': True|False,
                        'OptimizePerformance': True|False,
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3JsonSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'gzip'|'bzip2',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'JsonPath': 'string',
                        'Multiline': True|False,
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3ParquetSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'CompressionType': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                        'Exclusions': [
                            'string',
                        ],
                        'GroupSize': 'string',
                        'GroupFiles': 'string',
                        'Recurse': True|False,
                        'MaxBand': 123,
                        'MaxFilesInBand': 123,
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'RelationalCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'DynamoDBCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'JDBCConnectorTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ConnectionName': 'string',
                        'ConnectionTable': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkConnectorTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ConnectionName': 'string',
                        'ConnectorName': 'string',
                        'ConnectionType': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'RedshiftTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string',
                        'RedshiftTmpDir': 'string',
                        'TmpDirIAMRole': 'string',
                        'UpsertRedshiftOptions': {
                            'TableLocation': 'string',
                            'ConnectionName': 'string',
                            'UpsertKeys': [
                                'string',
                            ]
                        }
                    },
                    'S3CatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3GlueParquetTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'S3DirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'string',
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'ApplyMapping': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Mapping': [
                            {
                                'ToKey': 'string',
                                'FromPath': [
                                    'string',
                                ],
                                'FromType': 'string',
                                'ToType': 'string',
                                'Dropped': True|False,
                                'Children': {'... recursive ...'}
                            },
                        ]
                    },
                    'SelectFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'DropFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'RenameField': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'SourcePath': [
                            'string',
                        ],
                        'TargetPath': [
                            'string',
                        ]
                    },
                    'Spigot': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Path': 'string',
                        'Topk': 123,
                        'Prob': 123.0
                    },
                    'Join': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'JoinType': 'equijoin'|'left'|'right'|'outer'|'leftsemi'|'leftanti',
                        'Columns': [
                            {
                                'From': 'string',
                                'Keys': [
                                    [
                                        'string',
                                    ],
                                ]
                            },
                        ]
                    },
                    'SplitFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Paths': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'SelectFromCollection': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Index': 123
                    },
                    'FillMissingValues': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'ImputedPath': 'string',
                        'FilledPath': 'string'
                    },
                    'Filter': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'LogicalOperator': 'AND'|'OR',
                        'Filters': [
                            {
                                'Operation': 'EQ'|'LT'|'GT'|'LTE'|'GTE'|'REGEX'|'ISNULL',
                                'Negated': True|False,
                                'Values': [
                                    {
                                        'Type': 'COLUMNEXTRACTED'|'CONSTANT',
                                        'Value': [
                                            'string',
                                        ]
                                    },
                                ]
                            },
                        ]
                    },
                    'CustomCode': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Code': 'string',
                        'ClassName': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'SparkSQL': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'SqlQuery': 'string',
                        'SqlAliases': [
                            {
                                'From': 'string',
                                'Alias': 'string'
                            },
                        ],
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'DirectKinesisSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'StreamingOptions': {
                            'EndpointUrl': 'string',
                            'StreamName': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                            'MaxFetchTimeInMs': 123,
                            'MaxFetchRecordsPerShard': 123,
                            'MaxRecordPerRead': 123,
                            'AddIdleTimeBetweenReads': True|False,
                            'IdleTimeBetweenReadsInMs': 123,
                            'DescribeShardInterval': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxRetryIntervalMs': 123,
                            'AvoidEmptyBatches': True|False,
                            'StreamArn': 'string',
                            'RoleArn': 'string',
                            'RoleSessionName': 'string',
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'DirectKafkaSource': {
                        'Name': 'string',
                        'StreamingOptions': {
                            'BootstrapServers': 'string',
                            'SecurityProtocol': 'string',
                            'ConnectionName': 'string',
                            'TopicName': 'string',
                            'Assign': 'string',
                            'SubscribePattern': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingOffsets': 'string',
                            'EndingOffsets': 'string',
                            'PollTimeoutMs': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxOffsetsPerTrigger': 123,
                            'MinPartitions': 123,
                            'IncludeHeaders': True|False,
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'CatalogKinesisSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'Table': 'string',
                        'Database': 'string',
                        'StreamingOptions': {
                            'EndpointUrl': 'string',
                            'StreamName': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                            'MaxFetchTimeInMs': 123,
                            'MaxFetchRecordsPerShard': 123,
                            'MaxRecordPerRead': 123,
                            'AddIdleTimeBetweenReads': True|False,
                            'IdleTimeBetweenReadsInMs': 123,
                            'DescribeShardInterval': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxRetryIntervalMs': 123,
                            'AvoidEmptyBatches': True|False,
                            'StreamArn': 'string',
                            'RoleArn': 'string',
                            'RoleSessionName': 'string',
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'CatalogKafkaSource': {
                        'Name': 'string',
                        'WindowSize': 123,
                        'DetectSchema': True|False,
                        'Table': 'string',
                        'Database': 'string',
                        'StreamingOptions': {
                            'BootstrapServers': 'string',
                            'SecurityProtocol': 'string',
                            'ConnectionName': 'string',
                            'TopicName': 'string',
                            'Assign': 'string',
                            'SubscribePattern': 'string',
                            'Classification': 'string',
                            'Delimiter': 'string',
                            'StartingOffsets': 'string',
                            'EndingOffsets': 'string',
                            'PollTimeoutMs': 123,
                            'NumRetries': 123,
                            'RetryIntervalMs': 123,
                            'MaxOffsetsPerTrigger': 123,
                            'MinPartitions': 123,
                            'IncludeHeaders': True|False,
                            'AddRecordTimestamp': 'string',
                            'EmitConsumerLagMetrics': 'string'
                        },
                        'DataPreviewOptions': {
                            'PollingTime': 123,
                            'RecordPollingLimit': 123
                        }
                    },
                    'DropNullFields': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'NullCheckBoxList': {
                            'IsEmpty': True|False,
                            'IsNullString': True|False,
                            'IsNegOne': True|False
                        },
                        'NullTextList': [
                            {
                                'Value': 'string',
                                'Datatype': {
                                    'Id': 'string',
                                    'Label': 'string'
                                }
                            },
                        ]
                    },
                    'Merge': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Source': 'string',
                        'PrimaryKeys': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'Union': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'UnionType': 'ALL'|'DISTINCT'
                    },
                    'PIIDetection': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PiiType': 'RowAudit'|'RowMasking'|'ColumnAudit'|'ColumnMasking',
                        'EntityTypesToDetect': [
                            'string',
                        ],
                        'OutputColumnName': 'string',
                        'SampleFraction': 123.0,
                        'ThresholdFraction': 123.0,
                        'MaskValue': 'string'
                    },
                    'Aggregate': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Groups': [
                            [
                                'string',
                            ],
                        ],
                        'Aggs': [
                            {
                                'Column': [
                                    'string',
                                ],
                                'AggFunc': 'avg'|'countDistinct'|'count'|'first'|'last'|'kurtosis'|'max'|'min'|'skewness'|'stddev_samp'|'stddev_pop'|'sum'|'sumDistinct'|'var_samp'|'var_pop'
                            },
                        ]
                    },
                    'DropDuplicates': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Columns': [
                            [
                                'string',
                            ],
                        ]
                    },
                    'GovernedCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'GovernedCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'PartitionPredicate': 'string',
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123
                        }
                    },
                    'MicrosoftSQLServerCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MySQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'OracleSQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'PostgreSQLCatalogSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MicrosoftSQLServerCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'MySQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'OracleSQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'PostgreSQLCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Database': 'string',
                        'Table': 'string'
                    },
                    'DynamicTransform': {
                        'Name': 'string',
                        'TransformName': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Parameters': [
                            {
                                'Name': 'string',
                                'Type': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                                'ValidationRule': 'string',
                                'ValidationMessage': 'string',
                                'Value': [
                                    'string',
                                ],
                                'ListType': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                                'IsOptional': True|False
                            },
                        ],
                        'FunctionName': 'string',
                        'Path': 'string',
                        'Version': 'string',
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'EvaluateDataQuality': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Ruleset': 'string',
                        'Output': 'PrimaryInput'|'EvaluationResults',
                        'PublishingOptions': {
                            'EvaluationContext': 'string',
                            'ResultsS3Prefix': 'string',
                            'CloudWatchMetricsEnabled': True|False,
                            'ResultsPublishingEnabled': True|False
                        },
                        'StopJobOnFailureOptions': {
                            'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                        }
                    },
                    'S3CatalogHudiSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogHudiSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3HudiSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'AdditionalHudiOptions': {
                            'string': 'string'
                        },
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3HudiCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3HudiDirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'Path': 'string',
                        'Compression': 'gzip'|'lzo'|'uncompressed'|'snappy',
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'DirectJDBCSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'ConnectionName': 'string',
                        'ConnectionType': 'sqlserver'|'mysql'|'oracle'|'postgresql'|'redshift',
                        'RedshiftTmpDir': 'string'
                    },
                    'S3CatalogDeltaSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'CatalogDeltaSource': {
                        'Name': 'string',
                        'Database': 'string',
                        'Table': 'string',
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3DeltaSource': {
                        'Name': 'string',
                        'Paths': [
                            'string',
                        ],
                        'AdditionalDeltaOptions': {
                            'string': 'string'
                        },
                        'AdditionalOptions': {
                            'BoundedSize': 123,
                            'BoundedFiles': 123,
                            'EnableSamplePath': True|False,
                            'SamplePath': 'string'
                        },
                        'OutputSchemas': [
                            {
                                'Columns': [
                                    {
                                        'Name': 'string',
                                        'Type': 'string'
                                    },
                                ]
                            },
                        ]
                    },
                    'S3DeltaCatalogTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Table': 'string',
                        'Database': 'string',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                        }
                    },
                    'S3DeltaDirectTarget': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'PartitionKeys': [
                            [
                                'string',
                            ],
                        ],
                        'Path': 'string',
                        'Compression': 'uncompressed'|'snappy',
                        'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'SchemaChangePolicy': {
                            'EnableUpdateCatalog': True|False,
                            'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                            'Table': 'string',
                            'Database': 'string'
                        }
                    },
                    'AmazonRedshiftSource': {
                        'Name': 'string',
                        'Data': {
                            'AccessType': 'string',
                            'SourceType': 'string',
                            'Connection': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Schema': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Table': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogDatabase': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogTable': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogRedshiftSchema': 'string',
                            'CatalogRedshiftTable': 'string',
                            'TempDir': 'string',
                            'IamRole': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'AdvancedOptions': [
                                {
                                    'Key': 'string',
                                    'Value': 'string'
                                },
                            ],
                            'SampleQuery': 'string',
                            'PreAction': 'string',
                            'PostAction': 'string',
                            'Action': 'string',
                            'TablePrefix': 'string',
                            'Upsert': True|False,
                            'MergeAction': 'string',
                            'MergeWhenMatched': 'string',
                            'MergeWhenNotMatched': 'string',
                            'MergeClause': 'string',
                            'CrawlerConnection': 'string',
                            'TableSchema': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ],
                            'StagingTable': 'string',
                            'SelectedColumns': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ]
                        }
                    },
                    'AmazonRedshiftTarget': {
                        'Name': 'string',
                        'Data': {
                            'AccessType': 'string',
                            'SourceType': 'string',
                            'Connection': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Schema': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'Table': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogDatabase': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogTable': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'CatalogRedshiftSchema': 'string',
                            'CatalogRedshiftTable': 'string',
                            'TempDir': 'string',
                            'IamRole': {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                            'AdvancedOptions': [
                                {
                                    'Key': 'string',
                                    'Value': 'string'
                                },
                            ],
                            'SampleQuery': 'string',
                            'PreAction': 'string',
                            'PostAction': 'string',
                            'Action': 'string',
                            'TablePrefix': 'string',
                            'Upsert': True|False,
                            'MergeAction': 'string',
                            'MergeWhenMatched': 'string',
                            'MergeWhenNotMatched': 'string',
                            'MergeClause': 'string',
                            'CrawlerConnection': 'string',
                            'TableSchema': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ],
                            'StagingTable': 'string',
                            'SelectedColumns': [
                                {
                                    'Value': 'string',
                                    'Label': 'string',
                                    'Description': 'string'
                                },
                            ]
                        },
                        'Inputs': [
                            'string',
                        ]
                    },
                    'EvaluateDataQualityMultiFrame': {
                        'Name': 'string',
                        'Inputs': [
                            'string',
                        ],
                        'AdditionalDataSources': {
                            'string': 'string'
                        },
                        'Ruleset': 'string',
                        'PublishingOptions': {
                            'EvaluationContext': 'string',
                            'ResultsS3Prefix': 'string',
                            'CloudWatchMetricsEnabled': True|False,
                            'ResultsPublishingEnabled': True|False
                        },
                        'AdditionalOptions': {
                            'string': 'string'
                        },
                        'StopJobOnFailureOptions': {
                            'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                        }
                    }
                }
            },
            'ExecutionClass': 'FLEX'|'STANDARD',
            'SourceControlDetails': {
                'Provider': 'GITHUB'|'AWS_CODE_COMMIT',
                'Repository': 'string',
                'Owner': 'string',
                'Branch': 'string',
                'Folder': 'string',
                'LastCommitId': 'string',
                'AuthStrategy': 'PERSONAL_ACCESS_TOKEN'|'AWS_SECRETS_MANAGER',
                'AuthToken': 'string'
            }
        },
    ],
    'NextToken': 'string'
}


**Response Structure**
::

    # This section is too large to render.
    # Please see the AWS API Documentation linked below.

`AWS API Documentation <https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/GetJobs>`_
ListDataQualityRulesets (updated) Link ¶
Changes (request, response)
Request
{'Filter': {'TargetTable': {'CatalogId': 'string'}}}
Response
{'Rulesets': {'TargetTable': {'CatalogId': 'string'}}}

Returns a paginated list of rulesets for the specified list of Glue tables.

See also: AWS API Documentation

Request Syntax

client.list_data_quality_rulesets(
    NextToken='string',
    MaxResults=123,
    Filter={
        'Name': 'string',
        'Description': 'string',
        'CreatedBefore': datetime(2015, 1, 1),
        'CreatedAfter': datetime(2015, 1, 1),
        'LastModifiedBefore': datetime(2015, 1, 1),
        'LastModifiedAfter': datetime(2015, 1, 1),
        'TargetTable': {
            'TableName': 'string',
            'DatabaseName': 'string',
            'CatalogId': 'string'
        }
    },
    Tags={
        'string': 'string'
    }
)
type NextToken:

string

param NextToken:

A paginated token to offset the results.

type MaxResults:

integer

param MaxResults:

The maximum number of results to return.

type Filter:

dict

param Filter:

The filter criteria.

  • Name (string) --

    The name of the ruleset filter criteria.

  • Description (string) --

    The description of the ruleset filter criteria.

  • CreatedBefore (datetime) --

    Filter on rulesets created before this date.

  • CreatedAfter (datetime) --

    Filter on rulesets created after this date.

  • LastModifiedBefore (datetime) --

    Filter on rulesets last modified before this date.

  • LastModifiedAfter (datetime) --

    Filter on rulesets last modified after this date.

  • TargetTable (dict) --

    The name and database name of the target table.

    • TableName (string) -- [REQUIRED]

      The name of the Glue table.

    • DatabaseName (string) -- [REQUIRED]

      The name of the database where the Glue table exists.

    • CatalogId (string) --

      The catalog id where the Glue table exists.

type Tags:

dict

param Tags:

A list of key-value pair tags.

  • (string) --

    • (string) --

rtype:

dict

returns:

Response Syntax

{
    'Rulesets': [
        {
            'Name': 'string',
            'Description': 'string',
            'CreatedOn': datetime(2015, 1, 1),
            'LastModifiedOn': datetime(2015, 1, 1),
            'TargetTable': {
                'TableName': 'string',
                'DatabaseName': 'string',
                'CatalogId': 'string'
            },
            'RecommendationRunId': 'string',
            'RuleCount': 123
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • Rulesets (list) --

      A paginated list of rulesets for the specified list of Glue tables.

      • (dict) --

        Describes a data quality ruleset returned by GetDataQualityRuleset.

        • Name (string) --

          The name of the data quality ruleset.

        • Description (string) --

          A description of the data quality ruleset.

        • CreatedOn (datetime) --

          The date and time the data quality ruleset was created.

        • LastModifiedOn (datetime) --

          The date and time the data quality ruleset was last modified.

        • TargetTable (dict) --

          An object representing an Glue table.

          • TableName (string) --

            The name of the Glue table.

          • DatabaseName (string) --

            The name of the database where the Glue table exists.

          • CatalogId (string) --

            The catalog id where the Glue table exists.

        • RecommendationRunId (string) --

          When a ruleset was created from a recommendation run, this run ID is generated to link the two together.

        • RuleCount (integer) --

          The number of rules in the ruleset.

    • NextToken (string) --

      A pagination token, if more results are available.

StartDataQualityRulesetEvaluationRun (updated) Link ¶
Changes (request)
{'AdditionalDataSources': {'string': {'GlueTable': {'AdditionalOptions': {'string': 'string'},
                                                    'CatalogId': 'string',
                                                    'ConnectionName': 'string',
                                                    'DatabaseName': 'string',
                                                    'TableName': 'string'}}}}

Once you have a ruleset definition (either recommended or your own), you call this operation to evaluate the ruleset against a data source (Glue table). The evaluation computes results which you can retrieve with the GetDataQualityResult API.

See also: AWS API Documentation

Request Syntax

client.start_data_quality_ruleset_evaluation_run(
    DataSource={
        'GlueTable': {
            'DatabaseName': 'string',
            'TableName': 'string',
            'CatalogId': 'string',
            'ConnectionName': 'string',
            'AdditionalOptions': {
                'string': 'string'
            }
        }
    },
    Role='string',
    NumberOfWorkers=123,
    Timeout=123,
    ClientToken='string',
    AdditionalRunOptions={
        'CloudWatchMetricsEnabled': True|False,
        'ResultsS3Prefix': 'string'
    },
    RulesetNames=[
        'string',
    ],
    AdditionalDataSources={
        'string': {
            'GlueTable': {
                'DatabaseName': 'string',
                'TableName': 'string',
                'CatalogId': 'string',
                'ConnectionName': 'string',
                'AdditionalOptions': {
                    'string': 'string'
                }
            }
        }
    }
)
type DataSource:

dict

param DataSource:

[REQUIRED]

The data source (Glue table) associated with this run.

  • GlueTable (dict) -- [REQUIRED]

    An Glue table.

    • DatabaseName (string) -- [REQUIRED]

      A database name in the Glue Data Catalog.

    • TableName (string) -- [REQUIRED]

      A table name in the Glue Data Catalog.

    • CatalogId (string) --

      A unique identifier for the Glue Data Catalog.

    • ConnectionName (string) --

      The name of the connection to the Glue Data Catalog.

    • AdditionalOptions (dict) --

      Additional options for the table. Currently there are two keys supported:

      • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

      • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

      • (string) --

        • (string) --

type Role:

string

param Role:

[REQUIRED]

An IAM role supplied to encrypt the results of the run.

type NumberOfWorkers:

integer

param NumberOfWorkers:

The number of G.1X workers to be used in the run. The default is 5.

type Timeout:

integer

param Timeout:

The timeout for a run in minutes. This is the maximum time that a run can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).

type ClientToken:

string

param ClientToken:

Used for idempotency and is recommended to be set to a random ID (such as a UUID) to avoid creating or starting multiple instances of the same resource.

type AdditionalRunOptions:

dict

param AdditionalRunOptions:

Additional run options you can specify for an evaluation run.

  • CloudWatchMetricsEnabled (boolean) --

    Whether or not to enable CloudWatch metrics.

  • ResultsS3Prefix (string) --

    Prefix for Amazon S3 to store results.

type RulesetNames:

list

param RulesetNames:

[REQUIRED]

A list of ruleset names.

  • (string) --

type AdditionalDataSources:

dict

param AdditionalDataSources:

A map of reference strings to additional data sources you can specify for an evaluation run.

  • (string) --

    • (dict) --

      A data source (an Glue table) for which you want data quality results.

      • GlueTable (dict) -- [REQUIRED]

        An Glue table.

        • DatabaseName (string) -- [REQUIRED]

          A database name in the Glue Data Catalog.

        • TableName (string) -- [REQUIRED]

          A table name in the Glue Data Catalog.

        • CatalogId (string) --

          A unique identifier for the Glue Data Catalog.

        • ConnectionName (string) --

          The name of the connection to the Glue Data Catalog.

        • AdditionalOptions (dict) --

          Additional options for the table. Currently there are two keys supported:

          • pushDownPredicate: to filter on partitions without having to list and read all the files in your dataset.

          • catalogPartitionPredicate: to use server-side partition pruning using partition indexes in the Glue Data Catalog.

          • (string) --

            • (string) --

rtype:

dict

returns:

Response Syntax

{
    'RunId': 'string'
}

Response Structure

  • (dict) --

    • RunId (string) --

      The unique run identifier associated with this run.

UpdateJob (updated) Link ¶
Changes (request)
{'JobUpdate': {'CodeGenConfigurationNodes': {'EvaluateDataQualityMultiFrame': {'AdditionalDataSources': {'string': 'string'},
                                                                               'AdditionalOptions': {'performanceTuning.caching': 'string'},
                                                                               'Inputs': ['string'],
                                                                               'Name': 'string',
                                                                               'PublishingOptions': {'CloudWatchMetricsEnabled': 'boolean',
                                                                                                     'EvaluationContext': 'string',
                                                                                                     'ResultsPublishingEnabled': 'boolean',
                                                                                                     'ResultsS3Prefix': 'string'},
                                                                               'Ruleset': 'string',
                                                                               'StopJobOnFailureOptions': {'StopJobOnFailureTiming': 'Immediate '
                                                                                                                                     '| '
                                                                                                                                     'AfterDataLoad'}}}}}

Updates an existing job definition. The previous job definition is completely overwritten by this information.

See also: AWS API Documentation

Request Syntax

client.update_job(
    JobName='string',
    JobUpdate={
        'Description': 'string',
        'LogUri': 'string',
        'Role': 'string',
        'ExecutionProperty': {
            'MaxConcurrentRuns': 123
        },
        'Command': {
            'Name': 'string',
            'ScriptLocation': 'string',
            'PythonVersion': 'string'
        },
        'DefaultArguments': {
            'string': 'string'
        },
        'NonOverridableArguments': {
            'string': 'string'
        },
        'Connections': {
            'Connections': [
                'string',
            ]
        },
        'MaxRetries': 123,
        'AllocatedCapacity': 123,
        'Timeout': 123,
        'MaxCapacity': 123.0,
        'WorkerType': 'Standard'|'G.1X'|'G.2X'|'G.025X'|'G.4X'|'G.8X',
        'NumberOfWorkers': 123,
        'SecurityConfiguration': 'string',
        'NotificationProperty': {
            'NotifyDelayAfter': 123
        },
        'GlueVersion': 'string',
        'CodeGenConfigurationNodes': {
            'string': {
                'AthenaConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'ConnectionTable': 'string',
                    'SchemaName': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'JDBCConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'FilterPredicate': 'string',
                        'PartitionColumn': 'string',
                        'LowerBound': 123,
                        'UpperBound': 123,
                        'NumPartitions': 123,
                        'JobBookmarkKeys': [
                            'string',
                        ],
                        'JobBookmarkKeysSortOrder': 'string',
                        'DataTypeMapping': {
                            'string': 'DATE'|'STRING'|'TIMESTAMP'|'INT'|'FLOAT'|'LONG'|'BIGDECIMAL'|'BYTE'|'SHORT'|'DOUBLE'
                        }
                    },
                    'ConnectionTable': 'string',
                    'Query': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkConnectorSource': {
                    'Name': 'string',
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'RedshiftSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'RedshiftTmpDir': 'string',
                    'TmpDirIAMRole': 'string'
                },
                'S3CatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'PartitionPredicate': 'string',
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123
                    }
                },
                'S3CsvSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'gzip'|'bzip2',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'Separator': 'comma'|'ctrla'|'pipe'|'semicolon'|'tab',
                    'Escaper': 'string',
                    'QuoteChar': 'quote'|'quillemet'|'single_quote'|'disabled',
                    'Multiline': True|False,
                    'WithHeader': True|False,
                    'WriteHeader': True|False,
                    'SkipFirst': True|False,
                    'OptimizePerformance': True|False,
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3JsonSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'gzip'|'bzip2',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'JsonPath': 'string',
                    'Multiline': True|False,
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3ParquetSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'CompressionType': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                    'Exclusions': [
                        'string',
                    ],
                    'GroupSize': 'string',
                    'GroupFiles': 'string',
                    'Recurse': True|False,
                    'MaxBand': 123,
                    'MaxFilesInBand': 123,
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'RelationalCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'DynamoDBCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'JDBCConnectorTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ConnectionName': 'string',
                    'ConnectionTable': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkConnectorTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ConnectionName': 'string',
                    'ConnectorName': 'string',
                    'ConnectionType': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'RedshiftTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string',
                    'RedshiftTmpDir': 'string',
                    'TmpDirIAMRole': 'string',
                    'UpsertRedshiftOptions': {
                        'TableLocation': 'string',
                        'ConnectionName': 'string',
                        'UpsertKeys': [
                            'string',
                        ]
                    }
                },
                'S3CatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3GlueParquetTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'snappy'|'lzo'|'gzip'|'uncompressed'|'none',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'S3DirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'string',
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'ApplyMapping': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Mapping': [
                        {
                            'ToKey': 'string',
                            'FromPath': [
                                'string',
                            ],
                            'FromType': 'string',
                            'ToType': 'string',
                            'Dropped': True|False,
                            'Children': {'... recursive ...'}
                        },
                    ]
                },
                'SelectFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'DropFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'RenameField': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'SourcePath': [
                        'string',
                    ],
                    'TargetPath': [
                        'string',
                    ]
                },
                'Spigot': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Path': 'string',
                    'Topk': 123,
                    'Prob': 123.0
                },
                'Join': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'JoinType': 'equijoin'|'left'|'right'|'outer'|'leftsemi'|'leftanti',
                    'Columns': [
                        {
                            'From': 'string',
                            'Keys': [
                                [
                                    'string',
                                ],
                            ]
                        },
                    ]
                },
                'SplitFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Paths': [
                        [
                            'string',
                        ],
                    ]
                },
                'SelectFromCollection': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Index': 123
                },
                'FillMissingValues': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'ImputedPath': 'string',
                    'FilledPath': 'string'
                },
                'Filter': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'LogicalOperator': 'AND'|'OR',
                    'Filters': [
                        {
                            'Operation': 'EQ'|'LT'|'GT'|'LTE'|'GTE'|'REGEX'|'ISNULL',
                            'Negated': True|False,
                            'Values': [
                                {
                                    'Type': 'COLUMNEXTRACTED'|'CONSTANT',
                                    'Value': [
                                        'string',
                                    ]
                                },
                            ]
                        },
                    ]
                },
                'CustomCode': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Code': 'string',
                    'ClassName': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'SparkSQL': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'SqlQuery': 'string',
                    'SqlAliases': [
                        {
                            'From': 'string',
                            'Alias': 'string'
                        },
                    ],
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'DirectKinesisSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'StreamingOptions': {
                        'EndpointUrl': 'string',
                        'StreamName': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                        'MaxFetchTimeInMs': 123,
                        'MaxFetchRecordsPerShard': 123,
                        'MaxRecordPerRead': 123,
                        'AddIdleTimeBetweenReads': True|False,
                        'IdleTimeBetweenReadsInMs': 123,
                        'DescribeShardInterval': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxRetryIntervalMs': 123,
                        'AvoidEmptyBatches': True|False,
                        'StreamArn': 'string',
                        'RoleArn': 'string',
                        'RoleSessionName': 'string',
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'DirectKafkaSource': {
                    'Name': 'string',
                    'StreamingOptions': {
                        'BootstrapServers': 'string',
                        'SecurityProtocol': 'string',
                        'ConnectionName': 'string',
                        'TopicName': 'string',
                        'Assign': 'string',
                        'SubscribePattern': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingOffsets': 'string',
                        'EndingOffsets': 'string',
                        'PollTimeoutMs': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxOffsetsPerTrigger': 123,
                        'MinPartitions': 123,
                        'IncludeHeaders': True|False,
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'CatalogKinesisSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'Table': 'string',
                    'Database': 'string',
                    'StreamingOptions': {
                        'EndpointUrl': 'string',
                        'StreamName': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingPosition': 'latest'|'trim_horizon'|'earliest',
                        'MaxFetchTimeInMs': 123,
                        'MaxFetchRecordsPerShard': 123,
                        'MaxRecordPerRead': 123,
                        'AddIdleTimeBetweenReads': True|False,
                        'IdleTimeBetweenReadsInMs': 123,
                        'DescribeShardInterval': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxRetryIntervalMs': 123,
                        'AvoidEmptyBatches': True|False,
                        'StreamArn': 'string',
                        'RoleArn': 'string',
                        'RoleSessionName': 'string',
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'CatalogKafkaSource': {
                    'Name': 'string',
                    'WindowSize': 123,
                    'DetectSchema': True|False,
                    'Table': 'string',
                    'Database': 'string',
                    'StreamingOptions': {
                        'BootstrapServers': 'string',
                        'SecurityProtocol': 'string',
                        'ConnectionName': 'string',
                        'TopicName': 'string',
                        'Assign': 'string',
                        'SubscribePattern': 'string',
                        'Classification': 'string',
                        'Delimiter': 'string',
                        'StartingOffsets': 'string',
                        'EndingOffsets': 'string',
                        'PollTimeoutMs': 123,
                        'NumRetries': 123,
                        'RetryIntervalMs': 123,
                        'MaxOffsetsPerTrigger': 123,
                        'MinPartitions': 123,
                        'IncludeHeaders': True|False,
                        'AddRecordTimestamp': 'string',
                        'EmitConsumerLagMetrics': 'string'
                    },
                    'DataPreviewOptions': {
                        'PollingTime': 123,
                        'RecordPollingLimit': 123
                    }
                },
                'DropNullFields': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'NullCheckBoxList': {
                        'IsEmpty': True|False,
                        'IsNullString': True|False,
                        'IsNegOne': True|False
                    },
                    'NullTextList': [
                        {
                            'Value': 'string',
                            'Datatype': {
                                'Id': 'string',
                                'Label': 'string'
                            }
                        },
                    ]
                },
                'Merge': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Source': 'string',
                    'PrimaryKeys': [
                        [
                            'string',
                        ],
                    ]
                },
                'Union': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'UnionType': 'ALL'|'DISTINCT'
                },
                'PIIDetection': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PiiType': 'RowAudit'|'RowMasking'|'ColumnAudit'|'ColumnMasking',
                    'EntityTypesToDetect': [
                        'string',
                    ],
                    'OutputColumnName': 'string',
                    'SampleFraction': 123.0,
                    'ThresholdFraction': 123.0,
                    'MaskValue': 'string'
                },
                'Aggregate': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Groups': [
                        [
                            'string',
                        ],
                    ],
                    'Aggs': [
                        {
                            'Column': [
                                'string',
                            ],
                            'AggFunc': 'avg'|'countDistinct'|'count'|'first'|'last'|'kurtosis'|'max'|'min'|'skewness'|'stddev_samp'|'stddev_pop'|'sum'|'sumDistinct'|'var_samp'|'var_pop'
                        },
                    ]
                },
                'DropDuplicates': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Columns': [
                        [
                            'string',
                        ],
                    ]
                },
                'GovernedCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'GovernedCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'PartitionPredicate': 'string',
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123
                    }
                },
                'MicrosoftSQLServerCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'MySQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'OracleSQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'PostgreSQLCatalogSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string'
                },
                'MicrosoftSQLServerCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'MySQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'OracleSQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'PostgreSQLCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Database': 'string',
                    'Table': 'string'
                },
                'DynamicTransform': {
                    'Name': 'string',
                    'TransformName': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Parameters': [
                        {
                            'Name': 'string',
                            'Type': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                            'ValidationRule': 'string',
                            'ValidationMessage': 'string',
                            'Value': [
                                'string',
                            ],
                            'ListType': 'str'|'int'|'float'|'complex'|'bool'|'list'|'null',
                            'IsOptional': True|False
                        },
                    ],
                    'FunctionName': 'string',
                    'Path': 'string',
                    'Version': 'string',
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'EvaluateDataQuality': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Ruleset': 'string',
                    'Output': 'PrimaryInput'|'EvaluationResults',
                    'PublishingOptions': {
                        'EvaluationContext': 'string',
                        'ResultsS3Prefix': 'string',
                        'CloudWatchMetricsEnabled': True|False,
                        'ResultsPublishingEnabled': True|False
                    },
                    'StopJobOnFailureOptions': {
                        'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                    }
                },
                'S3CatalogHudiSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogHudiSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3HudiSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'AdditionalHudiOptions': {
                        'string': 'string'
                    },
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3HudiCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3HudiDirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'Path': 'string',
                    'Compression': 'gzip'|'lzo'|'uncompressed'|'snappy',
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'DirectJDBCSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'ConnectionName': 'string',
                    'ConnectionType': 'sqlserver'|'mysql'|'oracle'|'postgresql'|'redshift',
                    'RedshiftTmpDir': 'string'
                },
                'S3CatalogDeltaSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'CatalogDeltaSource': {
                    'Name': 'string',
                    'Database': 'string',
                    'Table': 'string',
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3DeltaSource': {
                    'Name': 'string',
                    'Paths': [
                        'string',
                    ],
                    'AdditionalDeltaOptions': {
                        'string': 'string'
                    },
                    'AdditionalOptions': {
                        'BoundedSize': 123,
                        'BoundedFiles': 123,
                        'EnableSamplePath': True|False,
                        'SamplePath': 'string'
                    },
                    'OutputSchemas': [
                        {
                            'Columns': [
                                {
                                    'Name': 'string',
                                    'Type': 'string'
                                },
                            ]
                        },
                    ]
                },
                'S3DeltaCatalogTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Table': 'string',
                    'Database': 'string',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG'
                    }
                },
                'S3DeltaDirectTarget': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'PartitionKeys': [
                        [
                            'string',
                        ],
                    ],
                    'Path': 'string',
                    'Compression': 'uncompressed'|'snappy',
                    'Format': 'json'|'csv'|'avro'|'orc'|'parquet'|'hudi'|'delta',
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'SchemaChangePolicy': {
                        'EnableUpdateCatalog': True|False,
                        'UpdateBehavior': 'UPDATE_IN_DATABASE'|'LOG',
                        'Table': 'string',
                        'Database': 'string'
                    }
                },
                'AmazonRedshiftSource': {
                    'Name': 'string',
                    'Data': {
                        'AccessType': 'string',
                        'SourceType': 'string',
                        'Connection': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Schema': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Table': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogDatabase': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogTable': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogRedshiftSchema': 'string',
                        'CatalogRedshiftTable': 'string',
                        'TempDir': 'string',
                        'IamRole': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'AdvancedOptions': [
                            {
                                'Key': 'string',
                                'Value': 'string'
                            },
                        ],
                        'SampleQuery': 'string',
                        'PreAction': 'string',
                        'PostAction': 'string',
                        'Action': 'string',
                        'TablePrefix': 'string',
                        'Upsert': True|False,
                        'MergeAction': 'string',
                        'MergeWhenMatched': 'string',
                        'MergeWhenNotMatched': 'string',
                        'MergeClause': 'string',
                        'CrawlerConnection': 'string',
                        'TableSchema': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ],
                        'StagingTable': 'string',
                        'SelectedColumns': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ]
                    }
                },
                'AmazonRedshiftTarget': {
                    'Name': 'string',
                    'Data': {
                        'AccessType': 'string',
                        'SourceType': 'string',
                        'Connection': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Schema': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'Table': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogDatabase': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogTable': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'CatalogRedshiftSchema': 'string',
                        'CatalogRedshiftTable': 'string',
                        'TempDir': 'string',
                        'IamRole': {
                            'Value': 'string',
                            'Label': 'string',
                            'Description': 'string'
                        },
                        'AdvancedOptions': [
                            {
                                'Key': 'string',
                                'Value': 'string'
                            },
                        ],
                        'SampleQuery': 'string',
                        'PreAction': 'string',
                        'PostAction': 'string',
                        'Action': 'string',
                        'TablePrefix': 'string',
                        'Upsert': True|False,
                        'MergeAction': 'string',
                        'MergeWhenMatched': 'string',
                        'MergeWhenNotMatched': 'string',
                        'MergeClause': 'string',
                        'CrawlerConnection': 'string',
                        'TableSchema': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ],
                        'StagingTable': 'string',
                        'SelectedColumns': [
                            {
                                'Value': 'string',
                                'Label': 'string',
                                'Description': 'string'
                            },
                        ]
                    },
                    'Inputs': [
                        'string',
                    ]
                },
                'EvaluateDataQualityMultiFrame': {
                    'Name': 'string',
                    'Inputs': [
                        'string',
                    ],
                    'AdditionalDataSources': {
                        'string': 'string'
                    },
                    'Ruleset': 'string',
                    'PublishingOptions': {
                        'EvaluationContext': 'string',
                        'ResultsS3Prefix': 'string',
                        'CloudWatchMetricsEnabled': True|False,
                        'ResultsPublishingEnabled': True|False
                    },
                    'AdditionalOptions': {
                        'string': 'string'
                    },
                    'StopJobOnFailureOptions': {
                        'StopJobOnFailureTiming': 'Immediate'|'AfterDataLoad'
                    }
                }
            }
        },
        'ExecutionClass': 'FLEX'|'STANDARD',
        'SourceControlDetails': {
            'Provider': 'GITHUB'|'AWS_CODE_COMMIT',
            'Repository': 'string',
            'Owner': 'string',
            'Branch': 'string',
            'Folder': 'string',
            'LastCommitId': 'string',
            'AuthStrategy': 'PERSONAL_ACCESS_TOKEN'|'AWS_SECRETS_MANAGER',
            'AuthToken': 'string'
        }
    }
)


  **Parameters**
  ::

      # This section is too large to render.
      # Please see the AWS API Documentation linked below.

  `AWS API Documentation <https://docs.aws.amazon.com/goto/WebAPI/glue-2017-03-31/UpdateJob>`_
rtype:

dict

returns:

Response Syntax

{
    'JobName': 'string'
}

Response Structure

  • (dict) --

    • JobName (string) --

      Returns the name of the updated job definition.