2020/08/07 - AWS Glue - 9 updated api methods
Changes AWS Glue now adds support for Network connection type enabling you to access resources inside your VPC using Glue crawlers and Glue ETL jobs.
{'Crawlers': {'Targets': {'S3Targets': {'ConnectionName': 'string'}}}}
Returns a list of resource metadata for a given list of crawler names. After calling the ListCrawlers operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
See also: AWS API Documentation
Request Syntax
client.batch_get_crawlers(
CrawlerNames=[
'string',
]
)
list
[REQUIRED]
A list of crawler names, which might be the names returned from the ListCrawlers operation.
(string) --
dict
Response Syntax
{
'Crawlers': [
{
'Name': 'string',
'Role': 'string',
'Targets': {
'S3Targets': [
{
'Path': 'string',
'Exclusions': [
'string',
],
'ConnectionName': 'string'
},
],
'JdbcTargets': [
{
'ConnectionName': 'string',
'Path': 'string',
'Exclusions': [
'string',
]
},
],
'DynamoDBTargets': [
{
'Path': 'string',
'scanAll': True|False,
'scanRate': 123.0
},
],
'CatalogTargets': [
{
'DatabaseName': 'string',
'Tables': [
'string',
]
},
]
},
'DatabaseName': 'string',
'Description': 'string',
'Classifiers': [
'string',
],
'SchemaChangePolicy': {
'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
},
'State': 'READY'|'RUNNING'|'STOPPING',
'TablePrefix': 'string',
'Schedule': {
'ScheduleExpression': 'string',
'State': 'SCHEDULED'|'NOT_SCHEDULED'|'TRANSITIONING'
},
'CrawlElapsedTime': 123,
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'LastCrawl': {
'Status': 'SUCCEEDED'|'CANCELLED'|'FAILED',
'ErrorMessage': 'string',
'LogGroup': 'string',
'LogStream': 'string',
'MessagePrefix': 'string',
'StartTime': datetime(2015, 1, 1)
},
'Version': 123,
'Configuration': 'string',
'CrawlerSecurityConfiguration': 'string'
},
],
'CrawlersNotFound': [
'string',
]
}
Response Structure
(dict) --
Crawlers (list) --
A list of crawler definitions.
(dict) --
Specifies a crawler program that examines a data source and uses classifiers to try to determine its schema. If successful, the crawler records metadata concerning the data source in the AWS Glue Data Catalog.
Name (string) --
The name of the crawler.
Role (string) --
The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.
Targets (dict) --
A collection of targets to crawl.
S3Targets (list) --
Specifies Amazon Simple Storage Service (Amazon S3) targets.
(dict) --
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Path (string) --
The path to the Amazon S3 target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
ConnectionName (string) --
The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).
JdbcTargets (list) --
Specifies JDBC targets.
(dict) --
Specifies a JDBC data store to crawl.
ConnectionName (string) --
The name of the connection to use to connect to the JDBC target.
Path (string) --
The path of the JDBC target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
DynamoDBTargets (list) --
Specifies Amazon DynamoDB targets.
(dict) --
Specifies an Amazon DynamoDB table to crawl.
Path (string) --
The name of the DynamoDB table to crawl.
scanAll (boolean) --
Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.
A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true .
scanRate (float) --
The percentage of the configured read capacity units to use by the AWS Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.
The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).
CatalogTargets (list) --
Specifies AWS Glue Data Catalog targets.
(dict) --
Specifies an AWS Glue Data Catalog target.
DatabaseName (string) --
The name of the database to be synchronized.
Tables (list) --
A list of the tables to be synchronized.
(string) --
DatabaseName (string) --
The name of the database in which the crawler's output is stored.
Description (string) --
A description of the crawler.
Classifiers (list) --
A list of UTF-8 strings that specify the custom classifiers that are associated with the crawler.
(string) --
SchemaChangePolicy (dict) --
The policy that specifies update and delete behaviors for the crawler.
UpdateBehavior (string) --
The update behavior when the crawler finds a changed schema.
DeleteBehavior (string) --
The deletion behavior when the crawler finds a deleted object.
State (string) --
Indicates whether the crawler is running, or whether a run is pending.
TablePrefix (string) --
The prefix added to the names of tables that are created.
Schedule (dict) --
For scheduled crawlers, the schedule when the crawler runs.
ScheduleExpression (string) --
A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *) .
State (string) --
The state of the schedule.
CrawlElapsedTime (integer) --
If the crawler is running, contains the total time elapsed since the last crawl began.
CreationTime (datetime) --
The time that the crawler was created.
LastUpdated (datetime) --
The time that the crawler was last updated.
LastCrawl (dict) --
The status of the last crawl, and potentially error information if an error occurred.
Status (string) --
Status of the last crawl.
ErrorMessage (string) --
If an error occurred, the error information about the last crawl.
LogGroup (string) --
The log group for the last crawl.
LogStream (string) --
The log stream for the last crawl.
MessagePrefix (string) --
The prefix for a message about this crawl.
StartTime (datetime) --
The time at which the crawl started.
Version (integer) --
The version of the crawler.
Configuration (string) --
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
CrawlerSecurityConfiguration (string) --
The name of the SecurityConfiguration structure to be used by this crawler.
CrawlersNotFound (list) --
A list of names of crawlers that were not found.
(string) --
{'ConnectionInput': {'ConnectionType': {'NETWORK'}}}
Creates a connection definition in the Data Catalog.
See also: AWS API Documentation
Request Syntax
client.create_connection(
CatalogId='string',
ConnectionInput={
'Name': 'string',
'Description': 'string',
'ConnectionType': 'JDBC'|'SFTP'|'MONGODB'|'KAFKA'|'NETWORK',
'MatchCriteria': [
'string',
],
'ConnectionProperties': {
'string': 'string'
},
'PhysicalConnectionRequirements': {
'SubnetId': 'string',
'SecurityGroupIdList': [
'string',
],
'AvailabilityZone': 'string'
}
}
)
string
The ID of the Data Catalog in which to create the connection. If none is provided, the AWS account ID is used by default.
dict
[REQUIRED]
A ConnectionInput object defining the connection to create.
Name (string) -- [REQUIRED]
The name of the connection.
Description (string) --
The description of the connection.
ConnectionType (string) -- [REQUIRED]
The type of the connection. Currently, these types are supported:
JDBC - Designates a connection to a database through Java Database Connectivity (JDBC).
KAFKA - Designates a connection to an Apache Kafka streaming platform.
MONGODB - Designates a connection to a MongoDB document database.
NETWORK - Designates a network connection to a data source within an Amazon Virtual Private Cloud environment (Amazon VPC).
SFTP is not supported.
MatchCriteria (list) --
A list of criteria that can be used in selecting this connection.
(string) --
ConnectionProperties (dict) -- [REQUIRED]
These key-value pairs define parameters for the connection.
(string) --
(string) --
PhysicalConnectionRequirements (dict) --
A map of physical connection requirements, such as virtual private cloud (VPC) and SecurityGroup , that are needed to successfully make this connection.
SubnetId (string) --
The subnet ID used by the connection.
SecurityGroupIdList (list) --
The security group ID list used by the connection.
(string) --
AvailabilityZone (string) --
The connection's Availability Zone. This field is redundant because the specified subnet implies the Availability Zone to be used. Currently the field must be populated, but it will be deprecated in the future.
dict
Response Syntax
{}
Response Structure
(dict) --
{'Targets': {'S3Targets': {'ConnectionName': 'string'}}}
Creates a new crawler with specified targets, role, configuration, and optional schedule. At least one crawl target must be specified, in the s3Targets field, the jdbcTargets field, or the DynamoDBTargets field.
See also: AWS API Documentation
Request Syntax
client.create_crawler(
Name='string',
Role='string',
DatabaseName='string',
Description='string',
Targets={
'S3Targets': [
{
'Path': 'string',
'Exclusions': [
'string',
],
'ConnectionName': 'string'
},
],
'JdbcTargets': [
{
'ConnectionName': 'string',
'Path': 'string',
'Exclusions': [
'string',
]
},
],
'DynamoDBTargets': [
{
'Path': 'string',
'scanAll': True|False,
'scanRate': 123.0
},
],
'CatalogTargets': [
{
'DatabaseName': 'string',
'Tables': [
'string',
]
},
]
},
Schedule='string',
Classifiers=[
'string',
],
TablePrefix='string',
SchemaChangePolicy={
'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
},
Configuration='string',
CrawlerSecurityConfiguration='string',
Tags={
'string': 'string'
}
)
string
[REQUIRED]
Name of the new crawler.
string
[REQUIRED]
The IAM role or Amazon Resource Name (ARN) of an IAM role used by the new crawler to access customer resources.
string
The AWS Glue database where results are written, such as: arn:aws:daylight:us-east-1::database/sometable/* .
string
A description of the new crawler.
dict
[REQUIRED]
A list of collection of targets to crawl.
S3Targets (list) --
Specifies Amazon Simple Storage Service (Amazon S3) targets.
(dict) --
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Path (string) --
The path to the Amazon S3 target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
ConnectionName (string) --
The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).
JdbcTargets (list) --
Specifies JDBC targets.
(dict) --
Specifies a JDBC data store to crawl.
ConnectionName (string) --
The name of the connection to use to connect to the JDBC target.
Path (string) --
The path of the JDBC target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
DynamoDBTargets (list) --
Specifies Amazon DynamoDB targets.
(dict) --
Specifies an Amazon DynamoDB table to crawl.
Path (string) --
The name of the DynamoDB table to crawl.
scanAll (boolean) --
Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.
A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true .
scanRate (float) --
The percentage of the configured read capacity units to use by the AWS Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.
The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).
CatalogTargets (list) --
Specifies AWS Glue Data Catalog targets.
(dict) --
Specifies an AWS Glue Data Catalog target.
DatabaseName (string) -- [REQUIRED]
The name of the database to be synchronized.
Tables (list) -- [REQUIRED]
A list of the tables to be synchronized.
(string) --
string
A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *) .
list
A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.
(string) --
string
The table prefix used for catalog tables that are created.
dict
The policy for the crawler's update and deletion behavior.
UpdateBehavior (string) --
The update behavior when the crawler finds a changed schema.
DeleteBehavior (string) --
The deletion behavior when the crawler finds a deleted object.
string
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
string
The name of the SecurityConfiguration structure to be used by this crawler.
dict
The tags to use with this crawler request. You may use tags to limit access to the crawler. For more information about tags in AWS Glue, see AWS Tags in AWS Glue in the developer guide.
(string) --
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
{'Connection': {'ConnectionType': {'NETWORK'}}}
Retrieves a connection definition from the Data Catalog.
See also: AWS API Documentation
Request Syntax
client.get_connection(
CatalogId='string',
Name='string',
HidePassword=True|False
)
string
The ID of the Data Catalog in which the connection resides. If none is provided, the AWS account ID is used by default.
string
[REQUIRED]
The name of the connection definition to retrieve.
boolean
Allows you to retrieve the connection metadata without returning the password. For instance, the AWS Glue console uses this flag to retrieve the connection, and does not display the password. Set this parameter when the caller might not have permission to use the AWS KMS key to decrypt the password, but it does have permission to access the rest of the connection properties.
dict
Response Syntax
{
'Connection': {
'Name': 'string',
'Description': 'string',
'ConnectionType': 'JDBC'|'SFTP'|'MONGODB'|'KAFKA'|'NETWORK',
'MatchCriteria': [
'string',
],
'ConnectionProperties': {
'string': 'string'
},
'PhysicalConnectionRequirements': {
'SubnetId': 'string',
'SecurityGroupIdList': [
'string',
],
'AvailabilityZone': 'string'
},
'CreationTime': datetime(2015, 1, 1),
'LastUpdatedTime': datetime(2015, 1, 1),
'LastUpdatedBy': 'string'
}
}
Response Structure
(dict) --
Connection (dict) --
The requested connection definition.
Name (string) --
The name of the connection definition.
Description (string) --
The description of the connection.
ConnectionType (string) --
The type of the connection. Currently, SFTP is not supported.
MatchCriteria (list) --
A list of criteria that can be used in selecting this connection.
(string) --
ConnectionProperties (dict) --
These key-value pairs define parameters for the connection:
HOST - The host URI: either the fully qualified domain name (FQDN) or the IPv4 address of the database host.
PORT - The port number, between 1024 and 65535, of the port on which the database host is listening for database connections.
USER_NAME - The name under which to log in to the database. The value string for USER_NAME is " USERNAME ".
PASSWORD - A password, if one is used, for the user name.
ENCRYPTED_PASSWORD - When you enable connection password protection by setting ConnectionPasswordEncryption in the Data Catalog encryption settings, this field stores the encrypted password.
JDBC_DRIVER_JAR_URI - The Amazon Simple Storage Service (Amazon S3) path of the JAR file that contains the JDBC driver to use.
JDBC_DRIVER_CLASS_NAME - The class name of the JDBC driver to use.
JDBC_ENGINE - The name of the JDBC engine to use.
JDBC_ENGINE_VERSION - The version of the JDBC engine to use.
CONFIG_FILES - (Reserved for future use.)
INSTANCE_ID - The instance ID to use.
JDBC_CONNECTION_URL - The URL for connecting to a JDBC data source.
JDBC_ENFORCE_SSL - A Boolean string (true, false) specifying whether Secure Sockets Layer (SSL) with hostname matching is enforced for the JDBC connection on the client. The default is false.
CUSTOM_JDBC_CERT - An Amazon S3 location specifying the customer's root certificate. AWS Glue uses this root certificate to validate the customer’s certificate when connecting to the customer database. AWS Glue only handles X.509 certificates. The certificate provided must be DER-encoded and supplied in Base64 encoding PEM format.
SKIP_CUSTOM_JDBC_CERT_VALIDATION - By default, this is false . AWS Glue validates the Signature algorithm and Subject Public Key Algorithm for the customer certificate. The only permitted algorithms for the Signature algorithm are SHA256withRSA, SHA384withRSA or SHA512withRSA. For the Subject Public Key Algorithm, the key length must be at least 2048. You can set the value of this property to true to skip AWS Glue’s validation of the customer certificate.
CUSTOM_JDBC_CERT_STRING - A custom JDBC certificate string which is used for domain match or distinguished name match to prevent a man-in-the-middle attack. In Oracle database, this is used as the SSL_SERVER_CERT_DN ; in Microsoft SQL Server, this is used as the hostNameInCertificate .
CONNECTION_URL - The URL for connecting to a general (non-JDBC) data source.
KAFKA_BOOTSTRAP_SERVERS - A comma-separated list of host and port pairs that are the addresses of the Apache Kafka brokers in a Kafka cluster to which a Kafka client will connect to and bootstrap itself.
KAFKA_SSL_ENABLED - Whether to enable or disable SSL on an Apache Kafka connection. Default value is "true".
KAFKA_CUSTOM_CERT - The Amazon S3 URL for the private CA cert file (.pem format). The default is an empty string.
KAFKA_SKIP_CUSTOM_CERT_VALIDATION - Whether to skip the validation of the CA cert file or not. AWS Glue validates for three algorithms: SHA256withRSA, SHA384withRSA and SHA512withRSA. Default value is "false".
(string) --
(string) --
PhysicalConnectionRequirements (dict) --
A map of physical connection requirements, such as virtual private cloud (VPC) and SecurityGroup , that are needed to make this connection successfully.
SubnetId (string) --
The subnet ID used by the connection.
SecurityGroupIdList (list) --
The security group ID list used by the connection.
(string) --
AvailabilityZone (string) --
The connection's Availability Zone. This field is redundant because the specified subnet implies the Availability Zone to be used. Currently the field must be populated, but it will be deprecated in the future.
CreationTime (datetime) --
The time that this connection definition was created.
LastUpdatedTime (datetime) --
The last time that this connection definition was updated.
LastUpdatedBy (string) --
The user, group, or role that last updated this connection definition.
{'Filter': {'ConnectionType': {'NETWORK'}}}
Response {'ConnectionList': {'ConnectionType': {'NETWORK'}}}
Retrieves a list of connection definitions from the Data Catalog.
See also: AWS API Documentation
Request Syntax
client.get_connections(
CatalogId='string',
Filter={
'MatchCriteria': [
'string',
],
'ConnectionType': 'JDBC'|'SFTP'|'MONGODB'|'KAFKA'|'NETWORK'
},
HidePassword=True|False,
NextToken='string',
MaxResults=123
)
string
The ID of the Data Catalog in which the connections reside. If none is provided, the AWS account ID is used by default.
dict
A filter that controls which connections are returned.
MatchCriteria (list) --
A criteria string that must match the criteria recorded in the connection definition for that connection definition to be returned.
(string) --
ConnectionType (string) --
The type of connections to return. Currently, SFTP is not supported.
boolean
Allows you to retrieve the connection metadata without returning the password. For instance, the AWS Glue console uses this flag to retrieve the connection, and does not display the password. Set this parameter when the caller might not have permission to use the AWS KMS key to decrypt the password, but it does have permission to access the rest of the connection properties.
string
A continuation token, if this is a continuation call.
integer
The maximum number of connections to return in one response.
dict
Response Syntax
{
'ConnectionList': [
{
'Name': 'string',
'Description': 'string',
'ConnectionType': 'JDBC'|'SFTP'|'MONGODB'|'KAFKA'|'NETWORK',
'MatchCriteria': [
'string',
],
'ConnectionProperties': {
'string': 'string'
},
'PhysicalConnectionRequirements': {
'SubnetId': 'string',
'SecurityGroupIdList': [
'string',
],
'AvailabilityZone': 'string'
},
'CreationTime': datetime(2015, 1, 1),
'LastUpdatedTime': datetime(2015, 1, 1),
'LastUpdatedBy': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
ConnectionList (list) --
A list of requested connection definitions.
(dict) --
Defines a connection to a data source.
Name (string) --
The name of the connection definition.
Description (string) --
The description of the connection.
ConnectionType (string) --
The type of the connection. Currently, SFTP is not supported.
MatchCriteria (list) --
A list of criteria that can be used in selecting this connection.
(string) --
ConnectionProperties (dict) --
These key-value pairs define parameters for the connection:
HOST - The host URI: either the fully qualified domain name (FQDN) or the IPv4 address of the database host.
PORT - The port number, between 1024 and 65535, of the port on which the database host is listening for database connections.
USER_NAME - The name under which to log in to the database. The value string for USER_NAME is " USERNAME ".
PASSWORD - A password, if one is used, for the user name.
ENCRYPTED_PASSWORD - When you enable connection password protection by setting ConnectionPasswordEncryption in the Data Catalog encryption settings, this field stores the encrypted password.
JDBC_DRIVER_JAR_URI - The Amazon Simple Storage Service (Amazon S3) path of the JAR file that contains the JDBC driver to use.
JDBC_DRIVER_CLASS_NAME - The class name of the JDBC driver to use.
JDBC_ENGINE - The name of the JDBC engine to use.
JDBC_ENGINE_VERSION - The version of the JDBC engine to use.
CONFIG_FILES - (Reserved for future use.)
INSTANCE_ID - The instance ID to use.
JDBC_CONNECTION_URL - The URL for connecting to a JDBC data source.
JDBC_ENFORCE_SSL - A Boolean string (true, false) specifying whether Secure Sockets Layer (SSL) with hostname matching is enforced for the JDBC connection on the client. The default is false.
CUSTOM_JDBC_CERT - An Amazon S3 location specifying the customer's root certificate. AWS Glue uses this root certificate to validate the customer’s certificate when connecting to the customer database. AWS Glue only handles X.509 certificates. The certificate provided must be DER-encoded and supplied in Base64 encoding PEM format.
SKIP_CUSTOM_JDBC_CERT_VALIDATION - By default, this is false . AWS Glue validates the Signature algorithm and Subject Public Key Algorithm for the customer certificate. The only permitted algorithms for the Signature algorithm are SHA256withRSA, SHA384withRSA or SHA512withRSA. For the Subject Public Key Algorithm, the key length must be at least 2048. You can set the value of this property to true to skip AWS Glue’s validation of the customer certificate.
CUSTOM_JDBC_CERT_STRING - A custom JDBC certificate string which is used for domain match or distinguished name match to prevent a man-in-the-middle attack. In Oracle database, this is used as the SSL_SERVER_CERT_DN ; in Microsoft SQL Server, this is used as the hostNameInCertificate .
CONNECTION_URL - The URL for connecting to a general (non-JDBC) data source.
KAFKA_BOOTSTRAP_SERVERS - A comma-separated list of host and port pairs that are the addresses of the Apache Kafka brokers in a Kafka cluster to which a Kafka client will connect to and bootstrap itself.
KAFKA_SSL_ENABLED - Whether to enable or disable SSL on an Apache Kafka connection. Default value is "true".
KAFKA_CUSTOM_CERT - The Amazon S3 URL for the private CA cert file (.pem format). The default is an empty string.
KAFKA_SKIP_CUSTOM_CERT_VALIDATION - Whether to skip the validation of the CA cert file or not. AWS Glue validates for three algorithms: SHA256withRSA, SHA384withRSA and SHA512withRSA. Default value is "false".
(string) --
(string) --
PhysicalConnectionRequirements (dict) --
A map of physical connection requirements, such as virtual private cloud (VPC) and SecurityGroup , that are needed to make this connection successfully.
SubnetId (string) --
The subnet ID used by the connection.
SecurityGroupIdList (list) --
The security group ID list used by the connection.
(string) --
AvailabilityZone (string) --
The connection's Availability Zone. This field is redundant because the specified subnet implies the Availability Zone to be used. Currently the field must be populated, but it will be deprecated in the future.
CreationTime (datetime) --
The time that this connection definition was created.
LastUpdatedTime (datetime) --
The last time that this connection definition was updated.
LastUpdatedBy (string) --
The user, group, or role that last updated this connection definition.
NextToken (string) --
A continuation token, if the list of connections returned does not include the last of the filtered connections.
{'Crawler': {'Targets': {'S3Targets': {'ConnectionName': 'string'}}}}
Retrieves metadata for a specified crawler.
See also: AWS API Documentation
Request Syntax
client.get_crawler(
Name='string'
)
string
[REQUIRED]
The name of the crawler to retrieve metadata for.
dict
Response Syntax
{
'Crawler': {
'Name': 'string',
'Role': 'string',
'Targets': {
'S3Targets': [
{
'Path': 'string',
'Exclusions': [
'string',
],
'ConnectionName': 'string'
},
],
'JdbcTargets': [
{
'ConnectionName': 'string',
'Path': 'string',
'Exclusions': [
'string',
]
},
],
'DynamoDBTargets': [
{
'Path': 'string',
'scanAll': True|False,
'scanRate': 123.0
},
],
'CatalogTargets': [
{
'DatabaseName': 'string',
'Tables': [
'string',
]
},
]
},
'DatabaseName': 'string',
'Description': 'string',
'Classifiers': [
'string',
],
'SchemaChangePolicy': {
'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
},
'State': 'READY'|'RUNNING'|'STOPPING',
'TablePrefix': 'string',
'Schedule': {
'ScheduleExpression': 'string',
'State': 'SCHEDULED'|'NOT_SCHEDULED'|'TRANSITIONING'
},
'CrawlElapsedTime': 123,
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'LastCrawl': {
'Status': 'SUCCEEDED'|'CANCELLED'|'FAILED',
'ErrorMessage': 'string',
'LogGroup': 'string',
'LogStream': 'string',
'MessagePrefix': 'string',
'StartTime': datetime(2015, 1, 1)
},
'Version': 123,
'Configuration': 'string',
'CrawlerSecurityConfiguration': 'string'
}
}
Response Structure
(dict) --
Crawler (dict) --
The metadata for the specified crawler.
Name (string) --
The name of the crawler.
Role (string) --
The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.
Targets (dict) --
A collection of targets to crawl.
S3Targets (list) --
Specifies Amazon Simple Storage Service (Amazon S3) targets.
(dict) --
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Path (string) --
The path to the Amazon S3 target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
ConnectionName (string) --
The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).
JdbcTargets (list) --
Specifies JDBC targets.
(dict) --
Specifies a JDBC data store to crawl.
ConnectionName (string) --
The name of the connection to use to connect to the JDBC target.
Path (string) --
The path of the JDBC target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
DynamoDBTargets (list) --
Specifies Amazon DynamoDB targets.
(dict) --
Specifies an Amazon DynamoDB table to crawl.
Path (string) --
The name of the DynamoDB table to crawl.
scanAll (boolean) --
Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.
A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true .
scanRate (float) --
The percentage of the configured read capacity units to use by the AWS Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.
The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).
CatalogTargets (list) --
Specifies AWS Glue Data Catalog targets.
(dict) --
Specifies an AWS Glue Data Catalog target.
DatabaseName (string) --
The name of the database to be synchronized.
Tables (list) --
A list of the tables to be synchronized.
(string) --
DatabaseName (string) --
The name of the database in which the crawler's output is stored.
Description (string) --
A description of the crawler.
Classifiers (list) --
A list of UTF-8 strings that specify the custom classifiers that are associated with the crawler.
(string) --
SchemaChangePolicy (dict) --
The policy that specifies update and delete behaviors for the crawler.
UpdateBehavior (string) --
The update behavior when the crawler finds a changed schema.
DeleteBehavior (string) --
The deletion behavior when the crawler finds a deleted object.
State (string) --
Indicates whether the crawler is running, or whether a run is pending.
TablePrefix (string) --
The prefix added to the names of tables that are created.
Schedule (dict) --
For scheduled crawlers, the schedule when the crawler runs.
ScheduleExpression (string) --
A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *) .
State (string) --
The state of the schedule.
CrawlElapsedTime (integer) --
If the crawler is running, contains the total time elapsed since the last crawl began.
CreationTime (datetime) --
The time that the crawler was created.
LastUpdated (datetime) --
The time that the crawler was last updated.
LastCrawl (dict) --
The status of the last crawl, and potentially error information if an error occurred.
Status (string) --
Status of the last crawl.
ErrorMessage (string) --
If an error occurred, the error information about the last crawl.
LogGroup (string) --
The log group for the last crawl.
LogStream (string) --
The log stream for the last crawl.
MessagePrefix (string) --
The prefix for a message about this crawl.
StartTime (datetime) --
The time at which the crawl started.
Version (integer) --
The version of the crawler.
Configuration (string) --
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
CrawlerSecurityConfiguration (string) --
The name of the SecurityConfiguration structure to be used by this crawler.
{'Crawlers': {'Targets': {'S3Targets': {'ConnectionName': 'string'}}}}
Retrieves metadata for all crawlers defined in the customer account.
See also: AWS API Documentation
Request Syntax
client.get_crawlers(
MaxResults=123,
NextToken='string'
)
integer
The number of crawlers to return on each call.
string
A continuation token, if this is a continuation request.
dict
Response Syntax
{
'Crawlers': [
{
'Name': 'string',
'Role': 'string',
'Targets': {
'S3Targets': [
{
'Path': 'string',
'Exclusions': [
'string',
],
'ConnectionName': 'string'
},
],
'JdbcTargets': [
{
'ConnectionName': 'string',
'Path': 'string',
'Exclusions': [
'string',
]
},
],
'DynamoDBTargets': [
{
'Path': 'string',
'scanAll': True|False,
'scanRate': 123.0
},
],
'CatalogTargets': [
{
'DatabaseName': 'string',
'Tables': [
'string',
]
},
]
},
'DatabaseName': 'string',
'Description': 'string',
'Classifiers': [
'string',
],
'SchemaChangePolicy': {
'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
},
'State': 'READY'|'RUNNING'|'STOPPING',
'TablePrefix': 'string',
'Schedule': {
'ScheduleExpression': 'string',
'State': 'SCHEDULED'|'NOT_SCHEDULED'|'TRANSITIONING'
},
'CrawlElapsedTime': 123,
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'LastCrawl': {
'Status': 'SUCCEEDED'|'CANCELLED'|'FAILED',
'ErrorMessage': 'string',
'LogGroup': 'string',
'LogStream': 'string',
'MessagePrefix': 'string',
'StartTime': datetime(2015, 1, 1)
},
'Version': 123,
'Configuration': 'string',
'CrawlerSecurityConfiguration': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Crawlers (list) --
A list of crawler metadata.
(dict) --
Specifies a crawler program that examines a data source and uses classifiers to try to determine its schema. If successful, the crawler records metadata concerning the data source in the AWS Glue Data Catalog.
Name (string) --
The name of the crawler.
Role (string) --
The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.
Targets (dict) --
A collection of targets to crawl.
S3Targets (list) --
Specifies Amazon Simple Storage Service (Amazon S3) targets.
(dict) --
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Path (string) --
The path to the Amazon S3 target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
ConnectionName (string) --
The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).
JdbcTargets (list) --
Specifies JDBC targets.
(dict) --
Specifies a JDBC data store to crawl.
ConnectionName (string) --
The name of the connection to use to connect to the JDBC target.
Path (string) --
The path of the JDBC target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
DynamoDBTargets (list) --
Specifies Amazon DynamoDB targets.
(dict) --
Specifies an Amazon DynamoDB table to crawl.
Path (string) --
The name of the DynamoDB table to crawl.
scanAll (boolean) --
Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.
A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true .
scanRate (float) --
The percentage of the configured read capacity units to use by the AWS Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.
The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).
CatalogTargets (list) --
Specifies AWS Glue Data Catalog targets.
(dict) --
Specifies an AWS Glue Data Catalog target.
DatabaseName (string) --
The name of the database to be synchronized.
Tables (list) --
A list of the tables to be synchronized.
(string) --
DatabaseName (string) --
The name of the database in which the crawler's output is stored.
Description (string) --
A description of the crawler.
Classifiers (list) --
A list of UTF-8 strings that specify the custom classifiers that are associated with the crawler.
(string) --
SchemaChangePolicy (dict) --
The policy that specifies update and delete behaviors for the crawler.
UpdateBehavior (string) --
The update behavior when the crawler finds a changed schema.
DeleteBehavior (string) --
The deletion behavior when the crawler finds a deleted object.
State (string) --
Indicates whether the crawler is running, or whether a run is pending.
TablePrefix (string) --
The prefix added to the names of tables that are created.
Schedule (dict) --
For scheduled crawlers, the schedule when the crawler runs.
ScheduleExpression (string) --
A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *) .
State (string) --
The state of the schedule.
CrawlElapsedTime (integer) --
If the crawler is running, contains the total time elapsed since the last crawl began.
CreationTime (datetime) --
The time that the crawler was created.
LastUpdated (datetime) --
The time that the crawler was last updated.
LastCrawl (dict) --
The status of the last crawl, and potentially error information if an error occurred.
Status (string) --
Status of the last crawl.
ErrorMessage (string) --
If an error occurred, the error information about the last crawl.
LogGroup (string) --
The log group for the last crawl.
LogStream (string) --
The log stream for the last crawl.
MessagePrefix (string) --
The prefix for a message about this crawl.
StartTime (datetime) --
The time at which the crawl started.
Version (integer) --
The version of the crawler.
Configuration (string) --
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
CrawlerSecurityConfiguration (string) --
The name of the SecurityConfiguration structure to be used by this crawler.
NextToken (string) --
A continuation token, if the returned list has not reached the end of those defined in this customer account.
{'ConnectionInput': {'ConnectionType': {'NETWORK'}}}
Updates a connection definition in the Data Catalog.
See also: AWS API Documentation
Request Syntax
client.update_connection(
CatalogId='string',
Name='string',
ConnectionInput={
'Name': 'string',
'Description': 'string',
'ConnectionType': 'JDBC'|'SFTP'|'MONGODB'|'KAFKA'|'NETWORK',
'MatchCriteria': [
'string',
],
'ConnectionProperties': {
'string': 'string'
},
'PhysicalConnectionRequirements': {
'SubnetId': 'string',
'SecurityGroupIdList': [
'string',
],
'AvailabilityZone': 'string'
}
}
)
string
The ID of the Data Catalog in which the connection resides. If none is provided, the AWS account ID is used by default.
string
[REQUIRED]
The name of the connection definition to update.
dict
[REQUIRED]
A ConnectionInput object that redefines the connection in question.
Name (string) -- [REQUIRED]
The name of the connection.
Description (string) --
The description of the connection.
ConnectionType (string) -- [REQUIRED]
The type of the connection. Currently, these types are supported:
JDBC - Designates a connection to a database through Java Database Connectivity (JDBC).
KAFKA - Designates a connection to an Apache Kafka streaming platform.
MONGODB - Designates a connection to a MongoDB document database.
NETWORK - Designates a network connection to a data source within an Amazon Virtual Private Cloud environment (Amazon VPC).
SFTP is not supported.
MatchCriteria (list) --
A list of criteria that can be used in selecting this connection.
(string) --
ConnectionProperties (dict) -- [REQUIRED]
These key-value pairs define parameters for the connection.
(string) --
(string) --
PhysicalConnectionRequirements (dict) --
A map of physical connection requirements, such as virtual private cloud (VPC) and SecurityGroup , that are needed to successfully make this connection.
SubnetId (string) --
The subnet ID used by the connection.
SecurityGroupIdList (list) --
The security group ID list used by the connection.
(string) --
AvailabilityZone (string) --
The connection's Availability Zone. This field is redundant because the specified subnet implies the Availability Zone to be used. Currently the field must be populated, but it will be deprecated in the future.
dict
Response Syntax
{}
Response Structure
(dict) --
{'Targets': {'S3Targets': {'ConnectionName': 'string'}}}
Updates a crawler. If a crawler is running, you must stop it using StopCrawler before updating it.
See also: AWS API Documentation
Request Syntax
client.update_crawler(
Name='string',
Role='string',
DatabaseName='string',
Description='string',
Targets={
'S3Targets': [
{
'Path': 'string',
'Exclusions': [
'string',
],
'ConnectionName': 'string'
},
],
'JdbcTargets': [
{
'ConnectionName': 'string',
'Path': 'string',
'Exclusions': [
'string',
]
},
],
'DynamoDBTargets': [
{
'Path': 'string',
'scanAll': True|False,
'scanRate': 123.0
},
],
'CatalogTargets': [
{
'DatabaseName': 'string',
'Tables': [
'string',
]
},
]
},
Schedule='string',
Classifiers=[
'string',
],
TablePrefix='string',
SchemaChangePolicy={
'UpdateBehavior': 'LOG'|'UPDATE_IN_DATABASE',
'DeleteBehavior': 'LOG'|'DELETE_FROM_DATABASE'|'DEPRECATE_IN_DATABASE'
},
Configuration='string',
CrawlerSecurityConfiguration='string'
)
string
[REQUIRED]
Name of the new crawler.
string
The IAM role or Amazon Resource Name (ARN) of an IAM role that is used by the new crawler to access customer resources.
string
The AWS Glue database where results are stored, such as: arn:aws:daylight:us-east-1::database/sometable/* .
string
A description of the new crawler.
dict
A list of targets to crawl.
S3Targets (list) --
Specifies Amazon Simple Storage Service (Amazon S3) targets.
(dict) --
Specifies a data store in Amazon Simple Storage Service (Amazon S3).
Path (string) --
The path to the Amazon S3 target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
ConnectionName (string) --
The name of a connection which allows a job or crawler to access data in Amazon S3 within an Amazon Virtual Private Cloud environment (Amazon VPC).
JdbcTargets (list) --
Specifies JDBC targets.
(dict) --
Specifies a JDBC data store to crawl.
ConnectionName (string) --
The name of the connection to use to connect to the JDBC target.
Path (string) --
The path of the JDBC target.
Exclusions (list) --
A list of glob patterns used to exclude from the crawl. For more information, see Catalog Tables with a Crawler.
(string) --
DynamoDBTargets (list) --
Specifies Amazon DynamoDB targets.
(dict) --
Specifies an Amazon DynamoDB table to crawl.
Path (string) --
The name of the DynamoDB table to crawl.
scanAll (boolean) --
Indicates whether to scan all the records, or to sample rows from the table. Scanning all the records can take a long time when the table is not a high throughput table.
A value of true means to scan all records, while a value of false means to sample the records. If no value is specified, the value defaults to true .
scanRate (float) --
The percentage of the configured read capacity units to use by the AWS Glue crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second.
The valid values are null or a value between 0.1 to 1.5. A null value is used when user does not provide a value, and defaults to 0.5 of the configured Read Capacity Unit (for provisioned tables), or 0.25 of the max configured Read Capacity Unit (for tables using on-demand mode).
CatalogTargets (list) --
Specifies AWS Glue Data Catalog targets.
(dict) --
Specifies an AWS Glue Data Catalog target.
DatabaseName (string) -- [REQUIRED]
The name of the database to be synchronized.
Tables (list) -- [REQUIRED]
A list of the tables to be synchronized.
(string) --
string
A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *) .
list
A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.
(string) --
string
The table prefix used for catalog tables that are created.
dict
The policy for the crawler's update and deletion behavior.
UpdateBehavior (string) --
The update behavior when the crawler finds a changed schema.
DeleteBehavior (string) --
The deletion behavior when the crawler finds a deleted object.
string
Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Configuring a Crawler.
string
The name of the SecurityConfiguration structure to be used by this crawler.
dict
Response Syntax
{}
Response Structure
(dict) --