Amazon Athena

2024/09/23 - Amazon Athena - 5 updated api methods

Changes  List/Get/Update/Delete/CreateDataCatalog now integrate with AWS Glue connections. Users can create a Glue connection through Athena or use a Glue connection to define their Athena federated parameters.

CreateDataCatalog (updated) Link ¶
Changes (request, response)
Request 
{'Type': {'FEDERATED'}}
Response 
{'DataCatalog': {'ConnectionType': 'DYNAMODB | MYSQL | POSTGRESQL | REDSHIFT | '
                                   'ORACLE | SYNAPSE | SQLSERVER | DB2 | '
                                   'OPENSEARCH | BIGQUERY | GOOGLECLOUDSTORAGE '
                                   '| HBASE | DOCUMENTDB | MSK | NEPTUNE | '
                                   'CMDB | TPCDS | REDIS | CLOUDWATCH | '
                                   'TIMESTREAM | SAPHANA | SNOWFLAKE | '
                                   'TERADATA | VERTICA | CLOUDERAIMPALA | '
                                   'CLOUDERAHIVE | HORTONWORKSHIVE | '
                                   'DATALAKEGEN2 | DB2AS400 | '
                                   'CLOUDWATCHMETRICS',
                 'Description': 'string',
                 'Error': 'string',
                 'Name': 'string',
                 'Parameters': {'string': 'string'},
                 'Status': 'CREATE_IN_PROGRESS | CREATE_COMPLETE | '
                           'CREATE_FAILED | CREATE_FAILED_CLEANUP_IN_PROGRESS '
                           '| CREATE_FAILED_CLEANUP_COMPLETE | '
                           'CREATE_FAILED_CLEANUP_FAILED | DELETE_IN_PROGRESS '
                           '| DELETE_COMPLETE | DELETE_FAILED',
                 'Type': 'LAMBDA | GLUE | HIVE | FEDERATED'}}

Creates (registers) a data catalog with the specified name and properties. Catalogs created are visible to all users of the same Amazon Web Services account.

See also: AWS API Documentation

Request Syntax

client.create_data_catalog(
    Name='string',
    Type='LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
    Description='string',
    Parameters={
        'string': 'string'
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
type Name

string

param Name

[REQUIRED]

The name of the data catalog to create. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

type Type

string

param Type

[REQUIRED]

The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass.

type Description

string

param Description

A description of the data catalog to be created.

type Parameters

dict

param Parameters

Specifies the Lambda function or functions to use for creating the data catalog. This is a mapping whose values depend on the catalog type.

  • For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number

  • For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both.

    • If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn

    • If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn

  • The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue Data Catalog belongs. catalog-id=catalog_id

    • The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify.

  • The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection.

    • connection-arn:<glue_connection_arn_to_reuse>

    • lambda-role-arn (optional): The execution role to use for the Lambda function. If not provided, one is created.

    • connection-type:MYSQL|REDSHIFT|...., connection-properties:"<json_string>" For <json_string> , use escaped JSON text, as in the following example. "{\"spill_bucket\":\"my_spill\",\"spill_prefix\":\"athena-spill\",\"host\":\"abc12345.snowflakecomputing.com\",\"port\":\"1234\",\"warehouse\":\"DEV_WH\",\"database\":\"TEST\",\"schema\":\"PUBLIC\",\"SecretArn\":\"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j\"}"

  • (string) --

    • (string) --

type Tags

list

param Tags

A list of comma separated tags to add to the data catalog that is created.

  • (dict) --

    A label that you assign to a resource. Athena resources include workgroups, data catalogs, and capacity reservations. Each tag consists of a key and an optional value, both of which you define. For example, you can use tags to categorize Athena resources by purpose, owner, or environment. Use a consistent set of tag keys to make it easier to search and filter the resources in your account. For best practices, see Tagging Best Practices. Tag keys can be from 1 to 128 UTF-8 Unicode characters, and tag values can be from 0 to 256 UTF-8 Unicode characters. Tags can use letters and numbers representable in UTF-8, and the following characters: + - = . _ : / @. Tag keys and values are case-sensitive. Tag keys must be unique per resource. If you specify more than one tag, separate them by commas.

    • Key (string) --

      A tag key. The tag key length is from 1 to 128 Unicode characters in UTF-8. You can use letters and numbers representable in UTF-8, and the following characters: + - = . _ : / @. Tag keys are case-sensitive and must be unique per resource.

    • Value (string) --

      A tag value. The tag value length is from 0 to 256 Unicode characters in UTF-8. You can use letters and numbers representable in UTF-8, and the following characters: + - = . _ : / @. Tag values are case-sensitive.

rtype

dict

returns

Response Syntax

{
    'DataCatalog': {
        'Name': 'string',
        'Description': 'string',
        'Type': 'LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
        'Parameters': {
            'string': 'string'
        },
        'Status': 'CREATE_IN_PROGRESS'|'CREATE_COMPLETE'|'CREATE_FAILED'|'CREATE_FAILED_CLEANUP_IN_PROGRESS'|'CREATE_FAILED_CLEANUP_COMPLETE'|'CREATE_FAILED_CLEANUP_FAILED'|'DELETE_IN_PROGRESS'|'DELETE_COMPLETE'|'DELETE_FAILED',
        'ConnectionType': 'DYNAMODB'|'MYSQL'|'POSTGRESQL'|'REDSHIFT'|'ORACLE'|'SYNAPSE'|'SQLSERVER'|'DB2'|'OPENSEARCH'|'BIGQUERY'|'GOOGLECLOUDSTORAGE'|'HBASE'|'DOCUMENTDB'|'MSK'|'NEPTUNE'|'CMDB'|'TPCDS'|'REDIS'|'CLOUDWATCH'|'TIMESTREAM'|'SAPHANA'|'SNOWFLAKE'|'TERADATA'|'VERTICA'|'CLOUDERAIMPALA'|'CLOUDERAHIVE'|'HORTONWORKSHIVE'|'DATALAKEGEN2'|'DB2AS400'|'CLOUDWATCHMETRICS',
        'Error': 'string'
    }
}

Response Structure

  • (dict) --

    • DataCatalog (dict) --

      Contains information about a data catalog in an Amazon Web Services account.

      Note

      In the Athena console, data catalogs are listed as "data sources" on the Data sources page under the Data source name column.

      • Name (string) --

        The name of the data catalog. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

      • Description (string) --

        An optional description of the data catalog.

      • Type (string) --

        The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass.

      • Parameters (dict) --

        Specifies the Lambda function or functions to use for the data catalog. This is a mapping whose values depend on the catalog type.

        • For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number

        • For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both.

          • If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn

          • If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn

        • The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue catalog belongs. catalog-id=catalog_id

          • The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify.

        • The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection.

          • connection-arn:<glue_connection_arn_to_reuse>

          • connection-type:MYSQL|REDSHIFT|...., connection-properties:"<json_string>" For <json_string> , use escaped JSON text, as in the following example. "{\"spill_bucket\":\"my_spill\",\"spill_prefix\":\"athena-spill\",\"host\":\"abc12345.snowflakecomputing.com\",\"port\":\"1234\",\"warehouse\":\"DEV_WH\",\"database\":\"TEST\",\"schema\":\"PUBLIC\",\"SecretArn\":\"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j\"}"

        • (string) --

          • (string) --

      • Status (string) --

        The status of the creation or deletion of the data catalog.

        • The LAMBDA , GLUE , and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED .

        • The FEDERATED data catalog type is created asynchronously.

        Data catalog creation status:

        • CREATE_IN_PROGRESS : Federated data catalog creation in progress.

        • CREATE_COMPLETE : Data catalog creation complete.

        • CREATE_FAILED : Data catalog could not be created.

        • CREATE_FAILED_CLEANUP_IN_PROGRESS : Federated data catalog creation failed and is being removed.

        • CREATE_FAILED_CLEANUP_COMPLETE : Federated data catalog creation failed and was removed.

        • CREATE_FAILED_CLEANUP_FAILED : Federated data catalog creation failed but could not be removed.

        Data catalog deletion status:

        • DELETE_IN_PROGRESS : Federated data catalog deletion in progress.

        • DELETE_COMPLETE : Federated data catalog deleted.

        • DELETE_FAILED : Federated data catalog could not be deleted.

      • ConnectionType (string) --

        The type of connection for a FEDERATED data catalog (for example, REDSHIFT , MYSQL , or SQLSERVER ). For information about individual connectors, see Available data source connectors.

      • Error (string) --

        Text of the error that occurred during data catalog creation or deletion.

DeleteDataCatalog (updated) Link ¶
Changes (response) 
{'DataCatalog': {'ConnectionType': 'DYNAMODB | MYSQL | POSTGRESQL | REDSHIFT | '
                                   'ORACLE | SYNAPSE | SQLSERVER | DB2 | '
                                   'OPENSEARCH | BIGQUERY | GOOGLECLOUDSTORAGE '
                                   '| HBASE | DOCUMENTDB | MSK | NEPTUNE | '
                                   'CMDB | TPCDS | REDIS | CLOUDWATCH | '
                                   'TIMESTREAM | SAPHANA | SNOWFLAKE | '
                                   'TERADATA | VERTICA | CLOUDERAIMPALA | '
                                   'CLOUDERAHIVE | HORTONWORKSHIVE | '
                                   'DATALAKEGEN2 | DB2AS400 | '
                                   'CLOUDWATCHMETRICS',
                 'Description': 'string',
                 'Error': 'string',
                 'Name': 'string',
                 'Parameters': {'string': 'string'},
                 'Status': 'CREATE_IN_PROGRESS | CREATE_COMPLETE | '
                           'CREATE_FAILED | CREATE_FAILED_CLEANUP_IN_PROGRESS '
                           '| CREATE_FAILED_CLEANUP_COMPLETE | '
                           'CREATE_FAILED_CLEANUP_FAILED | DELETE_IN_PROGRESS '
                           '| DELETE_COMPLETE | DELETE_FAILED',
                 'Type': 'LAMBDA | GLUE | HIVE | FEDERATED'}}

Deletes a data catalog.

See also: AWS API Documentation

Request Syntax

client.delete_data_catalog(
    Name='string'
)
type Name

string

param Name

[REQUIRED]

The name of the data catalog to delete.

rtype

dict

returns

Response Syntax

{
    'DataCatalog': {
        'Name': 'string',
        'Description': 'string',
        'Type': 'LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
        'Parameters': {
            'string': 'string'
        },
        'Status': 'CREATE_IN_PROGRESS'|'CREATE_COMPLETE'|'CREATE_FAILED'|'CREATE_FAILED_CLEANUP_IN_PROGRESS'|'CREATE_FAILED_CLEANUP_COMPLETE'|'CREATE_FAILED_CLEANUP_FAILED'|'DELETE_IN_PROGRESS'|'DELETE_COMPLETE'|'DELETE_FAILED',
        'ConnectionType': 'DYNAMODB'|'MYSQL'|'POSTGRESQL'|'REDSHIFT'|'ORACLE'|'SYNAPSE'|'SQLSERVER'|'DB2'|'OPENSEARCH'|'BIGQUERY'|'GOOGLECLOUDSTORAGE'|'HBASE'|'DOCUMENTDB'|'MSK'|'NEPTUNE'|'CMDB'|'TPCDS'|'REDIS'|'CLOUDWATCH'|'TIMESTREAM'|'SAPHANA'|'SNOWFLAKE'|'TERADATA'|'VERTICA'|'CLOUDERAIMPALA'|'CLOUDERAHIVE'|'HORTONWORKSHIVE'|'DATALAKEGEN2'|'DB2AS400'|'CLOUDWATCHMETRICS',
        'Error': 'string'
    }
}

Response Structure

  • (dict) --

    • DataCatalog (dict) --

      Contains information about a data catalog in an Amazon Web Services account.

      Note

      In the Athena console, data catalogs are listed as "data sources" on the Data sources page under the Data source name column.

      • Name (string) --

        The name of the data catalog. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

      • Description (string) --

        An optional description of the data catalog.

      • Type (string) --

        The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass.

      • Parameters (dict) --

        Specifies the Lambda function or functions to use for the data catalog. This is a mapping whose values depend on the catalog type.

        • For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number

        • For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both.

          • If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn

          • If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn

        • The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue catalog belongs. catalog-id=catalog_id

          • The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify.

        • The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection.

          • connection-arn:<glue_connection_arn_to_reuse>

          • connection-type:MYSQL|REDSHIFT|...., connection-properties:"<json_string>" For <json_string> , use escaped JSON text, as in the following example. "{\"spill_bucket\":\"my_spill\",\"spill_prefix\":\"athena-spill\",\"host\":\"abc12345.snowflakecomputing.com\",\"port\":\"1234\",\"warehouse\":\"DEV_WH\",\"database\":\"TEST\",\"schema\":\"PUBLIC\",\"SecretArn\":\"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j\"}"

        • (string) --

          • (string) --

      • Status (string) --

        The status of the creation or deletion of the data catalog.

        • The LAMBDA , GLUE , and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED .

        • The FEDERATED data catalog type is created asynchronously.

        Data catalog creation status:

        • CREATE_IN_PROGRESS : Federated data catalog creation in progress.

        • CREATE_COMPLETE : Data catalog creation complete.

        • CREATE_FAILED : Data catalog could not be created.

        • CREATE_FAILED_CLEANUP_IN_PROGRESS : Federated data catalog creation failed and is being removed.

        • CREATE_FAILED_CLEANUP_COMPLETE : Federated data catalog creation failed and was removed.

        • CREATE_FAILED_CLEANUP_FAILED : Federated data catalog creation failed but could not be removed.

        Data catalog deletion status:

        • DELETE_IN_PROGRESS : Federated data catalog deletion in progress.

        • DELETE_COMPLETE : Federated data catalog deleted.

        • DELETE_FAILED : Federated data catalog could not be deleted.

      • ConnectionType (string) --

        The type of connection for a FEDERATED data catalog (for example, REDSHIFT , MYSQL , or SQLSERVER ). For information about individual connectors, see Available data source connectors.

      • Error (string) --

        Text of the error that occurred during data catalog creation or deletion.

GetDataCatalog (updated) Link ¶
Changes (response) 
{'DataCatalog': {'ConnectionType': 'DYNAMODB | MYSQL | POSTGRESQL | REDSHIFT | '
                                   'ORACLE | SYNAPSE | SQLSERVER | DB2 | '
                                   'OPENSEARCH | BIGQUERY | GOOGLECLOUDSTORAGE '
                                   '| HBASE | DOCUMENTDB | MSK | NEPTUNE | '
                                   'CMDB | TPCDS | REDIS | CLOUDWATCH | '
                                   'TIMESTREAM | SAPHANA | SNOWFLAKE | '
                                   'TERADATA | VERTICA | CLOUDERAIMPALA | '
                                   'CLOUDERAHIVE | HORTONWORKSHIVE | '
                                   'DATALAKEGEN2 | DB2AS400 | '
                                   'CLOUDWATCHMETRICS',
                 'Error': 'string',
                 'Status': 'CREATE_IN_PROGRESS | CREATE_COMPLETE | '
                           'CREATE_FAILED | CREATE_FAILED_CLEANUP_IN_PROGRESS '
                           '| CREATE_FAILED_CLEANUP_COMPLETE | '
                           'CREATE_FAILED_CLEANUP_FAILED | DELETE_IN_PROGRESS '
                           '| DELETE_COMPLETE | DELETE_FAILED',
                 'Type': {'FEDERATED'}}}

Returns the specified data catalog.

See also: AWS API Documentation

Request Syntax

client.get_data_catalog(
    Name='string',
    WorkGroup='string'
)
type Name

string

param Name

[REQUIRED]

The name of the data catalog to return.

type WorkGroup

string

param WorkGroup

The name of the workgroup. Required if making an IAM Identity Center request.

rtype

dict

returns

Response Syntax

{
    'DataCatalog': {
        'Name': 'string',
        'Description': 'string',
        'Type': 'LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
        'Parameters': {
            'string': 'string'
        },
        'Status': 'CREATE_IN_PROGRESS'|'CREATE_COMPLETE'|'CREATE_FAILED'|'CREATE_FAILED_CLEANUP_IN_PROGRESS'|'CREATE_FAILED_CLEANUP_COMPLETE'|'CREATE_FAILED_CLEANUP_FAILED'|'DELETE_IN_PROGRESS'|'DELETE_COMPLETE'|'DELETE_FAILED',
        'ConnectionType': 'DYNAMODB'|'MYSQL'|'POSTGRESQL'|'REDSHIFT'|'ORACLE'|'SYNAPSE'|'SQLSERVER'|'DB2'|'OPENSEARCH'|'BIGQUERY'|'GOOGLECLOUDSTORAGE'|'HBASE'|'DOCUMENTDB'|'MSK'|'NEPTUNE'|'CMDB'|'TPCDS'|'REDIS'|'CLOUDWATCH'|'TIMESTREAM'|'SAPHANA'|'SNOWFLAKE'|'TERADATA'|'VERTICA'|'CLOUDERAIMPALA'|'CLOUDERAHIVE'|'HORTONWORKSHIVE'|'DATALAKEGEN2'|'DB2AS400'|'CLOUDWATCHMETRICS',
        'Error': 'string'
    }
}

Response Structure

  • (dict) --

    • DataCatalog (dict) --

      The data catalog returned.

      • Name (string) --

        The name of the data catalog. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

      • Description (string) --

        An optional description of the data catalog.

      • Type (string) --

        The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass.

      • Parameters (dict) --

        Specifies the Lambda function or functions to use for the data catalog. This is a mapping whose values depend on the catalog type.

        • For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number

        • For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both.

          • If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn

          • If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn

        • The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue catalog belongs. catalog-id=catalog_id

          • The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify.

        • The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection.

          • connection-arn:<glue_connection_arn_to_reuse>

          • connection-type:MYSQL|REDSHIFT|...., connection-properties:"<json_string>" For <json_string> , use escaped JSON text, as in the following example. "{\"spill_bucket\":\"my_spill\",\"spill_prefix\":\"athena-spill\",\"host\":\"abc12345.snowflakecomputing.com\",\"port\":\"1234\",\"warehouse\":\"DEV_WH\",\"database\":\"TEST\",\"schema\":\"PUBLIC\",\"SecretArn\":\"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j\"}"

        • (string) --

          • (string) --

      • Status (string) --

        The status of the creation or deletion of the data catalog.

        • The LAMBDA , GLUE , and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED .

        • The FEDERATED data catalog type is created asynchronously.

        Data catalog creation status:

        • CREATE_IN_PROGRESS : Federated data catalog creation in progress.

        • CREATE_COMPLETE : Data catalog creation complete.

        • CREATE_FAILED : Data catalog could not be created.

        • CREATE_FAILED_CLEANUP_IN_PROGRESS : Federated data catalog creation failed and is being removed.

        • CREATE_FAILED_CLEANUP_COMPLETE : Federated data catalog creation failed and was removed.

        • CREATE_FAILED_CLEANUP_FAILED : Federated data catalog creation failed but could not be removed.

        Data catalog deletion status:

        • DELETE_IN_PROGRESS : Federated data catalog deletion in progress.

        • DELETE_COMPLETE : Federated data catalog deleted.

        • DELETE_FAILED : Federated data catalog could not be deleted.

      • ConnectionType (string) --

        The type of connection for a FEDERATED data catalog (for example, REDSHIFT , MYSQL , or SQLSERVER ). For information about individual connectors, see Available data source connectors.

      • Error (string) --

        Text of the error that occurred during data catalog creation or deletion.

ListDataCatalogs (updated) Link ¶
Changes (response) 
{'DataCatalogsSummary': {'ConnectionType': 'DYNAMODB | MYSQL | POSTGRESQL | '
                                           'REDSHIFT | ORACLE | SYNAPSE | '
                                           'SQLSERVER | DB2 | OPENSEARCH | '
                                           'BIGQUERY | GOOGLECLOUDSTORAGE | '
                                           'HBASE | DOCUMENTDB | MSK | NEPTUNE '
                                           '| CMDB | TPCDS | REDIS | '
                                           'CLOUDWATCH | TIMESTREAM | SAPHANA '
                                           '| SNOWFLAKE | TERADATA | VERTICA | '
                                           'CLOUDERAIMPALA | CLOUDERAHIVE | '
                                           'HORTONWORKSHIVE | DATALAKEGEN2 | '
                                           'DB2AS400 | CLOUDWATCHMETRICS',
                         'Error': 'string',
                         'Status': 'CREATE_IN_PROGRESS | CREATE_COMPLETE | '
                                   'CREATE_FAILED | '
                                   'CREATE_FAILED_CLEANUP_IN_PROGRESS | '
                                   'CREATE_FAILED_CLEANUP_COMPLETE | '
                                   'CREATE_FAILED_CLEANUP_FAILED | '
                                   'DELETE_IN_PROGRESS | DELETE_COMPLETE | '
                                   'DELETE_FAILED',
                         'Type': {'FEDERATED'}}}

Lists the data catalogs in the current Amazon Web Services account.

Note

In the Athena console, data catalogs are listed as "data sources" on the Data sources page under the Data source name column.

See also: AWS API Documentation

Request Syntax

client.list_data_catalogs(
    NextToken='string',
    MaxResults=123,
    WorkGroup='string'
)
type NextToken

string

param NextToken

A token generated by the Athena service that specifies where to continue pagination if a previous request was truncated. To obtain the next set of pages, pass in the NextToken from the response object of the previous page call.

type MaxResults

integer

param MaxResults

Specifies the maximum number of data catalogs to return.

type WorkGroup

string

param WorkGroup

The name of the workgroup. Required if making an IAM Identity Center request.

rtype

dict

returns

Response Syntax

{
    'DataCatalogsSummary': [
        {
            'CatalogName': 'string',
            'Type': 'LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
            'Status': 'CREATE_IN_PROGRESS'|'CREATE_COMPLETE'|'CREATE_FAILED'|'CREATE_FAILED_CLEANUP_IN_PROGRESS'|'CREATE_FAILED_CLEANUP_COMPLETE'|'CREATE_FAILED_CLEANUP_FAILED'|'DELETE_IN_PROGRESS'|'DELETE_COMPLETE'|'DELETE_FAILED',
            'ConnectionType': 'DYNAMODB'|'MYSQL'|'POSTGRESQL'|'REDSHIFT'|'ORACLE'|'SYNAPSE'|'SQLSERVER'|'DB2'|'OPENSEARCH'|'BIGQUERY'|'GOOGLECLOUDSTORAGE'|'HBASE'|'DOCUMENTDB'|'MSK'|'NEPTUNE'|'CMDB'|'TPCDS'|'REDIS'|'CLOUDWATCH'|'TIMESTREAM'|'SAPHANA'|'SNOWFLAKE'|'TERADATA'|'VERTICA'|'CLOUDERAIMPALA'|'CLOUDERAHIVE'|'HORTONWORKSHIVE'|'DATALAKEGEN2'|'DB2AS400'|'CLOUDWATCHMETRICS',
            'Error': 'string'
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • DataCatalogsSummary (list) --

      A summary list of data catalogs.

      • (dict) --

        The summary information for the data catalog, which includes its name and type.

        • CatalogName (string) --

          The name of the data catalog. The catalog name is unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

        • Type (string) --

          The data catalog type.

        • Status (string) --

          The status of the creation or deletion of the data catalog.

          • The LAMBDA , GLUE , and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED .

          • The FEDERATED data catalog type is created asynchronously.

          Data catalog creation status:

          • CREATE_IN_PROGRESS : Federated data catalog creation in progress.

          • CREATE_COMPLETE : Data catalog creation complete.

          • CREATE_FAILED : Data catalog could not be created.

          • CREATE_FAILED_CLEANUP_IN_PROGRESS : Federated data catalog creation failed and is being removed.

          • CREATE_FAILED_CLEANUP_COMPLETE : Federated data catalog creation failed and was removed.

          • CREATE_FAILED_CLEANUP_FAILED : Federated data catalog creation failed but could not be removed.

          Data catalog deletion status:

          • DELETE_IN_PROGRESS : Federated data catalog deletion in progress.

          • DELETE_COMPLETE : Federated data catalog deleted.

          • DELETE_FAILED : Federated data catalog could not be deleted.

        • ConnectionType (string) --

          The type of connection for a FEDERATED data catalog (for example, REDSHIFT , MYSQL , or SQLSERVER ). For information about individual connectors, see Available data source connectors.

        • Error (string) --

          Text of the error that occurred during data catalog creation or deletion.

    • NextToken (string) --

      A token generated by the Athena service that specifies where to continue pagination if a previous request was truncated. To obtain the next set of pages, pass in the NextToken from the response object of the previous page call.

UpdateDataCatalog (updated) Link ¶
Changes (request) 
{'Type': {'FEDERATED'}}

Updates the data catalog that has the specified name.

See also: AWS API Documentation

Request Syntax

client.update_data_catalog(
    Name='string',
    Type='LAMBDA'|'GLUE'|'HIVE'|'FEDERATED',
    Description='string',
    Parameters={
        'string': 'string'
    }
)
type Name

string

param Name

[REQUIRED]

The name of the data catalog to update. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena.

type Type

string

param Type

[REQUIRED]

Specifies the type of data catalog to update. Specify LAMBDA for a federated catalog, HIVE for an external hive metastore, or GLUE for an Glue Data Catalog.

type Description

string

param Description

New or modified text that describes the data catalog.

type Parameters

dict

param Parameters

Specifies the Lambda function or functions to use for updating the data catalog. This is a mapping whose values depend on the catalog type.

  • For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number

  • For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both.

    • If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn

    • If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn

  • (string) --

    • (string) --

rtype

dict

returns

Response Syntax

{}

Response Structure

  • (dict) --