AWS API Changes

2024/10/01 - AWS Supply Chain - 13 new api methods

Changes Release DataLakeDataset, DataIntegrationFlow and ResourceTagging APIs for AWS Supply Chain

UpdateDataLakeDataset (new)

Update a data lake dataset.

Request Syntax

client.update_data_lake_dataset(
    instanceId='string',
    namespace='string',
    name='string',
    description='string'
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Chain instance identifier.

type namespace

string

param namespace

[REQUIRED]

The name space of the dataset. The available values are:

asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.

type name

string

param name

[REQUIRED]

The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.

type description

string

param description

The updated description of the data lake dataset.

rtype

dict

returns

Response Syntax

{
    'dataset': {
        'instanceId': 'string',
        'namespace': 'string',
        'name': 'string',
        'arn': 'string',
        'schema': {
            'name': 'string',
            'fields': [
                {
                    'name': 'string',
                    'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
                    'isRequired': True|False
                },
            ]
        },
        'description': 'string',
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) --

The response parameters of UpdateDataLakeDataset.
- dataset (dict) --
  
  The updated dataset details.
  - instanceId (string) --
    
    The Amazon Web Services Supply Chain instance identifier.
  - namespace (string) --
    
    The name space of the dataset. The available values are:
    - asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
    - default - For datasets with custom user-defined schemas.
  - name (string) --
    
    The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
  - arn (string) --
    
    The arn of the dataset.
  - schema (dict) --
    
    The schema of the dataset.
    - name (string) --
      
      The name of the dataset schema.
    - fields (list) --
      
      The list of field details of the dataset schema.
      - (dict) --
        
        The dataset field details.
        
        name (string) --
        
        The dataset field name.
        
        type (string) --
        
        The dataset field type.
        
        isRequired (boolean) --
        
        Indicate if the field is required or not.
  - description (string) --
    
    The description of the dataset.
  - createdTime (datetime) --
    
    The creation time of the dataset.
  - lastModifiedTime (datetime) --
    
    The last modified time of the dataset.

ListTagsForResource (new)

Link ¶

List all the tags for an Amazon Web ServicesSupply Chain resource.

See also: AWS API Documentation

Request Syntax

client.list_tags_for_resource(
    resourceArn='string'
)

type resourceArn

string

param resourceArn

[REQUIRED]

The Amazon Web Services Supply chain resource ARN that needs tags to be listed.

rtype

dict

returns

Response Syntax

{
    'tags': {
        'string': 'string'
    }
}

Response Structure

(dict) --

The response parameters of ListTagsForResource.
- tags (dict) --
  
  The tags added to an Amazon Web Services Supply Chain resource.
  - (string) --
    - (string) --

CreateDataIntegrationFlow (new)

Link ¶

Create DataIntegrationFlow to map one or more different sources to one target using the SQL transformation query.

See also: AWS API Documentation

Request Syntax

client.create_data_integration_flow(
    instanceId='string',
    name='string',
    sources=[
        {
            'sourceType': 'S3'|'DATASET',
            'sourceName': 'string',
            's3Source': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetSource': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
    ],
    transformation={
        'transformationType': 'SQL'|'NONE',
        'sqlTransformation': {
            'query': 'string'
        }
    },
    target={
        'targetType': 'S3'|'DATASET',
        's3Target': {
            'bucketName': 'string',
            'prefix': 'string',
            'options': {
                'fileType': 'CSV'|'PARQUET'|'JSON'
            }
        },
        'datasetTarget': {
            'datasetIdentifier': 'string',
            'options': {
                'loadType': 'INCREMENTAL'|'REPLACE',
                'dedupeRecords': True|False
            }
        }
    },
    tags={
        'string': 'string'
    }
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type name

string

param name

[REQUIRED]

Name of the DataIntegrationFlow.

type sources

list

param sources

[REQUIRED]

The source configurations for DataIntegrationFlow.

(dict) --

The DataIntegrationFlow source parameters.
- sourceType (string) -- [REQUIRED]
  
  The DataIntegrationFlow source type.
- sourceName (string) -- [REQUIRED]
  
  The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
- s3Source (dict) --
  
  The S3 DataIntegrationFlow source.
  - bucketName (string) -- [REQUIRED]
    
    The bucketName of the S3 source objects.
  - prefix (string) -- [REQUIRED]
    
    The prefix of the S3 source objects.
  - options (dict) --
    
    The other options of the S3 DataIntegrationFlow source.
    - fileType (string) --
      
      The Amazon S3 file type in S3 options.
- datasetSource (dict) --
  
  The dataset DataIntegrationFlow source.
  - datasetIdentifier (string) -- [REQUIRED]
    
    The ARN of the dataset.
  - options (dict) --
    
    The dataset DataIntegrationFlow source options.
    - loadType (string) --
      
      The dataset data load type in dataset options.
    - dedupeRecords (boolean) --
      
      The dataset load option to remove duplicates.

type transformation

dict

param transformation

[REQUIRED]

The transformation configurations for DataIntegrationFlow.

transformationType (string) -- [REQUIRED]

The DataIntegrationFlow transformation type.
sqlTransformation (dict) --

The SQL DataIntegrationFlow transformation configuration.
- query (string) -- [REQUIRED]
  
  The transformation SQL query body based on SparkSQL.

type target

dict

param target

[REQUIRED]

The target configurations for DataIntegrationFlow.

targetType (string) -- [REQUIRED]

The DataIntegrationFlow target type.
s3Target (dict) --

The S3 DataIntegrationFlow target.
- bucketName (string) -- [REQUIRED]
  
  The bucketName of the S3 target objects.
- prefix (string) -- [REQUIRED]
  
  The prefix of the S3 target objects.
- options (dict) --
  
  The S3 DataIntegrationFlow target options.
  - fileType (string) --
    
    The Amazon S3 file type in S3 options.
datasetTarget (dict) --

The dataset DataIntegrationFlow target.
- datasetIdentifier (string) -- [REQUIRED]
  
  The dataset ARN.
- options (dict) --
  
  The dataset DataIntegrationFlow target options.
  - loadType (string) --
    
    The dataset data load type in dataset options.
  - dedupeRecords (boolean) --
    
    The dataset load option to remove duplicates.

type tags

dict

param tags

The tags of the DataIntegrationFlow to be created

(string) --
- (string) --

rtype

dict

returns

Response Syntax

{
    'instanceId': 'string',
    'name': 'string'
}

Response Structure

(dict) --

The response parameters for CreateDataIntegrationFlow.
- instanceId (string) --
  
  The Amazon Web Services Supply Chain instance identifier.
- name (string) --
  
  The name of the DataIntegrationFlow created.

TagResource (new)

Link ¶

Create tags for an Amazon Web Services Supply chain resource.

See also: AWS API Documentation

Request Syntax

client.tag_resource(
    resourceArn='string',
    tags={
        'string': 'string'
    }
)

type resourceArn

string

param resourceArn

[REQUIRED]

The Amazon Web Services Supply chain resource ARN that needs to be tagged.

type tags

dict

param tags

[REQUIRED]

The tags of the Amazon Web Services Supply chain resource to be created.

(string) --
- (string) --

rtype

dict

returns

Response Syntax

{}

Response Structure

(dict) --

The response parameters for TagResource.

ListDataIntegrationFlows (new)

Link ¶

Lists all the DataIntegrationFlows in a paginated way.

See also: AWS API Documentation

Request Syntax

client.list_data_integration_flows(
    instanceId='string',
    nextToken='string',
    maxResults=123
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type nextToken

string

param nextToken

The pagination token to fetch the next page of the DataIntegrationFlows.

type maxResults

integer

param maxResults

Specify the maximum number of DataIntegrationFlows to fetch in one paginated request.

rtype

dict

returns

Response Syntax

{
    'flows': [
        {
            'instanceId': 'string',
            'name': 'string',
            'sources': [
                {
                    'sourceType': 'S3'|'DATASET',
                    'sourceName': 'string',
                    's3Source': {
                        'bucketName': 'string',
                        'prefix': 'string',
                        'options': {
                            'fileType': 'CSV'|'PARQUET'|'JSON'
                        }
                    },
                    'datasetSource': {
                        'datasetIdentifier': 'string',
                        'options': {
                            'loadType': 'INCREMENTAL'|'REPLACE',
                            'dedupeRecords': True|False
                        }
                    }
                },
            ],
            'transformation': {
                'transformationType': 'SQL'|'NONE',
                'sqlTransformation': {
                    'query': 'string'
                }
            },
            'target': {
                'targetType': 'S3'|'DATASET',
                's3Target': {
                    'bucketName': 'string',
                    'prefix': 'string',
                    'options': {
                        'fileType': 'CSV'|'PARQUET'|'JSON'
                    }
                },
                'datasetTarget': {
                    'datasetIdentifier': 'string',
                    'options': {
                        'loadType': 'INCREMENTAL'|'REPLACE',
                        'dedupeRecords': True|False
                    }
                }
            },
            'createdTime': datetime(2015, 1, 1),
            'lastModifiedTime': datetime(2015, 1, 1)
        },
    ],
    'nextToken': 'string'
}

Response Structure

(dict) --

The response parameters for ListDataIntegrationFlows.
- flows (list) --
  
  The response parameters for ListDataIntegrationFlows.
  - (dict) --
    
    The DataIntegrationFlow details.
    - instanceId (string) --
      
      The DataIntegrationFlow instance ID.
    - name (string) --
      
      The DataIntegrationFlow name.
    - sources (list) --
      
      The DataIntegrationFlow source configurations.
      - (dict) --
        
        The DataIntegrationFlow source parameters.
        
        sourceType (string) --
        
        The DataIntegrationFlow source type.
        
        sourceName (string) --
        
        The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
        
        s3Source (dict) --
        
        The S3 DataIntegrationFlow source.
        
        bucketName (string) --
        
        The bucketName of the S3 source objects.
        
        prefix (string) --
        
        The prefix of the S3 source objects.
        
        options (dict) --
        
        The other options of the S3 DataIntegrationFlow source.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
        
        datasetSource (dict) --
        
        The dataset DataIntegrationFlow source.
        
        datasetIdentifier (string) --
        
        The ARN of the dataset.
        
        options (dict) --
        
        The dataset DataIntegrationFlow source options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
    - transformation (dict) --
      
      The DataIntegrationFlow transformation configurations.
      - transformationType (string) --
        
        The DataIntegrationFlow transformation type.
      - sqlTransformation (dict) --
        
        The SQL DataIntegrationFlow transformation configuration.
        
        query (string) --
        
        The transformation SQL query body based on SparkSQL.
    - target (dict) --
      
      The DataIntegrationFlow target configuration.
      - targetType (string) --
        
        The DataIntegrationFlow target type.
      - s3Target (dict) --
        
        The S3 DataIntegrationFlow target.
        
        bucketName (string) --
        
        The bucketName of the S3 target objects.
        
        prefix (string) --
        
        The prefix of the S3 target objects.
        
        options (dict) --
        
        The S3 DataIntegrationFlow target options.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
      - datasetTarget (dict) --
        
        The dataset DataIntegrationFlow target.
        
        datasetIdentifier (string) --
        
        The dataset ARN.
        
        options (dict) --
        
        The dataset DataIntegrationFlow target options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
    - createdTime (datetime) --
      
      The DataIntegrationFlow creation timestamp.
    - lastModifiedTime (datetime) --
      
      The DataIntegrationFlow last modified timestamp.
- nextToken (string) --
  
  The pagination token to fetch the next page of the DataIntegrationFlows.

CreateDataLakeDataset (new)

Link ¶

Create a data lake dataset.

See also: AWS API Documentation

Request Syntax

client.create_data_lake_dataset(
    instanceId='string',
    namespace='string',
    name='string',
    schema={
        'name': 'string',
        'fields': [
            {
                'name': 'string',
                'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
                'isRequired': True|False
            },
        ]
    },
    description='string',
    tags={
        'string': 'string'
    }
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type namespace

string

param namespace

[REQUIRED]

The name space of the dataset.

asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.

type name

string

param name

[REQUIRED]

The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.

type schema

dict

param schema

The custom schema of the data lake dataset and is only required when the name space is default .

name (string) -- [REQUIRED]

The name of the dataset schema.
fields (list) -- [REQUIRED]

The list of field details of the dataset schema.
- (dict) --
  
  The dataset field details.
  - name (string) -- [REQUIRED]
    
    The dataset field name.
  - type (string) -- [REQUIRED]
    
    The dataset field type.
  - isRequired (boolean) -- [REQUIRED]
    
    Indicate if the field is required or not.

type description

string

param description

The description of the dataset.

type tags

dict

param tags

The tags of the dataset.

(string) --
- (string) --

rtype

dict

returns

Response Syntax

{
    'dataset': {
        'instanceId': 'string',
        'namespace': 'string',
        'name': 'string',
        'arn': 'string',
        'schema': {
            'name': 'string',
            'fields': [
                {
                    'name': 'string',
                    'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
                    'isRequired': True|False
                },
            ]
        },
        'description': 'string',
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) --

The response parameters of CreateDataLakeDataset.
- dataset (dict) --
  
  The detail of created dataset.
  - instanceId (string) --
    
    The Amazon Web Services Supply Chain instance identifier.
  - namespace (string) --
    
    The name space of the dataset. The available values are:
    - asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
    - default - For datasets with custom user-defined schemas.
  - name (string) --
    
    The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
  - arn (string) --
    
    The arn of the dataset.
  - schema (dict) --
    
    The schema of the dataset.
    - name (string) --
      
      The name of the dataset schema.
    - fields (list) --
      
      The list of field details of the dataset schema.
      - (dict) --
        
        The dataset field details.
        
        name (string) --
        
        The dataset field name.
        
        type (string) --
        
        The dataset field type.
        
        isRequired (boolean) --
        
        Indicate if the field is required or not.
  - description (string) --
    
    The description of the dataset.
  - createdTime (datetime) --
    
    The creation time of the dataset.
  - lastModifiedTime (datetime) --
    
    The last modified time of the dataset.

GetDataIntegrationFlow (new)

Link ¶

View the DataIntegrationFlow details.

See also: AWS API Documentation

Request Syntax

client.get_data_integration_flow(
    instanceId='string',
    name='string'
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type name

string

param name

[REQUIRED]

The name of the DataIntegrationFlow created.

rtype

dict

returns

Response Syntax

{
    'flow': {
        'instanceId': 'string',
        'name': 'string',
        'sources': [
            {
                'sourceType': 'S3'|'DATASET',
                'sourceName': 'string',
                's3Source': {
                    'bucketName': 'string',
                    'prefix': 'string',
                    'options': {
                        'fileType': 'CSV'|'PARQUET'|'JSON'
                    }
                },
                'datasetSource': {
                    'datasetIdentifier': 'string',
                    'options': {
                        'loadType': 'INCREMENTAL'|'REPLACE',
                        'dedupeRecords': True|False
                    }
                }
            },
        ],
        'transformation': {
            'transformationType': 'SQL'|'NONE',
            'sqlTransformation': {
                'query': 'string'
            }
        },
        'target': {
            'targetType': 'S3'|'DATASET',
            's3Target': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetTarget': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) --

The response parameters for GetDataIntegrationFlow.
- flow (dict) --
  
  The details of the DataIntegrationFlow returned.
  - instanceId (string) --
    
    The DataIntegrationFlow instance ID.
  - name (string) --
    
    The DataIntegrationFlow name.
  - sources (list) --
    
    The DataIntegrationFlow source configurations.
    - (dict) --
      
      The DataIntegrationFlow source parameters.
      - sourceType (string) --
        
        The DataIntegrationFlow source type.
      - sourceName (string) --
        
        The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
      - s3Source (dict) --
        
        The S3 DataIntegrationFlow source.
        
        bucketName (string) --
        
        The bucketName of the S3 source objects.
        
        prefix (string) --
        
        The prefix of the S3 source objects.
        
        options (dict) --
        
        The other options of the S3 DataIntegrationFlow source.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
      - datasetSource (dict) --
        
        The dataset DataIntegrationFlow source.
        
        datasetIdentifier (string) --
        
        The ARN of the dataset.
        
        options (dict) --
        
        The dataset DataIntegrationFlow source options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
  - transformation (dict) --
    
    The DataIntegrationFlow transformation configurations.
    - transformationType (string) --
      
      The DataIntegrationFlow transformation type.
    - sqlTransformation (dict) --
      
      The SQL DataIntegrationFlow transformation configuration.
      - query (string) --
        
        The transformation SQL query body based on SparkSQL.
  - target (dict) --
    
    The DataIntegrationFlow target configuration.
    - targetType (string) --
      
      The DataIntegrationFlow target type.
    - s3Target (dict) --
      
      The S3 DataIntegrationFlow target.
      - bucketName (string) --
        
        The bucketName of the S3 target objects.
      - prefix (string) --
        
        The prefix of the S3 target objects.
      - options (dict) --
        
        The S3 DataIntegrationFlow target options.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
    - datasetTarget (dict) --
      
      The dataset DataIntegrationFlow target.
      - datasetIdentifier (string) --
        
        The dataset ARN.
      - options (dict) --
        
        The dataset DataIntegrationFlow target options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
  - createdTime (datetime) --
    
    The DataIntegrationFlow creation timestamp.
  - lastModifiedTime (datetime) --
    
    The DataIntegrationFlow last modified timestamp.

DeleteDataLakeDataset (new)

Link ¶

Delete a data lake dataset.

See also: AWS API Documentation

Request Syntax

client.delete_data_lake_dataset(
    instanceId='string',
    namespace='string',
    name='string'
)

type instanceId

string

param instanceId

[REQUIRED]

The AWS Supply Chain instance identifier.

type namespace

string

param namespace

[REQUIRED]

The namespace of the dataset. The available values are:

asc: for AWS Supply Chain supported datasets.
default: for datasets with custom user-defined schemas.

type name

string

param name

[REQUIRED]

The name of the dataset. If the namespace is asc , the name must be one of the supported data entities.

rtype

dict

returns

Response Syntax

{
    'instanceId': 'string',
    'namespace': 'string',
    'name': 'string'
}

Response Structure

(dict) --

The response parameters of DeleteDataLakeDataset.
- instanceId (string) --
  
  The AWS Supply Chain instance identifier.
- namespace (string) --
  
  The namespace of deleted dataset.
- name (string) --
  
  The name of deleted dataset.

DeleteDataIntegrationFlow (new)

Link ¶

Delete the DataIntegrationFlow.

See also: AWS API Documentation

Request Syntax

client.delete_data_integration_flow(
    instanceId='string',
    name='string'
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type name

string

param name

[REQUIRED]

The name of the DataIntegrationFlow to be deleted.

rtype

dict

returns

Response Syntax

{
    'instanceId': 'string',
    'name': 'string'
}

Response Structure

(dict) --

The response parameters for DeleteDataIntegrationFlow.
- instanceId (string) --
  
  The Amazon Web Services Supply Chain instance identifier.
- name (string) --
  
  The name of the DataIntegrationFlow deleted.

UntagResource (new)

Link ¶

Delete tags for an Amazon Web Services Supply chain resource.

See also: AWS API Documentation

Request Syntax

client.untag_resource(
    resourceArn='string',
    tagKeys=[
        'string',
    ]
)

type resourceArn

string

param resourceArn

[REQUIRED]

The Amazon Web Services Supply chain resource ARN that needs to be untagged.

type tagKeys

list

param tagKeys

[REQUIRED]

The list of tag keys to be deleted for an Amazon Web Services Supply Chain resource.

(string) --

rtype

dict

returns

Response Syntax

{}

Response Structure

(dict) --

The response parameters of UntagResource.

ListDataLakeDatasets (new)

Link ¶

List the data lake datasets for a specific instance and name space.

See also: AWS API Documentation

Request Syntax

client.list_data_lake_datasets(
    instanceId='string',
    namespace='string',
    nextToken='string',
    maxResults=123
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type namespace

string

param namespace

[REQUIRED]

The namespace of the dataset. The available values are:

asc: for AWS Supply Chain supported datasets.
default: for datasets with custom user-defined schemas.

type nextToken

string

param nextToken

The pagination token to fetch next page of datasets.

type maxResults

integer

param maxResults

The max number of datasets to fetch in this paginated request.

rtype

dict

returns

Response Syntax

{
    'datasets': [
        {
            'instanceId': 'string',
            'namespace': 'string',
            'name': 'string',
            'arn': 'string',
            'schema': {
                'name': 'string',
                'fields': [
                    {
                        'name': 'string',
                        'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
                        'isRequired': True|False
                    },
                ]
            },
            'description': 'string',
            'createdTime': datetime(2015, 1, 1),
            'lastModifiedTime': datetime(2015, 1, 1)
        },
    ],
    'nextToken': 'string'
}

Response Structure

(dict) --

The response parameters of ListDataLakeDatasets.
- datasets (list) --
  
  The list of fetched dataset details.
  - (dict) --
    
    The data lake dataset details.
    - instanceId (string) --
      
      The Amazon Web Services Supply Chain instance identifier.
    - namespace (string) --
      
      The name space of the dataset. The available values are:
      - asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
      - default - For datasets with custom user-defined schemas.
    - name (string) --
      
      The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
    - arn (string) --
      
      The arn of the dataset.
    - schema (dict) --
      
      The schema of the dataset.
      - name (string) --
        
        The name of the dataset schema.
      - fields (list) --
        
        The list of field details of the dataset schema.
        
        (dict) --
        
        The dataset field details.
        
        name (string) --
        
        The dataset field name.
        
        type (string) --
        
        The dataset field type.
        
        isRequired (boolean) --
        
        Indicate if the field is required or not.
    - description (string) --
      
      The description of the dataset.
    - createdTime (datetime) --
      
      The creation time of the dataset.
    - lastModifiedTime (datetime) --
      
      The last modified time of the dataset.
- nextToken (string) --
  
  The pagination token to fetch next page of datasets.

UpdateDataIntegrationFlow (new)

Link ¶

Update the DataIntegrationFlow.

See also: AWS API Documentation

Request Syntax

client.update_data_integration_flow(
    instanceId='string',
    name='string',
    sources=[
        {
            'sourceType': 'S3'|'DATASET',
            'sourceName': 'string',
            's3Source': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetSource': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
    ],
    transformation={
        'transformationType': 'SQL'|'NONE',
        'sqlTransformation': {
            'query': 'string'
        }
    },
    target={
        'targetType': 'S3'|'DATASET',
        's3Target': {
            'bucketName': 'string',
            'prefix': 'string',
            'options': {
                'fileType': 'CSV'|'PARQUET'|'JSON'
            }
        },
        'datasetTarget': {
            'datasetIdentifier': 'string',
            'options': {
                'loadType': 'INCREMENTAL'|'REPLACE',
                'dedupeRecords': True|False
            }
        }
    }
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type name

string

param name

[REQUIRED]

The name of the DataIntegrationFlow to be updated.

type sources

list

param sources

The new source configurations for the DataIntegrationFlow.

(dict) --

The DataIntegrationFlow source parameters.
- sourceType (string) -- [REQUIRED]
  
  The DataIntegrationFlow source type.
- sourceName (string) -- [REQUIRED]
  
  The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
- s3Source (dict) --
  
  The S3 DataIntegrationFlow source.
  - bucketName (string) -- [REQUIRED]
    
    The bucketName of the S3 source objects.
  - prefix (string) -- [REQUIRED]
    
    The prefix of the S3 source objects.
  - options (dict) --
    
    The other options of the S3 DataIntegrationFlow source.
    - fileType (string) --
      
      The Amazon S3 file type in S3 options.
- datasetSource (dict) --
  
  The dataset DataIntegrationFlow source.
  - datasetIdentifier (string) -- [REQUIRED]
    
    The ARN of the dataset.
  - options (dict) --
    
    The dataset DataIntegrationFlow source options.
    - loadType (string) --
      
      The dataset data load type in dataset options.
    - dedupeRecords (boolean) --
      
      The dataset load option to remove duplicates.

type transformation

dict

param transformation

The new transformation configurations for the DataIntegrationFlow.

transformationType (string) -- [REQUIRED]

The DataIntegrationFlow transformation type.
sqlTransformation (dict) --

The SQL DataIntegrationFlow transformation configuration.
- query (string) -- [REQUIRED]
  
  The transformation SQL query body based on SparkSQL.

type target

dict

param target

The new target configurations for the DataIntegrationFlow.

targetType (string) -- [REQUIRED]

The DataIntegrationFlow target type.
s3Target (dict) --

The S3 DataIntegrationFlow target.
- bucketName (string) -- [REQUIRED]
  
  The bucketName of the S3 target objects.
- prefix (string) -- [REQUIRED]
  
  The prefix of the S3 target objects.
- options (dict) --
  
  The S3 DataIntegrationFlow target options.
  - fileType (string) --
    
    The Amazon S3 file type in S3 options.
datasetTarget (dict) --

The dataset DataIntegrationFlow target.
- datasetIdentifier (string) -- [REQUIRED]
  
  The dataset ARN.
- options (dict) --
  
  The dataset DataIntegrationFlow target options.
  - loadType (string) --
    
    The dataset data load type in dataset options.
  - dedupeRecords (boolean) --
    
    The dataset load option to remove duplicates.

rtype

dict

returns

Response Syntax

{
    'flow': {
        'instanceId': 'string',
        'name': 'string',
        'sources': [
            {
                'sourceType': 'S3'|'DATASET',
                'sourceName': 'string',
                's3Source': {
                    'bucketName': 'string',
                    'prefix': 'string',
                    'options': {
                        'fileType': 'CSV'|'PARQUET'|'JSON'
                    }
                },
                'datasetSource': {
                    'datasetIdentifier': 'string',
                    'options': {
                        'loadType': 'INCREMENTAL'|'REPLACE',
                        'dedupeRecords': True|False
                    }
                }
            },
        ],
        'transformation': {
            'transformationType': 'SQL'|'NONE',
            'sqlTransformation': {
                'query': 'string'
            }
        },
        'target': {
            'targetType': 'S3'|'DATASET',
            's3Target': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetTarget': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) --

The response parameters for UpdateDataIntegrationFlow.
- flow (dict) --
  
  The details of the updated DataIntegrationFlow.
  - instanceId (string) --
    
    The DataIntegrationFlow instance ID.
  - name (string) --
    
    The DataIntegrationFlow name.
  - sources (list) --
    
    The DataIntegrationFlow source configurations.
    - (dict) --
      
      The DataIntegrationFlow source parameters.
      - sourceType (string) --
        
        The DataIntegrationFlow source type.
      - sourceName (string) --
        
        The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
      - s3Source (dict) --
        
        The S3 DataIntegrationFlow source.
        
        bucketName (string) --
        
        The bucketName of the S3 source objects.
        
        prefix (string) --
        
        The prefix of the S3 source objects.
        
        options (dict) --
        
        The other options of the S3 DataIntegrationFlow source.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
      - datasetSource (dict) --
        
        The dataset DataIntegrationFlow source.
        
        datasetIdentifier (string) --
        
        The ARN of the dataset.
        
        options (dict) --
        
        The dataset DataIntegrationFlow source options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
  - transformation (dict) --
    
    The DataIntegrationFlow transformation configurations.
    - transformationType (string) --
      
      The DataIntegrationFlow transformation type.
    - sqlTransformation (dict) --
      
      The SQL DataIntegrationFlow transformation configuration.
      - query (string) --
        
        The transformation SQL query body based on SparkSQL.
  - target (dict) --
    
    The DataIntegrationFlow target configuration.
    - targetType (string) --
      
      The DataIntegrationFlow target type.
    - s3Target (dict) --
      
      The S3 DataIntegrationFlow target.
      - bucketName (string) --
        
        The bucketName of the S3 target objects.
      - prefix (string) --
        
        The prefix of the S3 target objects.
      - options (dict) --
        
        The S3 DataIntegrationFlow target options.
        
        fileType (string) --
        
        The Amazon S3 file type in S3 options.
    - datasetTarget (dict) --
      
      The dataset DataIntegrationFlow target.
      - datasetIdentifier (string) --
        
        The dataset ARN.
      - options (dict) --
        
        The dataset DataIntegrationFlow target options.
        
        loadType (string) --
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) --
        
        The dataset load option to remove duplicates.
  - createdTime (datetime) --
    
    The DataIntegrationFlow creation timestamp.
  - lastModifiedTime (datetime) --
    
    The DataIntegrationFlow last modified timestamp.

GetDataLakeDataset (new)

Link ¶

Get a data lake dataset.

See also: AWS API Documentation

Request Syntax

client.get_data_lake_dataset(
    instanceId='string',
    namespace='string',
    name='string'
)

type instanceId

string

param instanceId

[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.

type namespace

string

param namespace

[REQUIRED]

The name space of the dataset. The available values are:

asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.

type name

string

param name

[REQUIRED]

The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.

rtype

dict

returns

Response Syntax

{
    'dataset': {
        'instanceId': 'string',
        'namespace': 'string',
        'name': 'string',
        'arn': 'string',
        'schema': {
            'name': 'string',
            'fields': [
                {
                    'name': 'string',
                    'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
                    'isRequired': True|False
                },
            ]
        },
        'description': 'string',
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) --

The response parameters for UpdateDataLakeDataset.
- dataset (dict) --
  
  The fetched dataset details.
  - instanceId (string) --
    
    The Amazon Web Services Supply Chain instance identifier.
  - namespace (string) --
    
    The name space of the dataset. The available values are:
    - asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
    - default - For datasets with custom user-defined schemas.
  - name (string) --
    
    The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
  - arn (string) --
    
    The arn of the dataset.
  - schema (dict) --
    
    The schema of the dataset.
    - name (string) --
      
      The name of the dataset schema.
    - fields (list) --
      
      The list of field details of the dataset schema.
      - (dict) --
        
        The dataset field details.
        
        name (string) --
        
        The dataset field name.
        
        type (string) --
        
        The dataset field type.
        
        isRequired (boolean) --
        
        Indicate if the field is required or not.
  - description (string) --
    
    The description of the dataset.
  - createdTime (datetime) --
    
    The creation time of the dataset.
  - lastModifiedTime (datetime) --
    
    The last modified time of the dataset.