2024/10/01 - AWS Supply Chain - 13 new api methods
Changes Release DataLakeDataset, DataIntegrationFlow and ResourceTagging APIs for AWS Supply Chain
Update a data lake dataset.
See also: AWS API Documentation
Request Syntax
client.update_data_lake_dataset(
instanceId='string',
namespace='string',
name='string',
description='string'
)
string
[REQUIRED]
The Amazon Web Services Chain instance identifier.
string
[REQUIRED]
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
string
[REQUIRED]
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
string
The updated description of the data lake dataset.
dict
Response Syntax
{
'dataset': {
'instanceId': 'string',
'namespace': 'string',
'name': 'string',
'arn': 'string',
'schema': {
'name': 'string',
'fields': [
{
'name': 'string',
'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
'isRequired': True|False
},
]
},
'description': 'string',
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
}
}
Response Structure
(dict) --
The response parameters of UpdateDataLakeDataset.
dataset (dict) --
The updated dataset details.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
namespace (string) --
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
name (string) --
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
arn (string) --
The arn of the dataset.
schema (dict) --
The schema of the dataset.
name (string) --
The name of the dataset schema.
fields (list) --
The list of field details of the dataset schema.
(dict) --
The dataset field details.
name (string) --
The dataset field name.
type (string) --
The dataset field type.
isRequired (boolean) --
Indicate if the field is required or not.
description (string) --
The description of the dataset.
createdTime (datetime) --
The creation time of the dataset.
lastModifiedTime (datetime) --
The last modified time of the dataset.
Create tags for an Amazon Web Services Supply chain resource.
See also: AWS API Documentation
Request Syntax
client.tag_resource(
resourceArn='string',
tags={
'string': 'string'
}
)
string
[REQUIRED]
The Amazon Web Services Supply chain resource ARN that needs to be tagged.
dict
[REQUIRED]
The tags of the Amazon Web Services Supply chain resource to be created.
(string) --
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
The response parameters for TagResource.
Update the DataIntegrationFlow.
See also: AWS API Documentation
Request Syntax
client.update_data_integration_flow(
instanceId='string',
name='string',
sources=[
{
'sourceType': 'S3'|'DATASET',
'sourceName': 'string',
's3Source': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetSource': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
],
transformation={
'transformationType': 'SQL'|'NONE',
'sqlTransformation': {
'query': 'string'
}
},
target={
'targetType': 'S3'|'DATASET',
's3Target': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetTarget': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
}
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The name of the DataIntegrationFlow to be updated.
list
The new source configurations for the DataIntegrationFlow.
(dict) --
The DataIntegrationFlow source parameters.
sourceType (string) -- [REQUIRED]
The DataIntegrationFlow source type.
sourceName (string) -- [REQUIRED]
The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
s3Source (dict) --
The S3 DataIntegrationFlow source.
bucketName (string) -- [REQUIRED]
The bucketName of the S3 source objects.
prefix (string) -- [REQUIRED]
The prefix of the S3 source objects.
options (dict) --
The other options of the S3 DataIntegrationFlow source.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetSource (dict) --
The dataset DataIntegrationFlow source.
datasetIdentifier (string) -- [REQUIRED]
The ARN of the dataset.
options (dict) --
The dataset DataIntegrationFlow source options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
dict
The new transformation configurations for the DataIntegrationFlow.
transformationType (string) -- [REQUIRED]
The DataIntegrationFlow transformation type.
sqlTransformation (dict) --
The SQL DataIntegrationFlow transformation configuration.
query (string) -- [REQUIRED]
The transformation SQL query body based on SparkSQL.
dict
The new target configurations for the DataIntegrationFlow.
targetType (string) -- [REQUIRED]
The DataIntegrationFlow target type.
s3Target (dict) --
The S3 DataIntegrationFlow target.
bucketName (string) -- [REQUIRED]
The bucketName of the S3 target objects.
prefix (string) -- [REQUIRED]
The prefix of the S3 target objects.
options (dict) --
The S3 DataIntegrationFlow target options.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetTarget (dict) --
The dataset DataIntegrationFlow target.
datasetIdentifier (string) -- [REQUIRED]
The dataset ARN.
options (dict) --
The dataset DataIntegrationFlow target options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
dict
Response Syntax
{
'flow': {
'instanceId': 'string',
'name': 'string',
'sources': [
{
'sourceType': 'S3'|'DATASET',
'sourceName': 'string',
's3Source': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetSource': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
],
'transformation': {
'transformationType': 'SQL'|'NONE',
'sqlTransformation': {
'query': 'string'
}
},
'target': {
'targetType': 'S3'|'DATASET',
's3Target': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetTarget': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
}
}
Response Structure
(dict) --
The response parameters for UpdateDataIntegrationFlow.
flow (dict) --
The details of the updated DataIntegrationFlow.
instanceId (string) --
The DataIntegrationFlow instance ID.
name (string) --
The DataIntegrationFlow name.
sources (list) --
The DataIntegrationFlow source configurations.
(dict) --
The DataIntegrationFlow source parameters.
sourceType (string) --
The DataIntegrationFlow source type.
sourceName (string) --
The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
s3Source (dict) --
The S3 DataIntegrationFlow source.
bucketName (string) --
The bucketName of the S3 source objects.
prefix (string) --
The prefix of the S3 source objects.
options (dict) --
The other options of the S3 DataIntegrationFlow source.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetSource (dict) --
The dataset DataIntegrationFlow source.
datasetIdentifier (string) --
The ARN of the dataset.
options (dict) --
The dataset DataIntegrationFlow source options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
transformation (dict) --
The DataIntegrationFlow transformation configurations.
transformationType (string) --
The DataIntegrationFlow transformation type.
sqlTransformation (dict) --
The SQL DataIntegrationFlow transformation configuration.
query (string) --
The transformation SQL query body based on SparkSQL.
target (dict) --
The DataIntegrationFlow target configuration.
targetType (string) --
The DataIntegrationFlow target type.
s3Target (dict) --
The S3 DataIntegrationFlow target.
bucketName (string) --
The bucketName of the S3 target objects.
prefix (string) --
The prefix of the S3 target objects.
options (dict) --
The S3 DataIntegrationFlow target options.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetTarget (dict) --
The dataset DataIntegrationFlow target.
datasetIdentifier (string) --
The dataset ARN.
options (dict) --
The dataset DataIntegrationFlow target options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
createdTime (datetime) --
The DataIntegrationFlow creation timestamp.
lastModifiedTime (datetime) --
The DataIntegrationFlow last modified timestamp.
List all the tags for an Amazon Web ServicesSupply Chain resource.
See also: AWS API Documentation
Request Syntax
client.list_tags_for_resource(
resourceArn='string'
)
string
[REQUIRED]
The Amazon Web Services Supply chain resource ARN that needs tags to be listed.
dict
Response Syntax
{
'tags': {
'string': 'string'
}
}
Response Structure
(dict) --
The response parameters of ListTagsForResource.
tags (dict) --
The tags added to an Amazon Web Services Supply Chain resource.
(string) --
(string) --
Create a data lake dataset.
See also: AWS API Documentation
Request Syntax
client.create_data_lake_dataset(
instanceId='string',
namespace='string',
name='string',
schema={
'name': 'string',
'fields': [
{
'name': 'string',
'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
'isRequired': True|False
},
]
},
description='string',
tags={
'string': 'string'
}
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The name space of the dataset.
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
string
[REQUIRED]
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
dict
The custom schema of the data lake dataset and is only required when the name space is default.
name (string) -- [REQUIRED]
The name of the dataset schema.
fields (list) -- [REQUIRED]
The list of field details of the dataset schema.
(dict) --
The dataset field details.
name (string) -- [REQUIRED]
The dataset field name.
type (string) -- [REQUIRED]
The dataset field type.
isRequired (boolean) -- [REQUIRED]
Indicate if the field is required or not.
string
The description of the dataset.
dict
The tags of the dataset.
(string) --
(string) --
dict
Response Syntax
{
'dataset': {
'instanceId': 'string',
'namespace': 'string',
'name': 'string',
'arn': 'string',
'schema': {
'name': 'string',
'fields': [
{
'name': 'string',
'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
'isRequired': True|False
},
]
},
'description': 'string',
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
}
}
Response Structure
(dict) --
The response parameters of CreateDataLakeDataset.
dataset (dict) --
The detail of created dataset.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
namespace (string) --
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
name (string) --
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
arn (string) --
The arn of the dataset.
schema (dict) --
The schema of the dataset.
name (string) --
The name of the dataset schema.
fields (list) --
The list of field details of the dataset schema.
(dict) --
The dataset field details.
name (string) --
The dataset field name.
type (string) --
The dataset field type.
isRequired (boolean) --
Indicate if the field is required or not.
description (string) --
The description of the dataset.
createdTime (datetime) --
The creation time of the dataset.
lastModifiedTime (datetime) --
The last modified time of the dataset.
Get a data lake dataset.
See also: AWS API Documentation
Request Syntax
client.get_data_lake_dataset(
instanceId='string',
namespace='string',
name='string'
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
string
[REQUIRED]
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
dict
Response Syntax
{
'dataset': {
'instanceId': 'string',
'namespace': 'string',
'name': 'string',
'arn': 'string',
'schema': {
'name': 'string',
'fields': [
{
'name': 'string',
'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
'isRequired': True|False
},
]
},
'description': 'string',
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
}
}
Response Structure
(dict) --
The response parameters for UpdateDataLakeDataset.
dataset (dict) --
The fetched dataset details.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
namespace (string) --
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
name (string) --
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
arn (string) --
The arn of the dataset.
schema (dict) --
The schema of the dataset.
name (string) --
The name of the dataset schema.
fields (list) --
The list of field details of the dataset schema.
(dict) --
The dataset field details.
name (string) --
The dataset field name.
type (string) --
The dataset field type.
isRequired (boolean) --
Indicate if the field is required or not.
description (string) --
The description of the dataset.
createdTime (datetime) --
The creation time of the dataset.
lastModifiedTime (datetime) --
The last modified time of the dataset.
Lists all the DataIntegrationFlows in a paginated way.
See also: AWS API Documentation
Request Syntax
client.list_data_integration_flows(
instanceId='string',
nextToken='string',
maxResults=123
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
The pagination token to fetch the next page of the DataIntegrationFlows.
integer
Specify the maximum number of DataIntegrationFlows to fetch in one paginated request.
dict
Response Syntax
{
'flows': [
{
'instanceId': 'string',
'name': 'string',
'sources': [
{
'sourceType': 'S3'|'DATASET',
'sourceName': 'string',
's3Source': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetSource': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
],
'transformation': {
'transformationType': 'SQL'|'NONE',
'sqlTransformation': {
'query': 'string'
}
},
'target': {
'targetType': 'S3'|'DATASET',
's3Target': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetTarget': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
The response parameters for ListDataIntegrationFlows.
flows (list) --
The response parameters for ListDataIntegrationFlows.
(dict) --
The DataIntegrationFlow details.
instanceId (string) --
The DataIntegrationFlow instance ID.
name (string) --
The DataIntegrationFlow name.
sources (list) --
The DataIntegrationFlow source configurations.
(dict) --
The DataIntegrationFlow source parameters.
sourceType (string) --
The DataIntegrationFlow source type.
sourceName (string) --
The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
s3Source (dict) --
The S3 DataIntegrationFlow source.
bucketName (string) --
The bucketName of the S3 source objects.
prefix (string) --
The prefix of the S3 source objects.
options (dict) --
The other options of the S3 DataIntegrationFlow source.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetSource (dict) --
The dataset DataIntegrationFlow source.
datasetIdentifier (string) --
The ARN of the dataset.
options (dict) --
The dataset DataIntegrationFlow source options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
transformation (dict) --
The DataIntegrationFlow transformation configurations.
transformationType (string) --
The DataIntegrationFlow transformation type.
sqlTransformation (dict) --
The SQL DataIntegrationFlow transformation configuration.
query (string) --
The transformation SQL query body based on SparkSQL.
target (dict) --
The DataIntegrationFlow target configuration.
targetType (string) --
The DataIntegrationFlow target type.
s3Target (dict) --
The S3 DataIntegrationFlow target.
bucketName (string) --
The bucketName of the S3 target objects.
prefix (string) --
The prefix of the S3 target objects.
options (dict) --
The S3 DataIntegrationFlow target options.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetTarget (dict) --
The dataset DataIntegrationFlow target.
datasetIdentifier (string) --
The dataset ARN.
options (dict) --
The dataset DataIntegrationFlow target options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
createdTime (datetime) --
The DataIntegrationFlow creation timestamp.
lastModifiedTime (datetime) --
The DataIntegrationFlow last modified timestamp.
nextToken (string) --
The pagination token to fetch the next page of the DataIntegrationFlows.
Delete a data lake dataset.
See also: AWS API Documentation
Request Syntax
client.delete_data_lake_dataset(
instanceId='string',
namespace='string',
name='string'
)
string
[REQUIRED]
The AWS Supply Chain instance identifier.
string
[REQUIRED]
The namespace of the dataset. The available values are:
asc: for AWS Supply Chain supported datasets.
default: for datasets with custom user-defined schemas.
string
[REQUIRED]
The name of the dataset. If the namespace is asc, the name must be one of the supported data entities.
dict
Response Syntax
{
'instanceId': 'string',
'namespace': 'string',
'name': 'string'
}
Response Structure
(dict) --
The response parameters of DeleteDataLakeDataset.
instanceId (string) --
The AWS Supply Chain instance identifier.
namespace (string) --
The namespace of deleted dataset.
name (string) --
The name of deleted dataset.
Delete the DataIntegrationFlow.
See also: AWS API Documentation
Request Syntax
client.delete_data_integration_flow(
instanceId='string',
name='string'
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The name of the DataIntegrationFlow to be deleted.
dict
Response Syntax
{
'instanceId': 'string',
'name': 'string'
}
Response Structure
(dict) --
The response parameters for DeleteDataIntegrationFlow.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
name (string) --
The name of the DataIntegrationFlow deleted.
View the DataIntegrationFlow details.
See also: AWS API Documentation
Request Syntax
client.get_data_integration_flow(
instanceId='string',
name='string'
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The name of the DataIntegrationFlow created.
dict
Response Syntax
{
'flow': {
'instanceId': 'string',
'name': 'string',
'sources': [
{
'sourceType': 'S3'|'DATASET',
'sourceName': 'string',
's3Source': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetSource': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
],
'transformation': {
'transformationType': 'SQL'|'NONE',
'sqlTransformation': {
'query': 'string'
}
},
'target': {
'targetType': 'S3'|'DATASET',
's3Target': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetTarget': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
}
}
Response Structure
(dict) --
The response parameters for GetDataIntegrationFlow.
flow (dict) --
The details of the DataIntegrationFlow returned.
instanceId (string) --
The DataIntegrationFlow instance ID.
name (string) --
The DataIntegrationFlow name.
sources (list) --
The DataIntegrationFlow source configurations.
(dict) --
The DataIntegrationFlow source parameters.
sourceType (string) --
The DataIntegrationFlow source type.
sourceName (string) --
The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
s3Source (dict) --
The S3 DataIntegrationFlow source.
bucketName (string) --
The bucketName of the S3 source objects.
prefix (string) --
The prefix of the S3 source objects.
options (dict) --
The other options of the S3 DataIntegrationFlow source.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetSource (dict) --
The dataset DataIntegrationFlow source.
datasetIdentifier (string) --
The ARN of the dataset.
options (dict) --
The dataset DataIntegrationFlow source options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
transformation (dict) --
The DataIntegrationFlow transformation configurations.
transformationType (string) --
The DataIntegrationFlow transformation type.
sqlTransformation (dict) --
The SQL DataIntegrationFlow transformation configuration.
query (string) --
The transformation SQL query body based on SparkSQL.
target (dict) --
The DataIntegrationFlow target configuration.
targetType (string) --
The DataIntegrationFlow target type.
s3Target (dict) --
The S3 DataIntegrationFlow target.
bucketName (string) --
The bucketName of the S3 target objects.
prefix (string) --
The prefix of the S3 target objects.
options (dict) --
The S3 DataIntegrationFlow target options.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetTarget (dict) --
The dataset DataIntegrationFlow target.
datasetIdentifier (string) --
The dataset ARN.
options (dict) --
The dataset DataIntegrationFlow target options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
createdTime (datetime) --
The DataIntegrationFlow creation timestamp.
lastModifiedTime (datetime) --
The DataIntegrationFlow last modified timestamp.
Create DataIntegrationFlow to map one or more different sources to one target using the SQL transformation query.
See also: AWS API Documentation
Request Syntax
client.create_data_integration_flow(
instanceId='string',
name='string',
sources=[
{
'sourceType': 'S3'|'DATASET',
'sourceName': 'string',
's3Source': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetSource': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
],
transformation={
'transformationType': 'SQL'|'NONE',
'sqlTransformation': {
'query': 'string'
}
},
target={
'targetType': 'S3'|'DATASET',
's3Target': {
'bucketName': 'string',
'prefix': 'string',
'options': {
'fileType': 'CSV'|'PARQUET'|'JSON'
}
},
'datasetTarget': {
'datasetIdentifier': 'string',
'options': {
'loadType': 'INCREMENTAL'|'REPLACE',
'dedupeRecords': True|False
}
}
},
tags={
'string': 'string'
}
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
Name of the DataIntegrationFlow.
list
[REQUIRED]
The source configurations for DataIntegrationFlow.
(dict) --
The DataIntegrationFlow source parameters.
sourceType (string) -- [REQUIRED]
The DataIntegrationFlow source type.
sourceName (string) -- [REQUIRED]
The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
s3Source (dict) --
The S3 DataIntegrationFlow source.
bucketName (string) -- [REQUIRED]
The bucketName of the S3 source objects.
prefix (string) -- [REQUIRED]
The prefix of the S3 source objects.
options (dict) --
The other options of the S3 DataIntegrationFlow source.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetSource (dict) --
The dataset DataIntegrationFlow source.
datasetIdentifier (string) -- [REQUIRED]
The ARN of the dataset.
options (dict) --
The dataset DataIntegrationFlow source options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
dict
[REQUIRED]
The transformation configurations for DataIntegrationFlow.
transformationType (string) -- [REQUIRED]
The DataIntegrationFlow transformation type.
sqlTransformation (dict) --
The SQL DataIntegrationFlow transformation configuration.
query (string) -- [REQUIRED]
The transformation SQL query body based on SparkSQL.
dict
[REQUIRED]
The target configurations for DataIntegrationFlow.
targetType (string) -- [REQUIRED]
The DataIntegrationFlow target type.
s3Target (dict) --
The S3 DataIntegrationFlow target.
bucketName (string) -- [REQUIRED]
The bucketName of the S3 target objects.
prefix (string) -- [REQUIRED]
The prefix of the S3 target objects.
options (dict) --
The S3 DataIntegrationFlow target options.
fileType (string) --
The Amazon S3 file type in S3 options.
datasetTarget (dict) --
The dataset DataIntegrationFlow target.
datasetIdentifier (string) -- [REQUIRED]
The dataset ARN.
options (dict) --
The dataset DataIntegrationFlow target options.
loadType (string) --
The dataset data load type in dataset options.
dedupeRecords (boolean) --
The dataset load option to remove duplicates.
dict
The tags of the DataIntegrationFlow to be created
(string) --
(string) --
dict
Response Syntax
{
'instanceId': 'string',
'name': 'string'
}
Response Structure
(dict) --
The response parameters for CreateDataIntegrationFlow.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
name (string) --
The name of the DataIntegrationFlow created.
Delete tags for an Amazon Web Services Supply chain resource.
See also: AWS API Documentation
Request Syntax
client.untag_resource(
resourceArn='string',
tagKeys=[
'string',
]
)
string
[REQUIRED]
The Amazon Web Services Supply chain resource ARN that needs to be untagged.
list
[REQUIRED]
The list of tag keys to be deleted for an Amazon Web Services Supply Chain resource.
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
The response parameters of UntagResource.
List the data lake datasets for a specific instance and name space.
See also: AWS API Documentation
Request Syntax
client.list_data_lake_datasets(
instanceId='string',
namespace='string',
nextToken='string',
maxResults=123
)
string
[REQUIRED]
The Amazon Web Services Supply Chain instance identifier.
string
[REQUIRED]
The namespace of the dataset. The available values are:
asc: for AWS Supply Chain supported datasets.
default: for datasets with custom user-defined schemas.
string
The pagination token to fetch next page of datasets.
integer
The max number of datasets to fetch in this paginated request.
dict
Response Syntax
{
'datasets': [
{
'instanceId': 'string',
'namespace': 'string',
'name': 'string',
'arn': 'string',
'schema': {
'name': 'string',
'fields': [
{
'name': 'string',
'type': 'INT'|'DOUBLE'|'STRING'|'TIMESTAMP',
'isRequired': True|False
},
]
},
'description': 'string',
'createdTime': datetime(2015, 1, 1),
'lastModifiedTime': datetime(2015, 1, 1)
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
The response parameters of ListDataLakeDatasets.
datasets (list) --
The list of fetched dataset details.
(dict) --
The data lake dataset details.
instanceId (string) --
The Amazon Web Services Supply Chain instance identifier.
namespace (string) --
The name space of the dataset. The available values are:
asc - For information on the Amazon Web Services Supply Chain supported datasets see https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
default - For datasets with custom user-defined schemas.
name (string) --
The name of the dataset. For asc name space, the name must be one of the supported data entities under https://docs.aws.amazon.com/aws-supply-chain/latest/userguide/data-model-asc.html.
arn (string) --
The arn of the dataset.
schema (dict) --
The schema of the dataset.
name (string) --
The name of the dataset schema.
fields (list) --
The list of field details of the dataset schema.
(dict) --
The dataset field details.
name (string) --
The dataset field name.
type (string) --
The dataset field type.
isRequired (boolean) --
Indicate if the field is required or not.
description (string) --
The description of the dataset.
createdTime (datetime) --
The creation time of the dataset.
lastModifiedTime (datetime) --
The last modified time of the dataset.
nextToken (string) --
The pagination token to fetch next page of datasets.