2020/11/11 - AWS Glue DataBrew - 38 new api methods
Changes This is the initial SDK release for AWS Glue DataBrew. DataBrew is a visual data preparation tool that enables users to clean and normalize data without writing any code.
Returns the definition of a specific AWS Glue DataBrew recipe that is in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.describe_recipe(
Name='string',
RecipeVersion='string'
)
string
[REQUIRED]
The name of the recipe to be described.
string
The recipe version identifier. If this parameter isn't specified, then the latest published version is returned.
dict
Response Syntax
{
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'ProjectName': 'string',
'PublishedBy': 'string',
'PublishedDate': datetime(2015, 1, 1),
'Description': 'string',
'Name': 'string',
'Steps': [
{
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
],
'Tags': {
'string': 'string'
},
'ResourceArn': 'string',
'RecipeVersion': 'string'
}
Response Structure
(dict) --
CreatedBy (string) --
The identifier (user name) of the user who created the recipe.
CreateDate (datetime) --
The date and time that the recipe was created.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the recipe.
LastModifiedDate (datetime) --
The date and time that the recipe was last modified.
ProjectName (string) --
The name of the project associated with this recipe.
PublishedBy (string) --
The identifier (user name) of the user who last published the recipe.
PublishedDate (datetime) --
The date and time when the recipe was last published.
Description (string) --
The description of the recipe.
Name (string) --
The name of the recipe.
Steps (list) --
One or more steps to be performed by the recipe. Each step consists of an action, and the conditions under which the action should succeed.
(dict) --
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) --
The particular action to be performed in the recipe step.
Operation (string) --
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) --
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) --
A column to apply this condition to, within an AWS Glue DataBrew dataset.
Tags (dict) --
Metadata tags associated with this project.
(string) --
(string) --
ResourceArn (string) --
The ARN of the recipe.
RecipeVersion (string) --
The recipe version identifier.
Creates a new job for an existing AWS Glue DataBrew recipe in the current AWS account. You can create a standalone job using either a project, or a combination of a recipe and a dataset.
See also: AWS API Documentation
Request Syntax
client.create_recipe_job(
DatasetName='string',
EncryptionKeyArn='string',
EncryptionMode='SSE-KMS'|'SSE-S3',
Name='string',
LogSubscription='ENABLE'|'DISABLE',
MaxCapacity=123,
MaxRetries=123,
Outputs=[
{
'CompressionFormat': 'GZIP'|'LZ4'|'SNAPPY'|'BZIP2'|'DEFLATE'|'LZO'|'BROTLI'|'ZSTD'|'ZLIB',
'Format': 'CSV'|'JSON'|'PARQUET'|'GLUEPARQUET'|'AVRO'|'ORC'|'XML',
'PartitionColumns': [
'string',
],
'Location': {
'Bucket': 'string',
'Key': 'string'
},
'Overwrite': True|False
},
],
ProjectName='string',
RecipeReference={
'Name': 'string',
'RecipeVersion': 'string'
},
RoleArn='string',
Tags={
'string': 'string'
},
Timeout=123
)
string
The name of the dataset that this job processes.
string
The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
string
The encryption mode for the job, which can be one of the following:
SSE-KMS - Server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
string
[REQUIRED]
A unique name for the job.
string
A value that enables or disables Amazon CloudWatch logging for the current AWS account. If logging is enabled, CloudWatch writes one log stream for each job run.
integer
The maximum number of nodes that DataBrew can consume when the job processes data.
integer
The maximum number of times to retry the job after a job run fails.
list
[REQUIRED]
One or more artifacts that represent the output from running the job.
(dict) --
Represents individual output from a particular job run.
CompressionFormat (string) --
The compression algorithm used to compress the output text of the job.
Format (string) --
The data format of the output of the job.
PartitionColumns (list) --
The names of one or more partition columns for the output of the job.
(string) --
Location (dict) -- [REQUIRED]
The location in Amazon S3 where the job writes its output.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
Overwrite (boolean) --
A value that, if true, means that any data in the location specified for output is overwritten with new output.
string
Either the name of an existing project, or a combination of a recipe and a dataset to associate with the recipe.
dict
Represents all of the attributes of an AWS Glue DataBrew recipe.
Name (string) -- [REQUIRED]
The name of the recipe.
RecipeVersion (string) --
The identifier for the version for the recipe.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role to be assumed for this request.
dict
Metadata tags to apply to this job dataset.
(string) --
(string) --
integer
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the job that you created.
Creates a new AWS Glue DataBrew recipe for the current AWS account.
See also: AWS API Documentation
Request Syntax
client.create_recipe(
Description='string',
Name='string',
Steps=[
{
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
],
Tags={
'string': 'string'
}
)
string
A description for the recipe.
string
[REQUIRED]
A unique name for the recipe.
list
[REQUIRED]
An array containing the steps to be performed by the recipe. Each recipe step consists of one recipe action and (optionally) an array of condition expressions.
(dict) --
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) -- [REQUIRED]
The particular action to be performed in the recipe step.
Operation (string) -- [REQUIRED]
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) -- [REQUIRED]
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) -- [REQUIRED]
A column to apply this condition to, within an AWS Glue DataBrew dataset.
dict
Metadata tags to apply to this recipe.
(string) --
(string) --
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the recipe that you created.
Returns the definition of a specific AWS Glue DataBrew dataset that is in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.describe_dataset(
Name='string'
)
string
[REQUIRED]
The name of the dataset to be described.
dict
Response Syntax
{
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'Name': 'string',
'FormatOptions': {
'Json': {
'MultiLine': True|False
},
'Excel': {
'SheetNames': [
'string',
],
'SheetIndexes': [
123,
]
}
},
'Input': {
'S3InputDefinition': {
'Bucket': 'string',
'Key': 'string'
},
'DataCatalogInputDefinition': {
'CatalogId': 'string',
'DatabaseName': 'string',
'TableName': 'string',
'TempDirectory': {
'Bucket': 'string',
'Key': 'string'
}
}
},
'LastModifiedDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'Source': 'S3'|'DATA-CATALOG',
'Tags': {
'string': 'string'
},
'ResourceArn': 'string'
}
Response Structure
(dict) --
CreatedBy (string) --
The identifier (user name) of the user who created the dataset.
CreateDate (datetime) --
The date and time that the dataset was created.
Name (string) --
The name of the dataset.
FormatOptions (dict) --
Options that define how Microsoft Excel input is to be interpreted by DataBrew.
Json (dict) --
Options that define how JSON input is to be interpreted by DataBrew.
MultiLine (boolean) --
A value that specifies whether JSON input contains embedded new line characters.
Excel (dict) --
Options that define how Excel input is to be interpreted by DataBrew.
SheetNames (list) --
Specifies one or more named sheets in the Excel file, which will be included in the dataset.
(string) --
SheetIndexes (list) --
Specifies one or more sheet numbers in the Excel file, which will be included in the dataset.
(integer) --
Input (dict) --
Information on how AWS Glue DataBrew can find data, in either the AWS Glue Data Catalog or Amazon S3.
S3InputDefinition (dict) --
The Amazon S3 location where the data is stored.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
DataCatalogInputDefinition (dict) --
The AWS Glue Data Catalog parameters for the data.
CatalogId (string) --
The unique identifier of the AWS account that holds the Data Catalog that stores the data.
DatabaseName (string) --
The name of a database in the Data Catalog.
TableName (string) --
The name of a database table in the Data Catalog. This table corresponds to a DataBrew dataset.
TempDirectory (dict) --
An Amazon location that AWS Glue Data Catalog can use as a temporary directory.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
LastModifiedDate (datetime) --
The date and time that the dataset was last modified.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the dataset.
Source (string) --
The location of the data for this dataset, Amazon S3 or the AWS Glue Data Catalog.
Tags (dict) --
Metadata tags associated with this dataset.
(string) --
(string) --
ResourceArn (string) --
The Amazon Resource Name (ARN) of the dataset.
Runs an AWS Glue DataBrew job that exists in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.start_job_run(
Name='string'
)
string
[REQUIRED]
The name of the job to be run.
dict
Response Syntax
{
'RunId': 'string'
}
Response Structure
(dict) --
RunId (string) --
A system-generated identifier for this particular job run.
Lists the AWS Glue DataBrew jobs in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_jobs(
DatasetName='string',
MaxResults=123,
NextToken='string',
ProjectName='string'
)
string
The name of a dataset. Using this parameter indicates to return only those jobs that act on the specified dataset.
integer
The maximum number of results to return in this request.
string
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
string
The name of a project. Using this parameter indicates to return only those jobs that are associated with the specified project.
dict
Response Syntax
{
'Jobs': [
{
'AccountId': 'string',
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'DatasetName': 'string',
'EncryptionKeyArn': 'string',
'EncryptionMode': 'SSE-KMS'|'SSE-S3',
'Name': 'string',
'Type': 'PROFILE'|'RECIPE',
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'LogSubscription': 'ENABLE'|'DISABLE',
'MaxCapacity': 123,
'MaxRetries': 123,
'Outputs': [
{
'CompressionFormat': 'GZIP'|'LZ4'|'SNAPPY'|'BZIP2'|'DEFLATE'|'LZO'|'BROTLI'|'ZSTD'|'ZLIB',
'Format': 'CSV'|'JSON'|'PARQUET'|'GLUEPARQUET'|'AVRO'|'ORC'|'XML',
'PartitionColumns': [
'string',
],
'Location': {
'Bucket': 'string',
'Key': 'string'
},
'Overwrite': True|False
},
],
'ProjectName': 'string',
'RecipeReference': {
'Name': 'string',
'RecipeVersion': 'string'
},
'ResourceArn': 'string',
'RoleArn': 'string',
'Timeout': 123,
'Tags': {
'string': 'string'
}
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Jobs (list) --
A list of jobs that are defined in the current AWS account.
(dict) --
Represents all of the attributes of an AWS Glue DataBrew job.
AccountId (string) --
The ID of the AWS account that owns the job.
CreatedBy (string) --
The identifier (the user name) of the user who created the job.
CreateDate (datetime) --
The date and time that the job was created.
DatasetName (string) --
A dataset that the job is to process.
EncryptionKeyArn (string) --
The Amazon Resource Name (ARN) of an encryption key that is used to protect a job.
EncryptionMode (string) --
The encryption mode for the job, which can be one of the following:
SSE-KMS - Server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
Name (string) --
The unique name of the job.
Type (string) --
The job type of the job, which must be one of the following:
PROFILE - A job to analyze a dataset, to determine its size, data types, data distribution, and more.
RECIPE - A job to apply one or more transformations to a dataset.
LastModifiedBy (string) --
The identifier (the user name) of the user who last modified the job.
LastModifiedDate (datetime) --
The modification date and time of the job.
LogSubscription (string) --
The current status of Amazon CloudWatch logging for the job.
MaxCapacity (integer) --
The maximum number of nodes that can be consumed when the job processes data.
MaxRetries (integer) --
The maximum number of times to retry the job after a job run fails.
Outputs (list) --
One or more artifacts that represent output from running the job.
(dict) --
Represents individual output from a particular job run.
CompressionFormat (string) --
The compression algorithm used to compress the output text of the job.
Format (string) --
The data format of the output of the job.
PartitionColumns (list) --
The names of one or more partition columns for the output of the job.
(string) --
Location (dict) --
The location in Amazon S3 where the job writes its output.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
Overwrite (boolean) --
A value that, if true, means that any data in the location specified for output is overwritten with new output.
ProjectName (string) --
The name of the project that the job is associated with.
RecipeReference (dict) --
A set of steps that the job runs.
Name (string) --
The name of the recipe.
RecipeVersion (string) --
The identifier for the version for the recipe.
ResourceArn (string) --
The unique Amazon Resource Name (ARN) for the job.
RoleArn (string) --
The Amazon Resource Name (ARN) of the role that will be assumed for this job.
Timeout (integer) --
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
Tags (dict) --
Metadata tags that have been applied to the job.
(string) --
(string) --
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To obtain the next set of pages, pass in the NextToken from the response object of the previous page call.
Performs a recipe step within an interactive AWS Glue DataBrew session that's currently open.
See also: AWS API Documentation
Request Syntax
client.send_project_session_action(
Preview=True|False,
Name='string',
RecipeStep={
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
StepIndex=123,
ClientSessionId='string',
ViewFrame={
'StartColumnIndex': 123,
'ColumnRange': 123,
'HiddenColumns': [
'string',
]
}
)
boolean
Returns the result of the recipe step, without applying it. The result isn't added to the view frame stack.
string
[REQUIRED]
The name of the project to apply the action to.
dict
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) -- [REQUIRED]
The particular action to be performed in the recipe step.
Operation (string) -- [REQUIRED]
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) -- [REQUIRED]
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) -- [REQUIRED]
A column to apply this condition to, within an AWS Glue DataBrew dataset.
integer
The index from which to preview a step. This index is used to preview the result of steps that have already been applied, so that the resulting view frame is from earlier in the view frame stack.
string
A unique identifier for an interactive session that's currently open and ready for work. The action will be performed on this session.
dict
Represents the data being being transformed during an AWS Glue DataBrew project session.
StartColumnIndex (integer) -- [REQUIRED]
The starting index for the range of columns to return in the view frame.
ColumnRange (integer) --
The number of columns to include in the view frame, beginning with the StartColumnIndex value and ignoring any columns in the HiddenColumns list.
HiddenColumns (list) --
A list of columns to hide in the view frame.
(string) --
dict
Response Syntax
{
'Result': 'string',
'Name': 'string',
'ActionId': 123
}
Response Structure
(dict) --
Result (string) --
A message indicating the result of performing the action.
Name (string) --
The name of the project that was affected by the action.
ActionId (integer) --
A unique identifier for the action that was performed.
Removes metadata tags from an AWS Glue DataBrew resource.
See also: AWS API Documentation
Request Syntax
client.untag_resource(
ResourceArn='string',
TagKeys=[
'string',
]
)
string
[REQUIRED]
An DataBrew resource from which you want to remove a tag or tags. The value for this parameter is an Amazon Resource Name (ARN).
list
[REQUIRED]
The tag keys (names) of one or more tags to be removed.
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
Creates a new AWS Glue DataBrew dataset for this AWS account.
See also: AWS API Documentation
Request Syntax
client.create_dataset(
Name='string',
FormatOptions={
'Json': {
'MultiLine': True|False
},
'Excel': {
'SheetNames': [
'string',
],
'SheetIndexes': [
123,
]
}
},
Input={
'S3InputDefinition': {
'Bucket': 'string',
'Key': 'string'
},
'DataCatalogInputDefinition': {
'CatalogId': 'string',
'DatabaseName': 'string',
'TableName': 'string',
'TempDirectory': {
'Bucket': 'string',
'Key': 'string'
}
}
},
Tags={
'string': 'string'
}
)
string
[REQUIRED]
The name of the dataset to be created.
dict
Options that define how Microsoft Excel input is to be interpreted by DataBrew.
Json (dict) --
Options that define how JSON input is to be interpreted by DataBrew.
MultiLine (boolean) --
A value that specifies whether JSON input contains embedded new line characters.
Excel (dict) --
Options that define how Excel input is to be interpreted by DataBrew.
SheetNames (list) --
Specifies one or more named sheets in the Excel file, which will be included in the dataset.
(string) --
SheetIndexes (list) --
Specifies one or more sheet numbers in the Excel file, which will be included in the dataset.
(integer) --
dict
[REQUIRED]
Information on how AWS Glue DataBrew can find data, in either the AWS Glue Data Catalog or Amazon S3.
S3InputDefinition (dict) --
The Amazon S3 location where the data is stored.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
DataCatalogInputDefinition (dict) --
The AWS Glue Data Catalog parameters for the data.
CatalogId (string) --
The unique identifier of the AWS account that holds the Data Catalog that stores the data.
DatabaseName (string) -- [REQUIRED]
The name of a database in the Data Catalog.
TableName (string) -- [REQUIRED]
The name of a database table in the Data Catalog. This table corresponds to a DataBrew dataset.
TempDirectory (dict) --
An Amazon location that AWS Glue Data Catalog can use as a temporary directory.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
dict
Metadata tags to apply to this dataset.
(string) --
(string) --
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the dataset that you created.
Deletes the specified AWS Glue DataBrew schedule from the current AWS account.
See also: AWS API Documentation
Request Syntax
client.delete_schedule(
Name='string'
)
string
[REQUIRED]
The name of the schedule to be deleted.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the schedule that was deleted.
Lists all of the AWS Glue DataBrew datasets for the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_datasets(
MaxResults=123,
NextToken='string'
)
integer
The maximum number of results to return in this request.
string
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
dict
Response Syntax
{
'Datasets': [
{
'AccountId': 'string',
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'Name': 'string',
'FormatOptions': {
'Json': {
'MultiLine': True|False
},
'Excel': {
'SheetNames': [
'string',
],
'SheetIndexes': [
123,
]
}
},
'Input': {
'S3InputDefinition': {
'Bucket': 'string',
'Key': 'string'
},
'DataCatalogInputDefinition': {
'CatalogId': 'string',
'DatabaseName': 'string',
'TableName': 'string',
'TempDirectory': {
'Bucket': 'string',
'Key': 'string'
}
}
},
'LastModifiedDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'Source': 'S3'|'DATA-CATALOG',
'Tags': {
'string': 'string'
},
'ResourceArn': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Datasets (list) --
A list of datasets that are defined in the current AWS account.
(dict) --
Represents a dataset that can be processed by AWS Glue DataBrew.
AccountId (string) --
The ID of the AWS account that owns the dataset.
CreatedBy (string) --
The identifier (the user name) of the user who created the dataset.
CreateDate (datetime) --
The date and time that the dataset was created.
Name (string) --
The unique name of the dataset.
FormatOptions (dict) --
Options that define how DataBrew interprets the data in the dataset.
Json (dict) --
Options that define how JSON input is to be interpreted by DataBrew.
MultiLine (boolean) --
A value that specifies whether JSON input contains embedded new line characters.
Excel (dict) --
Options that define how Excel input is to be interpreted by DataBrew.
SheetNames (list) --
Specifies one or more named sheets in the Excel file, which will be included in the dataset.
(string) --
SheetIndexes (list) --
Specifies one or more sheet numbers in the Excel file, which will be included in the dataset.
(integer) --
Input (dict) --
Information on how DataBrew can find the dataset, in either the AWS Glue Data Catalog or Amazon S3.
S3InputDefinition (dict) --
The Amazon S3 location where the data is stored.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
DataCatalogInputDefinition (dict) --
The AWS Glue Data Catalog parameters for the data.
CatalogId (string) --
The unique identifier of the AWS account that holds the Data Catalog that stores the data.
DatabaseName (string) --
The name of a database in the Data Catalog.
TableName (string) --
The name of a database table in the Data Catalog. This table corresponds to a DataBrew dataset.
TempDirectory (dict) --
An Amazon location that AWS Glue Data Catalog can use as a temporary directory.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
LastModifiedDate (datetime) --
The last modification date and time of the dataset.
LastModifiedBy (string) --
The identifier (the user name) of the user who last modified the dataset.
Source (string) --
The location of the data for the dataset, either Amazon S3 or the AWS Glue Data Catalog.
Tags (dict) --
Metadata tags that have been applied to the dataset.
(string) --
(string) --
ResourceArn (string) --
The unique Amazon Resource Name (ARN) for the dataset.
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To obtain the next set of pages, pass in the NextToken from the response object of the previous page call.
Modifies the definition of the latest working version of an AWS Glue DataBrew recipe in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_recipe(
Description='string',
Name='string',
Steps=[
{
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
]
)
string
A description of the recipe.
string
[REQUIRED]
The name of the recipe to be updated.
list
One or more steps to be performed by the recipe. Each step consists of an action, and the conditions under which the action should succeed.
(dict) --
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) -- [REQUIRED]
The particular action to be performed in the recipe step.
Operation (string) -- [REQUIRED]
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) -- [REQUIRED]
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) -- [REQUIRED]
A column to apply this condition to, within an AWS Glue DataBrew dataset.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the recipe that was updated.
Deletes one or more versions of a recipe at a time.
See also: AWS API Documentation
Request Syntax
client.batch_delete_recipe_version(
Name='string',
RecipeVersions=[
'string',
]
)
string
[REQUIRED]
The name of the recipe to be modified.
list
[REQUIRED]
An array of version identifiers to be deleted.
(string) --
dict
Response Syntax
{
'Name': 'string',
'Errors': [
{
'ErrorCode': 'string',
'ErrorMessage': 'string',
'RecipeVersion': 'string'
},
]
}
Response Structure
(dict) --
Name (string) --
The name of the recipe that was modified.
Errors (list) --
Errors, if any, that were encountered when deleting the recipe versions.
(dict) --
Represents any errors encountered when attempting to delete multiple recipe versions.
ErrorCode (string) --
The HTTP status code for the error.
ErrorMessage (string) --
The text of the error message.
RecipeVersion (string) --
The identifier for the recipe version associated with this error.
Modifies the definition of an existing AWS Glue DataBrew dataset in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_dataset(
Name='string',
FormatOptions={
'Json': {
'MultiLine': True|False
},
'Excel': {
'SheetNames': [
'string',
],
'SheetIndexes': [
123,
]
}
},
Input={
'S3InputDefinition': {
'Bucket': 'string',
'Key': 'string'
},
'DataCatalogInputDefinition': {
'CatalogId': 'string',
'DatabaseName': 'string',
'TableName': 'string',
'TempDirectory': {
'Bucket': 'string',
'Key': 'string'
}
}
}
)
string
[REQUIRED]
The name of the dataset to be updated.
dict
Options that define how Microsoft Excel input is to be interpreted by DataBrew.
Json (dict) --
Options that define how JSON input is to be interpreted by DataBrew.
MultiLine (boolean) --
A value that specifies whether JSON input contains embedded new line characters.
Excel (dict) --
Options that define how Excel input is to be interpreted by DataBrew.
SheetNames (list) --
Specifies one or more named sheets in the Excel file, which will be included in the dataset.
(string) --
SheetIndexes (list) --
Specifies one or more sheet numbers in the Excel file, which will be included in the dataset.
(integer) --
dict
[REQUIRED]
Information on how AWS Glue DataBrew can find data, in either the AWS Glue Data Catalog or Amazon S3.
S3InputDefinition (dict) --
The Amazon S3 location where the data is stored.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
DataCatalogInputDefinition (dict) --
The AWS Glue Data Catalog parameters for the data.
CatalogId (string) --
The unique identifier of the AWS account that holds the Data Catalog that stores the data.
DatabaseName (string) -- [REQUIRED]
The name of a database in the Data Catalog.
TableName (string) -- [REQUIRED]
The name of a database table in the Data Catalog. This table corresponds to a DataBrew dataset.
TempDirectory (dict) --
An Amazon location that AWS Glue Data Catalog can use as a temporary directory.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the dataset that you updated.
Modifies the definition of an existing AWS Glue DataBrew project in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_project(
Sample={
'Size': 123,
'Type': 'FIRST_N'|'LAST_N'|'RANDOM'
},
RoleArn='string',
Name='string'
)
dict
Represents the sample size and sampling type for AWS Glue DataBrew to use for interactive data analysis.
Size (integer) --
The number of rows in the sample.
Type (string) -- [REQUIRED]
The way in which DataBrew obtains rows from a dataset.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the IAM role to be assumed for this request.
string
[REQUIRED]
The name of the project to be updated.
dict
Response Syntax
{
'LastModifiedDate': datetime(2015, 1, 1),
'Name': 'string'
}
Response Structure
(dict) --
LastModifiedDate (datetime) --
The date and time that the project was last modified.
Name (string) --
The name of the project that you updated.
Lists all the tags for an AWS Glue DataBrew resource.
See also: AWS API Documentation
Request Syntax
client.list_tags_for_resource(
ResourceArn='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) string that uniquely identifies the DataBrew resource.
dict
Response Syntax
{
'Tags': {
'string': 'string'
}
}
Response Structure
(dict) --
Tags (dict) --
A list of tags associated with the DataBrew resource.
(string) --
(string) --
Deletes the specified AWS Glue DataBrew job from the current AWS account. The job can be for a recipe or for a profile.
See also: AWS API Documentation
Request Syntax
client.delete_job(
Name='string'
)
string
[REQUIRED]
The name of the job to be deleted.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the job that you deleted.
Lists all of the DataBrew projects in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_projects(
NextToken='string',
MaxResults=123
)
string
A pagination token that can be used in a subsequent request.
integer
The maximum number of results to return in this request.
dict
Response Syntax
{
'Projects': [
{
'AccountId': 'string',
'CreateDate': datetime(2015, 1, 1),
'CreatedBy': 'string',
'DatasetName': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'Name': 'string',
'RecipeName': 'string',
'ResourceArn': 'string',
'Sample': {
'Size': 123,
'Type': 'FIRST_N'|'LAST_N'|'RANDOM'
},
'Tags': {
'string': 'string'
},
'RoleArn': 'string',
'OpenedBy': 'string',
'OpenDate': datetime(2015, 1, 1)
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Projects (list) --
A list of projects that are defined in the current AWS account.
(dict) --
Represents all of the attributes of an AWS Glue DataBrew project.
AccountId (string) --
The ID of the AWS account that owns the project.
CreateDate (datetime) --
The date and time that the project was created.
CreatedBy (string) --
The identifier (the user name) of the user who crated the project.
DatasetName (string) --
The dataset that the project is to act upon.
LastModifiedDate (datetime) --
The last modification date and time for the project.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the project.
Name (string) --
The unique name of a project.
RecipeName (string) --
The name of a recipe that will be developed during a project session.
ResourceArn (string) --
The Amazon Resource Name (ARN) for the project.
Sample (dict) --
The sample size and sampling type to apply to the data. If this parameter isn't specified, then the sample will consiste of the first 500 rows from the dataset.
Size (integer) --
The number of rows in the sample.
Type (string) --
The way in which DataBrew obtains rows from a dataset.
Tags (dict) --
Metadata tags that have been applied to the project.
(string) --
(string) --
RoleArn (string) --
The Amazon Resource Name (ARN) of the role that will be assumed for this project.
OpenedBy (string) --
The identifier (the user name) of the user that opened the project for use.
OpenDate (datetime) --
The date and time when the project was opened.
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
Modifies the definition of an existing AWS Glue DataBrew recipe job in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_recipe_job(
EncryptionKeyArn='string',
EncryptionMode='SSE-KMS'|'SSE-S3',
Name='string',
LogSubscription='ENABLE'|'DISABLE',
MaxCapacity=123,
MaxRetries=123,
Outputs=[
{
'CompressionFormat': 'GZIP'|'LZ4'|'SNAPPY'|'BZIP2'|'DEFLATE'|'LZO'|'BROTLI'|'ZSTD'|'ZLIB',
'Format': 'CSV'|'JSON'|'PARQUET'|'GLUEPARQUET'|'AVRO'|'ORC'|'XML',
'PartitionColumns': [
'string',
],
'Location': {
'Bucket': 'string',
'Key': 'string'
},
'Overwrite': True|False
},
],
RoleArn='string',
Timeout=123
)
string
The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
string
The encryption mode for the job, which can be one of the following:
SSE-KMS - Server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
string
[REQUIRED]
The name of the job to update.
string
A value that enables or disables Amazon CloudWatch logging for the current AWS account. If logging is enabled, CloudWatch writes one log stream for each job run.
integer
The maximum number of nodes that DataBrew can consume when the job processes data.
integer
The maximum number of times to retry the job after a job run fails.
list
[REQUIRED]
One or more artifacts that represent the output from running the job.
(dict) --
Represents individual output from a particular job run.
CompressionFormat (string) --
The compression algorithm used to compress the output text of the job.
Format (string) --
The data format of the output of the job.
PartitionColumns (list) --
The names of one or more partition columns for the output of the job.
(string) --
Location (dict) -- [REQUIRED]
The location in Amazon S3 where the job writes its output.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
Overwrite (boolean) --
A value that, if true, means that any data in the location specified for output is overwritten with new output.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role to be assumed for this request.
integer
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the job that you updated.
Returns the definition of a specific AWS Glue DataBrew project that is in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.describe_project(
Name='string'
)
string
[REQUIRED]
The name of the project to be described.
dict
Response Syntax
{
'CreateDate': datetime(2015, 1, 1),
'CreatedBy': 'string',
'DatasetName': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'Name': 'string',
'RecipeName': 'string',
'ResourceArn': 'string',
'Sample': {
'Size': 123,
'Type': 'FIRST_N'|'LAST_N'|'RANDOM'
},
'RoleArn': 'string',
'Tags': {
'string': 'string'
},
'SessionStatus': 'ASSIGNED'|'FAILED'|'INITIALIZING'|'PROVISIONING'|'READY'|'RECYCLING'|'ROTATING'|'TERMINATED'|'TERMINATING'|'UPDATING',
'OpenedBy': 'string',
'OpenDate': datetime(2015, 1, 1)
}
Response Structure
(dict) --
CreateDate (datetime) --
The date and time that the project was created.
CreatedBy (string) --
The identifier (user name) of the user who created the project.
DatasetName (string) --
The dataset associated with the project.
LastModifiedDate (datetime) --
The date and time that the project was last modified.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the project.
Name (string) --
The name of the project.
RecipeName (string) --
The recipe associated with this job.
ResourceArn (string) --
The Amazon Resource Name (ARN) of the project.
Sample (dict) --
Represents the sample size and sampling type for AWS Glue DataBrew to use for interactive data analysis.
Size (integer) --
The number of rows in the sample.
Type (string) --
The way in which DataBrew obtains rows from a dataset.
RoleArn (string) --
The ARN of the AWS Identity and Access Management (IAM) role that was assumed for this request.
Tags (dict) --
Metadata tags associated with this project.
(string) --
(string) --
SessionStatus (string) --
Describes the current state of the session:
PROVISIONING - allocating resources for the session.
INITIALIZING - getting the session ready for first use.
ASSIGNED - the session is ready for use.
OpenedBy (string) --
The identifier (user name) of the user that opened the project for use.
OpenDate (datetime) --
The date and time when the project was opened.
Creates a new job to profile an AWS Glue DataBrew dataset that exists in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.create_profile_job(
DatasetName='string',
EncryptionKeyArn='string',
EncryptionMode='SSE-KMS'|'SSE-S3',
Name='string',
LogSubscription='ENABLE'|'DISABLE',
MaxCapacity=123,
MaxRetries=123,
OutputLocation={
'Bucket': 'string',
'Key': 'string'
},
RoleArn='string',
Tags={
'string': 'string'
},
Timeout=123
)
string
[REQUIRED]
The name of the dataset that this job is to act upon.
string
The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
string
The encryption mode for the job, which can be one of the following:
SSE-KMS - para> SSE-KMS - server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
string
[REQUIRED]
The name of the job to be created.
string
A value that enables or disables Amazon CloudWatch logging for the current AWS account. If logging is enabled, CloudWatch writes one log stream for each job run.
integer
The maximum number of nodes that DataBrew can use when the job processes data.
integer
The maximum number of times to retry the job after a job run fails.
dict
[REQUIRED]
An Amazon S3 location (bucket name an object key) where DataBrew can read input data, or write output from a job.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role to be assumed for this request.
dict
Metadata tags to apply to this job.
(string) --
(string) --
integer
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the job that was created.
Deletes a single version of an AWS Glue DataBrew recipe.
See also: AWS API Documentation
Request Syntax
client.delete_recipe_version(
Name='string',
RecipeVersion='string'
)
string
[REQUIRED]
The name of the recipe to be deleted.
string
[REQUIRED]
The version of the recipe to be deleted.
dict
Response Syntax
{
'Name': 'string',
'RecipeVersion': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the recipe that was deleted.
RecipeVersion (string) --
The version of the recipe that was deleted.
Modifies the definition of an existing AWS Glue DataBrew schedule in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_schedule(
JobNames=[
'string',
],
CronExpression='string',
Name='string'
)
list
The name or names of one or more jobs to be run for this schedule.
(string) --
string
[REQUIRED]
The date or dates and time or times, in cron format, when the jobs are to be run.
string
[REQUIRED]
The name of the schedule to update.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the schedule that was updated.
Deletes an existing AWS Glue DataBrew project from the current AWS account.
See also: AWS API Documentation
Request Syntax
client.delete_project(
Name='string'
)
string
[REQUIRED]
The name of the project to be deleted.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the project that you deleted.
Modifies the definition of an existing AWS Glue DataBrew job in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.update_profile_job(
EncryptionKeyArn='string',
EncryptionMode='SSE-KMS'|'SSE-S3',
Name='string',
LogSubscription='ENABLE'|'DISABLE',
MaxCapacity=123,
MaxRetries=123,
OutputLocation={
'Bucket': 'string',
'Key': 'string'
},
RoleArn='string',
Timeout=123
)
string
The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
string
The encryption mode for the job, which can be one of the following:
SSE-KMS - Server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
string
[REQUIRED]
The name of the job to be updated.
string
A value that enables or disables Amazon CloudWatch logging for the current AWS account. If logging is enabled, CloudWatch writes one log stream for each job run.
integer
The maximum number of nodes that DataBrew can use when the job processes data.
integer
The maximum number of times to retry the job after a job run fails.
dict
[REQUIRED]
An Amazon S3 location (bucket name an object key) where DataBrew can read input data, or write output from a job.
Bucket (string) -- [REQUIRED]
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role to be assumed for this request.
integer
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the job that was updated.
Adds metadata tags to an AWS Glue DataBrew resource, such as a dataset, job, project, or recipe.
See also: AWS API Documentation
Request Syntax
client.tag_resource(
ResourceArn='string',
Tags={
'string': 'string'
}
)
string
[REQUIRED]
The DataBrew resource to which tags should be added. The value for this parameter is an Amazon Resource Name (ARN). For DataBrew, you can tag a dataset, a job, a project, or a recipe.
dict
[REQUIRED]
One or more tags to be assigned to the resource.
(string) --
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
Publishes a new major version of an AWS Glue DataBrew recipe that exists in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.publish_recipe(
Description='string',
Name='string'
)
string
A description of the recipe to be published, for this version of the recipe.
string
[REQUIRED]
The name of the recipe to be published.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the recipe that you published.
Lists the AWS Glue DataBrew schedules in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_schedules(
JobName='string',
MaxResults=123,
NextToken='string'
)
string
The name of the job that these schedules apply to.
integer
The maximum number of results to return in this request.
string
A pagination token that can be used in a subsequent request.
dict
Response Syntax
{
'Schedules': [
{
'AccountId': 'string',
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'JobNames': [
'string',
],
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'ResourceArn': 'string',
'CronExpression': 'string',
'Tags': {
'string': 'string'
},
'Name': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Schedules (list) --
A list of schedules in the current AWS account.
(dict) --
Represents one or more dates and times when a job is to run.
AccountId (string) --
The ID of the AWS account that owns the schedule.
CreatedBy (string) --
The identifier (the user name) of the user who created the schedule.
CreateDate (datetime) --
The date and time that the schedule was created.
JobNames (list) --
A list of jobs to be run, according to the schedule.
(string) --
LastModifiedBy (string) --
The identifier (the user name) of the user who last modified the schedule.
LastModifiedDate (datetime) --
The date and time when the schedule was last modified.
ResourceArn (string) --
The Amazon Resource Name (ARN) of the schedule.
CronExpression (string) --
The date(s) and time(s), in cron format, when the job will run.
Tags (dict) --
Metadata tags that have been applied to the schedule.
(string) --
(string) --
Name (string) --
The name of the schedule.
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
Creates a new AWS Glue DataBrew project in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.create_project(
DatasetName='string',
Name='string',
RecipeName='string',
Sample={
'Size': 123,
'Type': 'FIRST_N'|'LAST_N'|'RANDOM'
},
RoleArn='string',
Tags={
'string': 'string'
}
)
string
[REQUIRED]
The name of the dataset to associate this project with.
string
[REQUIRED]
A unique name for the new project.
string
[REQUIRED]
The name of an existing recipe to associate with the project.
dict
Represents the sample size and sampling type for AWS Glue DataBrew to use for interactive data analysis.
Size (integer) --
The number of rows in the sample.
Type (string) -- [REQUIRED]
The way in which DataBrew obtains rows from a dataset.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the AWS Identity and Access Management (IAM) role to be assumed for this request.
dict
Metadata tags to apply to this project.
(string) --
(string) --
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the project that you created.
Deletes a dataset from AWS Glue DataBrew.
See also: AWS API Documentation
Request Syntax
client.delete_dataset(
Name='string'
)
string
[REQUIRED]
The name of the dataset to be deleted.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the dataset that you deleted.
Creates a new schedule for one or more AWS Glue DataBrew jobs. Jobs can be run at a specific date and time, or at regular intervals.
See also: AWS API Documentation
Request Syntax
client.create_schedule(
JobNames=[
'string',
],
CronExpression='string',
Tags={
'string': 'string'
},
Name='string'
)
list
The name or names of one or more jobs to be run.
(string) --
string
[REQUIRED]
The date or dates and time or times, in cron format, when the jobs are to be run.
dict
Metadata tags to apply to this schedule.
(string) --
(string) --
string
[REQUIRED]
A unique name for the schedule.
dict
Response Syntax
{
'Name': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the schedule that was created.
Lists all of the versions of a particular AWS Glue DataBrew recipe in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_recipe_versions(
MaxResults=123,
NextToken='string',
Name='string'
)
integer
The maximum number of results to return in this request.
string
A pagination token that can be used in a subsequent request.
string
[REQUIRED]
The name of the recipe for which to return version information.
dict
Response Syntax
{
'NextToken': 'string',
'Recipes': [
{
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'ProjectName': 'string',
'PublishedBy': 'string',
'PublishedDate': datetime(2015, 1, 1),
'Description': 'string',
'Name': 'string',
'ResourceArn': 'string',
'Steps': [
{
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
],
'Tags': {
'string': 'string'
},
'RecipeVersion': 'string'
},
]
}
Response Structure
(dict) --
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
Recipes (list) --
A list of versions for the specified recipe.
(dict) --
Represents one or more actions to be performed on an AWS Glue DataBrew dataset.
CreatedBy (string) --
The identifier (the user name) of the user who created the recipe.
CreateDate (datetime) --
The date and time that the recipe was created.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the recipe.
LastModifiedDate (datetime) --
The last modification date and time of the recipe.
ProjectName (string) --
The name of the project that the recipe is associated with.
PublishedBy (string) --
The identifier (the user name) of the user who published the recipe.
PublishedDate (datetime) --
The date and time when the recipe was published.
Description (string) --
The description of the recipe.
Name (string) --
The unique name for the recipe.
ResourceArn (string) --
The Amazon Resource Name (ARN) for the recipe.
Steps (list) --
A list of steps that are defined by the recipe.
(dict) --
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) --
The particular action to be performed in the recipe step.
Operation (string) --
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) --
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) --
A column to apply this condition to, within an AWS Glue DataBrew dataset.
Tags (dict) --
Metadata tags that have been applied to the recipe.
(string) --
(string) --
RecipeVersion (string) --
The identifier for the version for the recipe.
Stops the specified job from running in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.stop_job_run(
Name='string',
RunId='string'
)
string
[REQUIRED]
The name of the job to be stopped.
string
[REQUIRED]
The ID of the job run to be stopped.
dict
Response Syntax
{
'RunId': 'string'
}
Response Structure
(dict) --
RunId (string) --
The ID of the job run that you stopped.
Lists all of the AWS Glue DataBrew recipes in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_recipes(
MaxResults=123,
NextToken='string',
RecipeVersion='string'
)
integer
The maximum number of results to return in this request.
string
A pagination token that can be used in a subsequent request.
string
A version identifier. Using this parameter indicates to return only those recipes that have this version identifier.
dict
Response Syntax
{
'Recipes': [
{
'CreatedBy': 'string',
'CreateDate': datetime(2015, 1, 1),
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'ProjectName': 'string',
'PublishedBy': 'string',
'PublishedDate': datetime(2015, 1, 1),
'Description': 'string',
'Name': 'string',
'ResourceArn': 'string',
'Steps': [
{
'Action': {
'Operation': 'string',
'Parameters': {
'string': 'string'
}
},
'ConditionExpressions': [
{
'Condition': 'string',
'Value': 'string',
'TargetColumn': 'string'
},
]
},
],
'Tags': {
'string': 'string'
},
'RecipeVersion': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Recipes (list) --
A list of recipes that are defined in the current AWS account.
(dict) --
Represents one or more actions to be performed on an AWS Glue DataBrew dataset.
CreatedBy (string) --
The identifier (the user name) of the user who created the recipe.
CreateDate (datetime) --
The date and time that the recipe was created.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the recipe.
LastModifiedDate (datetime) --
The last modification date and time of the recipe.
ProjectName (string) --
The name of the project that the recipe is associated with.
PublishedBy (string) --
The identifier (the user name) of the user who published the recipe.
PublishedDate (datetime) --
The date and time when the recipe was published.
Description (string) --
The description of the recipe.
Name (string) --
The unique name for the recipe.
ResourceArn (string) --
The Amazon Resource Name (ARN) for the recipe.
Steps (list) --
A list of steps that are defined by the recipe.
(dict) --
Represents a single step to be performed in an AWS Glue DataBrew recipe.
Action (dict) --
The particular action to be performed in the recipe step.
Operation (string) --
The name of a valid DataBrew transformation to be performed on the data.
Parameters (dict) --
Contextual parameters for the transformation.
(string) --
(string) --
ConditionExpressions (list) --
One or more conditions that must be met, in order for the recipe step to succeed.
Note
All of the conditions in the array must be met. In other words, all of the conditions must be combined using a logical AND operation.
(dict) --
Represents an individual condition that evaluates to true or false.
Conditions are used with recipe actions: The action is only performed for column values where the condition evaluates to true.
If a recipe requires more than one condition, then the recipe must specify multiple ConditionExpression elements. Each condition is applied to the rows in a dataset first, before the recipe action is performed.
Condition (string) --
A specific condition to apply to a recipe action. For more information, see Recipe structure in the AWS Glue DataBrew Developer Guide .
Value (string) --
A value that the condition must evaluate to for the condition to succeed.
TargetColumn (string) --
A column to apply this condition to, within an AWS Glue DataBrew dataset.
Tags (dict) --
Metadata tags that have been applied to the recipe.
(string) --
(string) --
RecipeVersion (string) --
The identifier for the version for the recipe.
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
Creates an interactive session, enabling you to manipulate an AWS Glue DataBrew project.
See also: AWS API Documentation
Request Syntax
client.start_project_session(
Name='string',
AssumeControl=True|False
)
string
[REQUIRED]
The name of the project to act upon.
boolean
A value that, if true, enables you to take control of a session, even if a different client is currently accessing the project.
dict
Response Syntax
{
'Name': 'string',
'ClientSessionId': 'string'
}
Response Structure
(dict) --
Name (string) --
The name of the project to be acted upon.
ClientSessionId (string) --
A system-generated identifier for the session.
Returns the definition of a specific AWS Glue DataBrew job that is in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.describe_job(
Name='string'
)
string
[REQUIRED]
The name of the job to be described.
dict
Response Syntax
{
'CreateDate': datetime(2015, 1, 1),
'CreatedBy': 'string',
'DatasetName': 'string',
'EncryptionKeyArn': 'string',
'EncryptionMode': 'SSE-KMS'|'SSE-S3',
'Name': 'string',
'Type': 'PROFILE'|'RECIPE',
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'LogSubscription': 'ENABLE'|'DISABLE',
'MaxCapacity': 123,
'MaxRetries': 123,
'Outputs': [
{
'CompressionFormat': 'GZIP'|'LZ4'|'SNAPPY'|'BZIP2'|'DEFLATE'|'LZO'|'BROTLI'|'ZSTD'|'ZLIB',
'Format': 'CSV'|'JSON'|'PARQUET'|'GLUEPARQUET'|'AVRO'|'ORC'|'XML',
'PartitionColumns': [
'string',
],
'Location': {
'Bucket': 'string',
'Key': 'string'
},
'Overwrite': True|False
},
],
'ProjectName': 'string',
'RecipeReference': {
'Name': 'string',
'RecipeVersion': 'string'
},
'ResourceArn': 'string',
'RoleArn': 'string',
'Tags': {
'string': 'string'
},
'Timeout': 123
}
Response Structure
(dict) --
CreateDate (datetime) --
The date and time that the job was created.
CreatedBy (string) --
The identifier (user name) of the user associated with the creation of the job.
DatasetName (string) --
The dataset that the job acts upon.
EncryptionKeyArn (string) --
The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
EncryptionMode (string) --
The encryption mode for the job, which can be one of the following:
SSE-KMS - Server-side encryption with AWS KMS-managed keys.
SSE-S3 - Server-side encryption with keys managed by Amazon S3.
Name (string) --
The name of the job.
Type (string) --
The job type, which must be one of the following:
PROFILE - The job analyzes the dataset to determine its size, data types, data distribution, and more.
RECIPE - The job applies one or more transformations to a dataset.
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the job.
LastModifiedDate (datetime) --
The date and time that the job was last modified.
LogSubscription (string) --
A value that indicates whether Amazon CloudWatch logging is enabled for this job.
MaxCapacity (integer) --
The maximum number of nodes that AWS Glue DataBrew can consume when the job processes data.
MaxRetries (integer) --
The maximum number of times to retry the job after a job run fails.
Outputs (list) --
One or more artifacts that represent the output from running the job.
(dict) --
Represents individual output from a particular job run.
CompressionFormat (string) --
The compression algorithm used to compress the output text of the job.
Format (string) --
The data format of the output of the job.
PartitionColumns (list) --
The names of one or more partition columns for the output of the job.
(string) --
Location (dict) --
The location in Amazon S3 where the job writes its output.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
Overwrite (boolean) --
A value that, if true, means that any data in the location specified for output is overwritten with new output.
ProjectName (string) --
The DataBrew project associated with this job.
RecipeReference (dict) --
Represents all of the attributes of an AWS Glue DataBrew recipe.
Name (string) --
The name of the recipe.
RecipeVersion (string) --
The identifier for the version for the recipe.
ResourceArn (string) --
The Amazon Resource Name (ARN) of the job.
RoleArn (string) --
The ARN of the AWS Identity and Access Management (IAM) role that was assumed for this request.
Tags (dict) --
Metadata tags associated with this job.
(string) --
(string) --
Timeout (integer) --
The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT .
Lists all of the previous runs of a particular AWS Glue DataBrew job in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.list_job_runs(
Name='string',
MaxResults=123,
NextToken='string'
)
string
[REQUIRED]
The name of the job.
integer
The maximum number of results to return in this request.
string
A token generated by AWS Glue DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
dict
Response Syntax
{
'JobRuns': [
{
'Attempt': 123,
'CompletedOn': datetime(2015, 1, 1),
'DatasetName': 'string',
'ErrorMessage': 'string',
'ExecutionTime': 123,
'JobName': 'string',
'RunId': 'string',
'State': 'STARTING'|'RUNNING'|'STOPPING'|'STOPPED'|'SUCCEEDED'|'FAILED'|'TIMEOUT',
'LogSubscription': 'ENABLE'|'DISABLE',
'LogGroupName': 'string',
'Outputs': [
{
'CompressionFormat': 'GZIP'|'LZ4'|'SNAPPY'|'BZIP2'|'DEFLATE'|'LZO'|'BROTLI'|'ZSTD'|'ZLIB',
'Format': 'CSV'|'JSON'|'PARQUET'|'GLUEPARQUET'|'AVRO'|'ORC'|'XML',
'PartitionColumns': [
'string',
],
'Location': {
'Bucket': 'string',
'Key': 'string'
},
'Overwrite': True|False
},
],
'RecipeReference': {
'Name': 'string',
'RecipeVersion': 'string'
},
'StartedBy': 'string',
'StartedOn': datetime(2015, 1, 1)
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
JobRuns (list) --
A list of job runs that have occurred for the specified job.
(dict) --
Represents one run of an AWS Glue DataBrew job.
Attempt (integer) --
The number of times that DataBrew has attempted to run the job.
CompletedOn (datetime) --
The date and time when the job completed processing.
DatasetName (string) --
The name of the dataset for the job to process.
ErrorMessage (string) --
A message indicating an error (if any) that was encountered when the job ran.
ExecutionTime (integer) --
The amount of time, in seconds, during which a job run consumed resources.
JobName (string) --
The name of the job being processed during this run.
RunId (string) --
The unique identifier of the job run.
State (string) --
The current state of the job run entity itself.
LogSubscription (string) --
The current status of Amazon CloudWatch logging for the job run.
LogGroupName (string) --
The name of an Amazon CloudWatch log group, where the job writes diagnostic messages when it runs.
Outputs (list) --
One or more output artifacts from a job run.
(dict) --
Represents individual output from a particular job run.
CompressionFormat (string) --
The compression algorithm used to compress the output text of the job.
Format (string) --
The data format of the output of the job.
PartitionColumns (list) --
The names of one or more partition columns for the output of the job.
(string) --
Location (dict) --
The location in Amazon S3 where the job writes its output.
Bucket (string) --
The S3 bucket name.
Key (string) --
The unique name of the object in the bucket.
Overwrite (boolean) --
A value that, if true, means that any data in the location specified for output is overwritten with new output.
RecipeReference (dict) --
The set of steps processed by the job.
Name (string) --
The name of the recipe.
RecipeVersion (string) --
The identifier for the version for the recipe.
StartedBy (string) --
The identifier (the user name) of the user who initiated the job run.
StartedOn (datetime) --
The date and time when the job run began.
NextToken (string) --
A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To obtain the next set of pages, pass in the NextToken from the response object of the previous page call.
Returns the definition of a specific AWS Glue DataBrew schedule that is in the current AWS account.
See also: AWS API Documentation
Request Syntax
client.describe_schedule(
Name='string'
)
string
[REQUIRED]
The name of the schedule to be described.
dict
Response Syntax
{
'CreateDate': datetime(2015, 1, 1),
'CreatedBy': 'string',
'JobNames': [
'string',
],
'LastModifiedBy': 'string',
'LastModifiedDate': datetime(2015, 1, 1),
'ResourceArn': 'string',
'CronExpression': 'string',
'Tags': {
'string': 'string'
},
'Name': 'string'
}
Response Structure
(dict) --
CreateDate (datetime) --
The date and time that the schedule was created.
CreatedBy (string) --
The identifier (user name) of the user who created the schedule.
JobNames (list) --
The name or names of one or more jobs to be run by using the schedule.
(string) --
LastModifiedBy (string) --
The identifier (user name) of the user who last modified the schedule.
LastModifiedDate (datetime) --
The date and time that the schedule was last modified.
ResourceArn (string) --
The Amazon Resource Name (ARN) of the schedule.
CronExpression (string) --
The date or dates and time or times, in cron format, when the jobs are to be run for the schedule.
Tags (dict) --
Metadata tags associated with this schedule.
(string) --
(string) --
Name (string) --
The name of the schedule.