2021/03/04 - AmazonMWAA - 3 updated api methods
Changes This release introduces a new MinWorker parameter to the CreateEnvironment and UpdateEnvironment APIs. MinWorker allows the users to set a minimum worker count for worker auto-scaling operations.
{'MinWorkers': 'integer'}
JSON blob that describes the environment to create.
See also: AWS API Documentation
Request Syntax
client.create_environment(
AirflowConfigurationOptions={
'string': 'string'
},
AirflowVersion='string',
DagS3Path='string',
EnvironmentClass='string',
ExecutionRoleArn='string',
KmsKey='string',
LoggingConfiguration={
'DagProcessingLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'SchedulerLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'TaskLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WebserverLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WorkerLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
}
},
MaxWorkers=123,
MinWorkers=123,
Name='string',
NetworkConfiguration={
'SecurityGroupIds': [
'string',
],
'SubnetIds': [
'string',
]
},
PluginsS3ObjectVersion='string',
PluginsS3Path='string',
RequirementsS3ObjectVersion='string',
RequirementsS3Path='string',
SourceBucketArn='string',
Tags={
'string': 'string'
},
WebserverAccessMode='PRIVATE_ONLY'|'PUBLIC_ONLY',
WeeklyMaintenanceWindowStart='string'
)
dict
The Apache Airflow configuration setting you want to override in your environment. For more information, see Environment configuration.
(string) --
(string) --
string
The Apache Airflow version you want to use for your environment.
string
[REQUIRED]
The relative path to the DAG folder on your Amazon S3 storage bucket. For example, dags . For more information, see Importing DAGs on Amazon MWAA.
string
The environment class you want to use for your environment. The environment class determines the size of the containers and database used for your Apache Airflow services.
string
[REQUIRED]
The Amazon Resource Name (ARN) of the execution role for your environment. An execution role is an AWS Identity and Access Management (IAM) role that grants MWAA permission to access AWS services and resources used by your environment. For example, arn:aws:iam::123456789:role/my-execution-role . For more information, see Managing access to Amazon Managed Workflows for Apache Airflow.
string
The AWS Key Management Service (KMS) key to encrypt and decrypt the data in your environment. You can use an AWS KMS key managed by MWAA, or a custom KMS key (advanced). For more information, see Customer master keys (CMKs) in the AWS KMS developer guide.
dict
The Apache Airflow logs you want to send to Amazon CloudWatch Logs.
DagProcessingLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
SchedulerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
TaskLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WebserverLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WorkerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
integer
The maximum number of workers that you want to run in your environment. MWAA scales the number of Apache Airflow workers and the Fargate containers that run your tasks up to the number you specify in this field. When there are no more tasks running, and no more in the queue, MWAA disposes of the extra containers leaving the one worker that is included with your environment.
integer
The minimum number of workers that you want to run in your environment. MWAA scales the number of Apache Airflow workers and the Fargate containers that run your tasks up to the number you specify in the MaxWorkers field. When there are no more tasks running, and no more in the queue, MWAA disposes of the extra containers leaving the worker count you specify in the MinWorkers field.
string
[REQUIRED]
The name of your MWAA environment.
dict
[REQUIRED]
The VPC networking components you want to use for your environment. At least two private subnet identifiers and one VPC security group identifier are required to create an environment. For more information, see Creating the VPC network for a MWAA environment.
SecurityGroupIds (list) --
A JSON list of 1 or more security groups IDs by name, in the same VPC as the subnets.
(string) --
SubnetIds (list) --
Provide a JSON list of 2 subnet IDs by name. These must be private subnets, in the same VPC, in two different availability zones.
(string) --
string
The plugins.zip file version you want to use.
string
The relative path to the plugins.zip file on your Amazon S3 storage bucket. For example, plugins.zip . If a relative path is provided in the request, then PluginsS3ObjectVersion is required. For more information, see Importing DAGs on Amazon MWAA.
string
The requirements.txt file version you want to use.
string
The relative path to the requirements.txt file on your Amazon S3 storage bucket. For example, requirements.txt . If a relative path is provided in the request, then RequirementsS3ObjectVersion is required. For more information, see Importing DAGs on Amazon MWAA.
string
[REQUIRED]
The Amazon Resource Name (ARN) of your Amazon S3 storage bucket. For example, arn:aws:s3:::airflow-mybucketname .
dict
The metadata tags you want to attach to your environment. For more information, see Tagging AWS resources.
(string) --
(string) --
string
The networking access of your Apache Airflow web server. A public network allows your Airflow UI to be accessed over the Internet by users granted access in your IAM policy. A private network limits access of your Airflow UI to users within your VPC. For more information, see Creating the VPC network for a MWAA environment.
string
The day and time you want MWAA to start weekly maintenance updates on your environment.
dict
Response Syntax
{
'Arn': 'string'
}
Response Structure
(dict) --
Arn (string) --
The resulting Amazon MWAA envirnonment ARN.
{'Environment': {'MinWorkers': 'integer'}}
Get details of an existing environment.
See also: AWS API Documentation
Request Syntax
client.get_environment(
Name='string'
)
string
[REQUIRED]
The name of the environment to retrieve.
dict
Response Syntax
{
'Environment': {
'AirflowConfigurationOptions': {
'string': 'string'
},
'AirflowVersion': 'string',
'Arn': 'string',
'CreatedAt': datetime(2015, 1, 1),
'DagS3Path': 'string',
'EnvironmentClass': 'string',
'ExecutionRoleArn': 'string',
'KmsKey': 'string',
'LastUpdate': {
'CreatedAt': datetime(2015, 1, 1),
'Error': {
'ErrorCode': 'string',
'ErrorMessage': 'string'
},
'Status': 'SUCCESS'|'PENDING'|'FAILED'
},
'LoggingConfiguration': {
'DagProcessingLogs': {
'CloudWatchLogGroupArn': 'string',
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'SchedulerLogs': {
'CloudWatchLogGroupArn': 'string',
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'TaskLogs': {
'CloudWatchLogGroupArn': 'string',
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WebserverLogs': {
'CloudWatchLogGroupArn': 'string',
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WorkerLogs': {
'CloudWatchLogGroupArn': 'string',
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
}
},
'MaxWorkers': 123,
'MinWorkers': 123,
'Name': 'string',
'NetworkConfiguration': {
'SecurityGroupIds': [
'string',
],
'SubnetIds': [
'string',
]
},
'PluginsS3ObjectVersion': 'string',
'PluginsS3Path': 'string',
'RequirementsS3ObjectVersion': 'string',
'RequirementsS3Path': 'string',
'ServiceRoleArn': 'string',
'SourceBucketArn': 'string',
'Status': 'CREATING'|'CREATE_FAILED'|'AVAILABLE'|'UPDATING'|'DELETING'|'DELETED',
'Tags': {
'string': 'string'
},
'WebserverAccessMode': 'PRIVATE_ONLY'|'PUBLIC_ONLY',
'WebserverUrl': 'string',
'WeeklyMaintenanceWindowStart': 'string'
}
}
Response Structure
(dict) --
Environment (dict) --
A JSON blob with environment details.
AirflowConfigurationOptions (dict) --
The Airflow Configuration Options of the Amazon MWAA Environment.
(string) --
(string) --
AirflowVersion (string) --
The AirflowV ersion of the Amazon MWAA Environment.
Arn (string) --
The ARN of the Amazon MWAA Environment.
CreatedAt (datetime) --
The Created At date of the Amazon MWAA Environment.
DagS3Path (string) --
The Dags S3 Path of the Amazon MWAA Environment.
EnvironmentClass (string) --
The Environment Class (size) of the Amazon MWAA Environment.
ExecutionRoleArn (string) --
The Execution Role ARN of the Amazon MWAA Environment.
KmsKey (string) --
The Kms Key of the Amazon MWAA Environment.
LastUpdate (dict) --
Last update information for the environment.
CreatedAt (datetime) --
Time that last update occurred.
Error (dict) --
Error string of last update, if applicable.
ErrorCode (string) --
Error code of update.
ErrorMessage (string) --
Error message of update.
Status (string) --
Status of last update of SUCCESS, FAILED, CREATING, DELETING.
LoggingConfiguration (dict) --
The Logging Configuration of the Amazon MWAA Environment.
DagProcessingLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
CloudWatchLogGroupArn (string) --
Provides the ARN for the CloudWatch group where the logs will be published.
Enabled (boolean) --
Defines that the logging module is enabled.
LogLevel (string) --
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
SchedulerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
CloudWatchLogGroupArn (string) --
Provides the ARN for the CloudWatch group where the logs will be published.
Enabled (boolean) --
Defines that the logging module is enabled.
LogLevel (string) --
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
TaskLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
CloudWatchLogGroupArn (string) --
Provides the ARN for the CloudWatch group where the logs will be published.
Enabled (boolean) --
Defines that the logging module is enabled.
LogLevel (string) --
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WebserverLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
CloudWatchLogGroupArn (string) --
Provides the ARN for the CloudWatch group where the logs will be published.
Enabled (boolean) --
Defines that the logging module is enabled.
LogLevel (string) --
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WorkerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
CloudWatchLogGroupArn (string) --
Provides the ARN for the CloudWatch group where the logs will be published.
Enabled (boolean) --
Defines that the logging module is enabled.
LogLevel (string) --
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
MaxWorkers (integer) --
The maximum number of workers to run in your Amazon MWAA Environment.
MinWorkers (integer) --
The minimum number of workers to run in your Amazon MWAA Environment.
Name (string) --
The name of the Amazon MWAA Environment.
NetworkConfiguration (dict) --
Provide the security group and subnet IDs for the workers and scheduler.
SecurityGroupIds (list) --
A JSON list of 1 or more security groups IDs by name, in the same VPC as the subnets.
(string) --
SubnetIds (list) --
Provide a JSON list of 2 subnet IDs by name. These must be private subnets, in the same VPC, in two different availability zones.
(string) --
PluginsS3ObjectVersion (string) --
The Plugins.zip S3 Object Version of the Amazon MWAA Environment.
PluginsS3Path (string) --
The Plugins.zip S3 Path of the Amazon MWAA Environment.
RequirementsS3ObjectVersion (string) --
The Requirements.txt file S3 Object Version of the Amazon MWAA Environment.
RequirementsS3Path (string) --
The Requirement.txt S3 Path of the Amazon MWAA Environment.
ServiceRoleArn (string) --
The Service Role ARN of the Amazon MWAA Environment.
SourceBucketArn (string) --
The Source S3 Bucket ARN of the Amazon MWAA Environment.
Status (string) --
The status of the Amazon MWAA Environment.
Tags (dict) --
The Tags of the Amazon MWAA Environment.
(string) --
(string) --
WebserverAccessMode (string) --
The Webserver Access Mode of the Amazon MWAA Environment (public or private only).
WebserverUrl (string) --
The Webserver URL of the Amazon MWAA Environment.
WeeklyMaintenanceWindowStart (string) --
The Weekly Maintenance Window Start of the Amazon MWAA Environment.
{'MinWorkers': 'integer'}
Update an MWAA environment.
See also: AWS API Documentation
Request Syntax
client.update_environment(
AirflowConfigurationOptions={
'string': 'string'
},
AirflowVersion='string',
DagS3Path='string',
EnvironmentClass='string',
ExecutionRoleArn='string',
LoggingConfiguration={
'DagProcessingLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'SchedulerLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'TaskLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WebserverLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
},
'WorkerLogs': {
'Enabled': True|False,
'LogLevel': 'CRITICAL'|'ERROR'|'WARNING'|'INFO'|'DEBUG'
}
},
MaxWorkers=123,
MinWorkers=123,
Name='string',
NetworkConfiguration={
'SecurityGroupIds': [
'string',
]
},
PluginsS3ObjectVersion='string',
PluginsS3Path='string',
RequirementsS3ObjectVersion='string',
RequirementsS3Path='string',
SourceBucketArn='string',
WebserverAccessMode='PRIVATE_ONLY'|'PUBLIC_ONLY',
WeeklyMaintenanceWindowStart='string'
)
dict
The Airflow Configuration Options to update of your Amazon MWAA environment.
(string) --
(string) --
string
The Airflow Version to update of your Amazon MWAA environment.
string
The Dags folder S3 Path to update of your Amazon MWAA environment.
string
The Environment Class to update of your Amazon MWAA environment.
string
The Executio Role ARN to update of your Amazon MWAA environment.
dict
The Logging Configuration to update of your Amazon MWAA environment.
DagProcessingLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
SchedulerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
TaskLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WebserverLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
WorkerLogs (dict) --
A JSON blob that provides configuration to use for logging with respect to the various Apache Airflow services: DagProcessingLogs, SchedulerLogs, TaskLogs, WebserverLogs, and WorkerLogs.
Enabled (boolean) -- [REQUIRED]
Defines that the logging module is enabled.
LogLevel (string) -- [REQUIRED]
Defines the log level, which can be CRITICAL, ERROR, WARNING, or INFO.
integer
The maximum number of workers to update of your Amazon MWAA environment.
integer
The minimum number of workers to update of your Amazon MWAA environment.
string
[REQUIRED]
The name of your Amazon MWAA environment that you wish to update.
dict
The Network Configuration to update of your Amazon MWAA environment.
SecurityGroupIds (list) -- [REQUIRED]
Provide a JSON list of 1 or more security groups IDs by name, in the same VPC as the subnets.
(string) --
string
The Plugins.zip S3 Object Version to update of your Amazon MWAA environment.
string
The Plugins.zip S3 Path to update of your Amazon MWAA environment.
string
The Requirements.txt S3 ObjectV ersion to update of your Amazon MWAA environment.
string
The Requirements.txt S3 Path to update of your Amazon MWAA environment.
string
The S3 Source Bucket ARN to update of your Amazon MWAA environment.
string
The Webserver Access Mode to update of your Amazon MWAA environment.
string
The Weekly Maintenance Window Start to update of your Amazon MWAA environment.
dict
Response Syntax
{
'Arn': 'string'
}
Response Structure
(dict) --
Arn (string) --
The ARN to update of your Amazon MWAA environment.