2022/11/04 - Amazon EMR Containers - 4 new1 updated api methods
Changes Adding support for Job templates. Job templates allow you to create and store templates to configure Spark applications parameters. This helps you ensure consistent settings across applications by reusing and enforcing configuration overrides in data pipelines.
Deletes a job template. Job template stores values of StartJobRun API request in a template and can be used to start a job run. Job template allows two use cases: avoid repeating recurring StartJobRun API request values, enforcing certain values in StartJobRun API request.
See also: AWS API Documentation
Request Syntax
client.delete_job_template(
id='string'
)
string
[REQUIRED]
The ID of the job template that will be deleted.
dict
Response Syntax
{
'id': 'string'
}
Response Structure
(dict) --
id (string) --
This output contains the ID of the job template that was deleted.
Lists job templates based on a set of parameters. Job template stores values of StartJobRun API request in a template and can be used to start a job run. Job template allows two use cases: avoid repeating recurring StartJobRun API request values, enforcing certain values in StartJobRun API request.
See also: AWS API Documentation
Request Syntax
client.list_job_templates(
createdAfter=datetime(2015, 1, 1),
createdBefore=datetime(2015, 1, 1),
maxResults=123,
nextToken='string'
)
datetime
The date and time after which the job templates were created.
datetime
The date and time before which the job templates were created.
integer
The maximum number of job templates that can be listed.
string
The token for the next set of job templates to return.
dict
Response Syntax
{
'templates': [
{
'name': 'string',
'id': 'string',
'arn': 'string',
'createdAt': datetime(2015, 1, 1),
'createdBy': 'string',
'tags': {
'string': 'string'
},
'jobTemplateData': {
'executionRoleArn': 'string',
'releaseLabel': 'string',
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'persistentAppUI': 'string',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string'
}
}
},
'jobDriver': {
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
'parameterConfiguration': {
'string': {
'type': 'NUMBER'|'STRING',
'defaultValue': 'string'
}
},
'jobTags': {
'string': 'string'
}
},
'kmsKeyArn': 'string',
'decryptionError': 'string'
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
templates (list) --
This output lists information about the specified job templates.
(dict) --
This entity describes a job template. Job template stores values of StartJobRun API request in a template and can be used to start a job run. Job template allows two use cases: avoid repeating recurring StartJobRun API request values, enforcing certain values in StartJobRun API request.
name (string) --
The name of the job template.
id (string) --
The ID of the job template.
arn (string) --
The ARN of the job template.
createdAt (datetime) --
The date and time when the job template was created.
createdBy (string) --
The user who created the job template.
tags (dict) --
The tags assigned to the job template.
(string) --
(string) --
jobTemplateData (dict) --
The job template data which holds values of StartJobRun API request.
executionRoleArn (string) --
The execution role ARN of the job run.
releaseLabel (string) --
The release version of Amazon EMR.
configurationOverrides (dict) --
The configuration settings that are used to override defaults configuration.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
jobDriver (dict) --
Specify the driver that the job runs on. Exactly one of the two available job drivers is required, either sparkSqlJobDriver or sparkSubmitJobDriver.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) --
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
parameterConfiguration (dict) --
The configuration of parameters existing in the job template.
(string) --
(dict) --
The configuration of a job template parameter.
type (string) --
The type of the job template parameter. Allowed values are: ‘String’, ‘Number’.
defaultValue (string) --
The default value for the job template parameter.
jobTags (dict) --
The tags assigned to jobs started using the job template.
(string) --
(string) --
kmsKeyArn (string) --
The KMS key ARN used to encrypt the job template.
decryptionError (string) --
The error message in case the decryption of job template fails.
nextToken (string) --
This output displays the token for the next set of job templates.
Creates a job template. Job template stores values of StartJobRun API request in a template and can be used to start a job run. Job template allows two use cases: avoid repeating recurring StartJobRun API request values, enforcing certain values in StartJobRun API request.
See also: AWS API Documentation
Request Syntax
client.create_job_template(
name='string',
clientToken='string',
jobTemplateData={
'executionRoleArn': 'string',
'releaseLabel': 'string',
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'persistentAppUI': 'string',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string'
}
}
},
'jobDriver': {
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
'parameterConfiguration': {
'string': {
'type': 'NUMBER'|'STRING',
'defaultValue': 'string'
}
},
'jobTags': {
'string': 'string'
}
},
tags={
'string': 'string'
},
kmsKeyArn='string'
)
string
[REQUIRED]
The specified name of the job template.
string
[REQUIRED]
The client token of the job template.
This field is autopopulated if not provided.
dict
[REQUIRED]
The job template data which holds values of StartJobRun API request.
executionRoleArn (string) -- [REQUIRED]
The execution role ARN of the job run.
releaseLabel (string) -- [REQUIRED]
The release version of Amazon EMR.
configurationOverrides (dict) --
The configuration settings that are used to override defaults configuration.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) -- [REQUIRED]
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
jobDriver (dict) -- [REQUIRED]
Specify the driver that the job runs on. Exactly one of the two available job drivers is required, either sparkSqlJobDriver or sparkSubmitJobDriver.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) -- [REQUIRED]
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
parameterConfiguration (dict) --
The configuration of parameters existing in the job template.
(string) --
(dict) --
The configuration of a job template parameter.
type (string) --
The type of the job template parameter. Allowed values are: ‘String’, ‘Number’.
defaultValue (string) --
The default value for the job template parameter.
jobTags (dict) --
The tags assigned to jobs started using the job template.
(string) --
(string) --
dict
The tags that are associated with the job template.
(string) --
(string) --
string
The KMS key ARN used to encrypt the job template.
dict
Response Syntax
{
'id': 'string',
'name': 'string',
'arn': 'string',
'createdAt': datetime(2015, 1, 1)
}
Response Structure
(dict) --
id (string) --
This output display the created job template ID.
name (string) --
This output displays the name of the created job template.
arn (string) --
This output display the ARN of the created job template.
createdAt (datetime) --
This output displays the date and time when the job template was created.
Displays detailed information about a specified job template. Job template stores values of StartJobRun API request in a template and can be used to start a job run. Job template allows two use cases: avoid repeating recurring StartJobRun API request values, enforcing certain values in StartJobRun API request.
See also: AWS API Documentation
Request Syntax
client.describe_job_template(
id='string'
)
string
[REQUIRED]
The ID of the job template that will be described.
dict
Response Syntax
{
'jobTemplate': {
'name': 'string',
'id': 'string',
'arn': 'string',
'createdAt': datetime(2015, 1, 1),
'createdBy': 'string',
'tags': {
'string': 'string'
},
'jobTemplateData': {
'executionRoleArn': 'string',
'releaseLabel': 'string',
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'persistentAppUI': 'string',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string'
}
}
},
'jobDriver': {
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
'parameterConfiguration': {
'string': {
'type': 'NUMBER'|'STRING',
'defaultValue': 'string'
}
},
'jobTags': {
'string': 'string'
}
},
'kmsKeyArn': 'string',
'decryptionError': 'string'
}
}
Response Structure
(dict) --
jobTemplate (dict) --
This output displays information about the specified job template.
name (string) --
The name of the job template.
id (string) --
The ID of the job template.
arn (string) --
The ARN of the job template.
createdAt (datetime) --
The date and time when the job template was created.
createdBy (string) --
The user who created the job template.
tags (dict) --
The tags assigned to the job template.
(string) --
(string) --
jobTemplateData (dict) --
The job template data which holds values of StartJobRun API request.
executionRoleArn (string) --
The execution role ARN of the job run.
releaseLabel (string) --
The release version of Amazon EMR.
configurationOverrides (dict) --
The configuration settings that are used to override defaults configuration.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
jobDriver (dict) --
Specify the driver that the job runs on. Exactly one of the two available job drivers is required, either sparkSqlJobDriver or sparkSubmitJobDriver.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) --
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
parameterConfiguration (dict) --
The configuration of parameters existing in the job template.
(string) --
(dict) --
The configuration of a job template parameter.
type (string) --
The type of the job template parameter. Allowed values are: ‘String’, ‘Number’.
defaultValue (string) --
The default value for the job template parameter.
jobTags (dict) --
The tags assigned to jobs started using the job template.
(string) --
(string) --
kmsKeyArn (string) --
The KMS key ARN used to encrypt the job template.
decryptionError (string) --
The error message in case the decryption of job template fails.
{'jobTemplateId': 'string', 'jobTemplateParameters': {'string': 'string'}}
Starts a job run. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.
See also: AWS API Documentation
Request Syntax
client.start_job_run(
name='string',
virtualClusterId='string',
clientToken='string',
executionRoleArn='string',
releaseLabel='string',
jobDriver={
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
configurationOverrides={
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string'
}
}
},
tags={
'string': 'string'
},
jobTemplateId='string',
jobTemplateParameters={
'string': 'string'
}
)
string
The name of the job run.
string
[REQUIRED]
The virtual cluster ID for which the job run request is submitted.
string
[REQUIRED]
The client idempotency token of the job run request.
This field is autopopulated if not provided.
string
The execution role ARN for the job run.
string
The Amazon EMR release version to use for the job run.
dict
The job driver for the job run.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) -- [REQUIRED]
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
dict
The configuration overrides for the job run.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) -- [REQUIRED]
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) -- [REQUIRED]
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) -- [REQUIRED]
Amazon S3 destination URI for log publishing.
dict
The tags assigned to job runs.
(string) --
(string) --
string
The job template ID to be used to start the job run.
dict
The values of job template parameters to start a job run.
(string) --
(string) --
dict
Response Syntax
{
'id': 'string',
'name': 'string',
'arn': 'string',
'virtualClusterId': 'string'
}
Response Structure
(dict) --
id (string) --
This output displays the started job run ID.
name (string) --
This output displays the name of the started job run.
arn (string) --
This output lists the ARN of job run.
virtualClusterId (string) --
This output displays the virtual cluster ID for which the job run was submitted.