2026/07/14 - Amazon EMR Containers - 10 updated api methods
Changes Introduced 5 new fields across 3 APIs as part of Spark Connect server launch for EMR on EKS. The fields added are sessionIdleTimeoutInMinutes, sessionEnabled, endpointToken, authProxyUrl and encryptionKeyArn.
{'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}},
'sessionIdleTimeoutInMinutes': 'integer'}
Creates a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual cluster.
See also: AWS API Documentation
Request Syntax
client.create_managed_endpoint(
name='string',
virtualClusterId='string',
type='string',
releaseLabel='string',
executionRoleArn='string',
certificateArn='string',
configurationOverrides={
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
clientToken='string',
tags={
'string': 'string'
},
sessionIdleTimeoutInMinutes=123
)
string
[REQUIRED]
The name of the managed endpoint.
string
[REQUIRED]
The ID of the virtual cluster for which a managed endpoint is created.
string
[REQUIRED]
The type of the managed endpoint.
string
[REQUIRED]
The Amazon EMR release version.
string
[REQUIRED]
The ARN of the execution role.
string
The certificate ARN provided by users for the managed endpoint. This field is under deprecation and will be removed in future releases.
dict
The configuration settings that will be used to override existing configurations.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) -- [REQUIRED]
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) -- [REQUIRED]
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) -- [REQUIRED]
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) -- [REQUIRED]
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) -- [REQUIRED]
The number of files to keep in container after rotation.
string
[REQUIRED]
The client idempotency token for this create call.
This field is autopopulated if not provided.
dict
The tags of the managed endpoint.
(string) --
(string) --
integer
The idle timeout in minutes for the managed endpoint session.
dict
Response Syntax
{
'id': 'string',
'name': 'string',
'arn': 'string',
'virtualClusterId': 'string'
}
Response Structure
(dict) --
id (string) --
The output contains the ID of the managed endpoint.
name (string) --
The output contains the name of the managed endpoint.
arn (string) --
The output contains the ARN of the managed endpoint.
virtualClusterId (string) --
The output contains the ID of the virtual cluster.
{'sessionEnabled': 'boolean'}
Creates a virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.
See also: AWS API Documentation
Request Syntax
client.create_virtual_cluster(
name='string',
containerProvider={
'type': 'EKS',
'id': 'string',
'info': {
'eksInfo': {
'namespace': 'string',
'nodeLabel': 'string'
}
}
},
clientToken='string',
tags={
'string': 'string'
},
securityConfigurationId='string',
sessionEnabled=True|False
)
string
[REQUIRED]
The specified name of the virtual cluster.
dict
[REQUIRED]
The container provider of the virtual cluster.
type (string) -- [REQUIRED]
The type of the container provider. Amazon EKS is the only supported type as of now.
id (string) -- [REQUIRED]
The ID of the container cluster.
info (dict) --
The information about the container cluster.
eksInfo (dict) --
The information about the Amazon EKS cluster.
namespace (string) --
The namespaces of the Amazon EKS cluster.
nodeLabel (string) --
The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
string
[REQUIRED]
The client token of the virtual cluster.
This field is autopopulated if not provided.
dict
The tags assigned to the virtual cluster.
(string) --
(string) --
string
The ID of the security configuration.
boolean
Indicates whether the virtual cluster has session support enabled.
dict
Response Syntax
{
'id': 'string',
'name': 'string',
'arn': 'string'
}
Response Structure
(dict) --
id (string) --
This output contains the virtual cluster ID.
name (string) --
This output contains the name of the virtual cluster.
arn (string) --
This output contains the ARN of virtual cluster.
{'jobRun': {'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}}}}
Displays detailed information about a job run. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.
See also: AWS API Documentation
Request Syntax
client.describe_job_run(
id='string',
virtualClusterId='string'
)
string
[REQUIRED]
The ID of the job run request.
string
[REQUIRED]
The ID of the virtual cluster for which the job run is submitted.
dict
Response Syntax
{
'jobRun': {
'id': 'string',
'name': 'string',
'virtualClusterId': 'string',
'arn': 'string',
'state': 'PENDING'|'SUBMITTED'|'RUNNING'|'FAILED'|'CANCELLED'|'CANCEL_PENDING'|'COMPLETED',
'clientToken': 'string',
'executionRoleArn': 'string',
'releaseLabel': 'string',
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
'jobDriver': {
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
'createdAt': datetime(2015, 1, 1),
'createdBy': 'string',
'finishedAt': datetime(2015, 1, 1),
'stateDetails': 'string',
'failureReason': 'INTERNAL_ERROR'|'USER_ERROR'|'VALIDATION_ERROR'|'CLUSTER_UNAVAILABLE',
'tags': {
'string': 'string'
},
'retryPolicyConfiguration': {
'maxAttempts': 123
},
'retryPolicyExecution': {
'currentAttemptCount': 123
}
}
}
Response Structure
(dict) --
jobRun (dict) --
The output displays information about a job run.
id (string) --
The ID of the job run.
name (string) --
The name of the job run.
virtualClusterId (string) --
The ID of the job run's virtual cluster.
arn (string) --
The ARN of job run.
state (string) --
The state of the job run.
clientToken (string) --
The client token used to start a job run.
executionRoleArn (string) --
The execution role ARN of the job run.
releaseLabel (string) --
The release version of Amazon EMR.
configurationOverrides (dict) --
The configuration settings that are used to override default configuration.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) --
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) --
The number of files to keep in container after rotation.
jobDriver (dict) --
Parameters of job driver for the job run.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) --
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
createdAt (datetime) --
The date and time when the job run was created.
createdBy (string) --
The user who created the job run.
finishedAt (datetime) --
The date and time when the job run has finished.
stateDetails (string) --
Additional details of the job run state.
failureReason (string) --
The reasons why the job run has failed.
tags (dict) --
The assigned tags of the job run.
(string) --
(string) --
retryPolicyConfiguration (dict) --
The configuration of the retry policy that the job runs on.
maxAttempts (integer) --
The maximum number of attempts on the job's driver.
retryPolicyExecution (dict) --
The current status of the retry policy executed on the job.
currentAttemptCount (integer) --
The current number of attempts made on the driver of the job.
{'endpoint': {'authProxyUrl': 'string',
'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}}}}
Displays detailed information about a managed endpoint. A managed endpoint is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual cluster.
See also: AWS API Documentation
Request Syntax
client.describe_managed_endpoint(
id='string',
virtualClusterId='string'
)
string
[REQUIRED]
This output displays ID of the managed endpoint.
string
[REQUIRED]
The ID of the endpoint's virtual cluster.
dict
Response Syntax
{
'endpoint': {
'id': 'string',
'name': 'string',
'arn': 'string',
'virtualClusterId': 'string',
'type': 'string',
'state': 'CREATING'|'ACTIVE'|'TERMINATING'|'TERMINATED'|'TERMINATED_WITH_ERRORS',
'releaseLabel': 'string',
'executionRoleArn': 'string',
'certificateArn': 'string',
'certificateAuthority': {
'certificateArn': 'string',
'certificateData': 'string'
},
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
'serverUrl': 'string',
'authProxyUrl': 'string',
'createdAt': datetime(2015, 1, 1),
'securityGroup': 'string',
'subnetIds': [
'string',
],
'stateDetails': 'string',
'failureReason': 'INTERNAL_ERROR'|'USER_ERROR'|'VALIDATION_ERROR'|'CLUSTER_UNAVAILABLE',
'tags': {
'string': 'string'
}
}
}
Response Structure
(dict) --
endpoint (dict) --
This output displays information about a managed endpoint.
id (string) --
The ID of the endpoint.
name (string) --
The name of the endpoint.
arn (string) --
The ARN of the endpoint.
virtualClusterId (string) --
The ID of the endpoint's virtual cluster.
type (string) --
The type of the endpoint.
state (string) --
The state of the endpoint.
releaseLabel (string) --
The EMR release version to be used for the endpoint.
executionRoleArn (string) --
The execution role ARN of the endpoint.
certificateArn (string) --
The certificate ARN of the endpoint. This field is under deprecation and will be removed in future.
certificateAuthority (dict) --
The certificate generated by emr control plane on customer behalf to secure the managed endpoint.
certificateArn (string) --
The ARN of the certificate generated for managed endpoint.
certificateData (string) --
The base64 encoded PEM certificate data generated for managed endpoint.
configurationOverrides (dict) --
The configuration settings that are used to override existing configurations for endpoints.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) --
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) --
The number of files to keep in container after rotation.
serverUrl (string) --
The server URL of the endpoint.
authProxyUrl (string) --
The auth proxy URL of the endpoint.
createdAt (datetime) --
The date and time when the endpoint was created.
securityGroup (string) --
The security group configuration of the endpoint.
subnetIds (list) --
The subnet IDs of the endpoint.
(string) --
stateDetails (string) --
Additional details of the endpoint state.
failureReason (string) --
The reasons why the endpoint has failed.
tags (dict) --
The tags of the endpoint.
(string) --
(string) --
{'virtualCluster': {'sessionEnabled': 'boolean'}}
Displays detailed information about a specified virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.
See also: AWS API Documentation
Request Syntax
client.describe_virtual_cluster(
id='string'
)
string
[REQUIRED]
The ID of the virtual cluster that will be described.
dict
Response Syntax
{
'virtualCluster': {
'id': 'string',
'name': 'string',
'arn': 'string',
'state': 'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
'containerProvider': {
'type': 'EKS',
'id': 'string',
'info': {
'eksInfo': {
'namespace': 'string',
'nodeLabel': 'string'
}
}
},
'createdAt': datetime(2015, 1, 1),
'tags': {
'string': 'string'
},
'securityConfigurationId': 'string',
'sessionEnabled': True|False
}
}
Response Structure
(dict) --
virtualCluster (dict) --
This output displays information about the specified virtual cluster.
id (string) --
The ID of the virtual cluster.
name (string) --
The name of the virtual cluster.
arn (string) --
The ARN of the virtual cluster.
state (string) --
The state of the virtual cluster.
containerProvider (dict) --
The container provider of the virtual cluster.
type (string) --
The type of the container provider. Amazon EKS is the only supported type as of now.
id (string) --
The ID of the container cluster.
info (dict) --
The information about the container cluster.
eksInfo (dict) --
The information about the Amazon EKS cluster.
namespace (string) --
The namespaces of the Amazon EKS cluster.
nodeLabel (string) --
The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
createdAt (datetime) --
The date and time when the virtual cluster is created.
tags (dict) --
The assigned tags of the virtual cluster.
(string) --
(string) --
securityConfigurationId (string) --
The ID of the security configuration.
sessionEnabled (boolean) --
Indicates whether the virtual cluster has session support enabled.
{'endpointCredentials': {'token': 'string'}}
Generate a session token to connect to a managed endpoint.
See also: AWS API Documentation
Request Syntax
client.get_managed_endpoint_session_credentials(
endpointIdentifier='string',
virtualClusterIdentifier='string',
executionRoleArn='string',
credentialType='string',
durationInSeconds=123,
logContext='string',
clientToken='string'
)
string
[REQUIRED]
The ARN of the managed endpoint for which the request is submitted.
string
[REQUIRED]
The ARN of the Virtual Cluster which the Managed Endpoint belongs to.
string
[REQUIRED]
The IAM Execution Role ARN that will be used by the job run.
string
[REQUIRED]
Type of the token requested. Currently supported and default value of this field is “TOKEN.”
integer
Duration in seconds for which the session token is valid. The default duration is 15 minutes and the maximum is 12 hours.
string
String identifier used to separate sections of the execution logs uploaded to S3.
string
The client idempotency token of the job run request.
This field is autopopulated if not provided.
dict
Response Syntax
{
'id': 'string',
'credentials': {
'token': 'string'
},
'endpointCredentials': {
'token': 'string'
},
'expiresAt': datetime(2015, 1, 1)
}
Response Structure
(dict) --
id (string) --
The identifier of the session token returned.
credentials (dict) --
The structure containing the session credentials.
token (string) --
The actual session token being returned.
endpointCredentials (dict) --
The structure containing the session token being returned.
token (string) --
The actual session token being returned.
expiresAt (datetime) --
The date and time when the session token will expire.
{'jobRuns': {'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}}}}
Lists job runs based on a set of parameters. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.
See also: AWS API Documentation
Request Syntax
client.list_job_runs(
virtualClusterId='string',
createdBefore=datetime(2015, 1, 1),
createdAfter=datetime(2015, 1, 1),
name='string',
states=[
'PENDING'|'SUBMITTED'|'RUNNING'|'FAILED'|'CANCELLED'|'CANCEL_PENDING'|'COMPLETED',
],
maxResults=123,
nextToken='string'
)
string
[REQUIRED]
The ID of the virtual cluster for which to list the job run.
datetime
The date and time before which the job runs were submitted.
datetime
The date and time after which the job runs were submitted.
string
The name of the job run.
list
The states of the job run.
(string) --
integer
The maximum number of job runs that can be listed.
string
The token for the next set of job runs to return.
dict
Response Syntax
{
'jobRuns': [
{
'id': 'string',
'name': 'string',
'virtualClusterId': 'string',
'arn': 'string',
'state': 'PENDING'|'SUBMITTED'|'RUNNING'|'FAILED'|'CANCELLED'|'CANCEL_PENDING'|'COMPLETED',
'clientToken': 'string',
'executionRoleArn': 'string',
'releaseLabel': 'string',
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
'jobDriver': {
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
'createdAt': datetime(2015, 1, 1),
'createdBy': 'string',
'finishedAt': datetime(2015, 1, 1),
'stateDetails': 'string',
'failureReason': 'INTERNAL_ERROR'|'USER_ERROR'|'VALIDATION_ERROR'|'CLUSTER_UNAVAILABLE',
'tags': {
'string': 'string'
},
'retryPolicyConfiguration': {
'maxAttempts': 123
},
'retryPolicyExecution': {
'currentAttemptCount': 123
}
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
jobRuns (list) --
This output lists information about the specified job runs.
(dict) --
This entity describes a job run. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.
id (string) --
The ID of the job run.
name (string) --
The name of the job run.
virtualClusterId (string) --
The ID of the job run's virtual cluster.
arn (string) --
The ARN of job run.
state (string) --
The state of the job run.
clientToken (string) --
The client token used to start a job run.
executionRoleArn (string) --
The execution role ARN of the job run.
releaseLabel (string) --
The release version of Amazon EMR.
configurationOverrides (dict) --
The configuration settings that are used to override default configuration.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) --
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) --
The number of files to keep in container after rotation.
jobDriver (dict) --
Parameters of job driver for the job run.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) --
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
createdAt (datetime) --
The date and time when the job run was created.
createdBy (string) --
The user who created the job run.
finishedAt (datetime) --
The date and time when the job run has finished.
stateDetails (string) --
Additional details of the job run state.
failureReason (string) --
The reasons why the job run has failed.
tags (dict) --
The assigned tags of the job run.
(string) --
(string) --
retryPolicyConfiguration (dict) --
The configuration of the retry policy that the job runs on.
maxAttempts (integer) --
The maximum number of attempts on the job's driver.
retryPolicyExecution (dict) --
The current status of the retry policy executed on the job.
currentAttemptCount (integer) --
The current number of attempts made on the driver of the job.
nextToken (string) --
This output displays the token for the next set of job runs.
{'endpoints': {'authProxyUrl': 'string',
'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}}}}
Lists managed endpoints based on a set of parameters. A managed endpoint is a gateway that connects Amazon EMR Studio to Amazon EMR on EKS so that Amazon EMR Studio can communicate with your virtual cluster.
See also: AWS API Documentation
Request Syntax
client.list_managed_endpoints(
virtualClusterId='string',
createdBefore=datetime(2015, 1, 1),
createdAfter=datetime(2015, 1, 1),
types=[
'string',
],
states=[
'CREATING'|'ACTIVE'|'TERMINATING'|'TERMINATED'|'TERMINATED_WITH_ERRORS',
],
maxResults=123,
nextToken='string'
)
string
[REQUIRED]
The ID of the virtual cluster.
datetime
The date and time before which the endpoints are created.
datetime
The date and time after which the endpoints are created.
list
The types of the managed endpoints.
(string) --
list
The states of the managed endpoints.
(string) --
integer
The maximum number of managed endpoints that can be listed.
string
The token for the next set of managed endpoints to return.
dict
Response Syntax
{
'endpoints': [
{
'id': 'string',
'name': 'string',
'arn': 'string',
'virtualClusterId': 'string',
'type': 'string',
'state': 'CREATING'|'ACTIVE'|'TERMINATING'|'TERMINATED'|'TERMINATED_WITH_ERRORS',
'releaseLabel': 'string',
'executionRoleArn': 'string',
'certificateArn': 'string',
'certificateAuthority': {
'certificateArn': 'string',
'certificateData': 'string'
},
'configurationOverrides': {
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
'serverUrl': 'string',
'authProxyUrl': 'string',
'createdAt': datetime(2015, 1, 1),
'securityGroup': 'string',
'subnetIds': [
'string',
],
'stateDetails': 'string',
'failureReason': 'INTERNAL_ERROR'|'USER_ERROR'|'VALIDATION_ERROR'|'CLUSTER_UNAVAILABLE',
'tags': {
'string': 'string'
}
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
endpoints (list) --
The managed endpoints to be listed.
(dict) --
This entity represents the endpoint that is managed by Amazon EMR on EKS.
id (string) --
The ID of the endpoint.
name (string) --
The name of the endpoint.
arn (string) --
The ARN of the endpoint.
virtualClusterId (string) --
The ID of the endpoint's virtual cluster.
type (string) --
The type of the endpoint.
state (string) --
The state of the endpoint.
releaseLabel (string) --
The EMR release version to be used for the endpoint.
executionRoleArn (string) --
The execution role ARN of the endpoint.
certificateArn (string) --
The certificate ARN of the endpoint. This field is under deprecation and will be removed in future.
certificateAuthority (dict) --
The certificate generated by emr control plane on customer behalf to secure the managed endpoint.
certificateArn (string) --
The ARN of the certificate generated for managed endpoint.
certificateData (string) --
The base64 encoded PEM certificate data generated for managed endpoint.
configurationOverrides (dict) --
The configuration settings that are used to override existing configurations for endpoints.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) --
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) --
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) --
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) --
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) --
The number of files to keep in container after rotation.
serverUrl (string) --
The server URL of the endpoint.
authProxyUrl (string) --
The auth proxy URL of the endpoint.
createdAt (datetime) --
The date and time when the endpoint was created.
securityGroup (string) --
The security group configuration of the endpoint.
subnetIds (list) --
The subnet IDs of the endpoint.
(string) --
stateDetails (string) --
Additional details of the endpoint state.
failureReason (string) --
The reasons why the endpoint has failed.
tags (dict) --
The tags of the endpoint.
(string) --
(string) --
nextToken (string) --
The token for the next set of endpoints to return.
{'virtualClusters': {'sessionEnabled': 'boolean'}}
Lists information about the specified virtual cluster. Virtual cluster is a managed entity on Amazon EMR on EKS. You can create, describe, list and delete virtual clusters. They do not consume any additional resource in your system. A single virtual cluster maps to a single Kubernetes namespace. Given this relationship, you can model virtual clusters the same way you model Kubernetes namespaces to meet your requirements.
See also: AWS API Documentation
Request Syntax
client.list_virtual_clusters(
containerProviderId='string',
containerProviderType='EKS',
createdAfter=datetime(2015, 1, 1),
createdBefore=datetime(2015, 1, 1),
states=[
'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
],
maxResults=123,
nextToken='string',
eksAccessEntryIntegrated=True|False
)
string
The container provider ID of the virtual cluster.
string
The container provider type of the virtual cluster. Amazon EKS is the only supported type as of now.
datetime
The date and time after which the virtual clusters are created.
datetime
The date and time before which the virtual clusters are created.
list
The states of the requested virtual clusters.
(string) --
integer
The maximum number of virtual clusters that can be listed.
string
The token for the next set of virtual clusters to return.
boolean
Optional Boolean that specifies whether the operation should return the virtual clusters that have the access entry integration enabled or disabled. If not specified, the operation returns all applicable virtual clusters.
dict
Response Syntax
{
'virtualClusters': [
{
'id': 'string',
'name': 'string',
'arn': 'string',
'state': 'RUNNING'|'TERMINATING'|'TERMINATED'|'ARRESTED',
'containerProvider': {
'type': 'EKS',
'id': 'string',
'info': {
'eksInfo': {
'namespace': 'string',
'nodeLabel': 'string'
}
}
},
'createdAt': datetime(2015, 1, 1),
'tags': {
'string': 'string'
},
'securityConfigurationId': 'string',
'sessionEnabled': True|False
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
virtualClusters (list) --
This output lists the specified virtual clusters.
(dict) --
This entity describes a virtual cluster. A virtual cluster is a Kubernetes namespace that Amazon EMR is registered with. Amazon EMR uses virtual clusters to run jobs and host endpoints. Multiple virtual clusters can be backed by the same physical cluster. However, each virtual cluster maps to one namespace on an Amazon EKS cluster. Virtual clusters do not create any active resources that contribute to your bill or that require lifecycle management outside the service.
id (string) --
The ID of the virtual cluster.
name (string) --
The name of the virtual cluster.
arn (string) --
The ARN of the virtual cluster.
state (string) --
The state of the virtual cluster.
containerProvider (dict) --
The container provider of the virtual cluster.
type (string) --
The type of the container provider. Amazon EKS is the only supported type as of now.
id (string) --
The ID of the container cluster.
info (dict) --
The information about the container cluster.
eksInfo (dict) --
The information about the Amazon EKS cluster.
namespace (string) --
The namespaces of the Amazon EKS cluster.
nodeLabel (string) --
The nodeLabel of the nodes where the resources of this virtual cluster can get scheduled. It requires relevant scaling and policy engine addons.
createdAt (datetime) --
The date and time when the virtual cluster is created.
tags (dict) --
The assigned tags of the virtual cluster.
(string) --
(string) --
securityConfigurationId (string) --
The ID of the security configuration.
sessionEnabled (boolean) --
Indicates whether the virtual cluster has session support enabled.
nextToken (string) --
This output displays the token for the next set of virtual clusters.
{'configurationOverrides': {'monitoringConfiguration': {'s3MonitoringConfiguration': {'encryptionKeyArn': 'string'}}}}
Starts a job run. A job run is a unit of work, such as a Spark jar, PySpark script, or SparkSQL query, that you submit to Amazon EMR on EKS.
See also: AWS API Documentation
Request Syntax
client.start_job_run(
name='string',
virtualClusterId='string',
clientToken='string',
executionRoleArn='string',
releaseLabel='string',
jobDriver={
'sparkSubmitJobDriver': {
'entryPoint': 'string',
'entryPointArguments': [
'string',
],
'sparkSubmitParameters': 'string'
},
'sparkSqlJobDriver': {
'entryPoint': 'string',
'sparkSqlParameters': 'string'
}
},
configurationOverrides={
'applicationConfiguration': [
{
'classification': 'string',
'properties': {
'string': 'string'
},
'configurations': {'... recursive ...'}
},
],
'monitoringConfiguration': {
'managedLogs': {
'allowAWSToRetainLogs': 'ENABLED'|'DISABLED',
'encryptionKeyArn': 'string'
},
'persistentAppUI': 'ENABLED'|'DISABLED',
'cloudWatchMonitoringConfiguration': {
'logGroupName': 'string',
'logStreamNamePrefix': 'string'
},
's3MonitoringConfiguration': {
'logUri': 'string',
'encryptionKeyArn': 'string'
},
'containerLogRotationConfiguration': {
'rotationSize': 'string',
'maxFilesToKeep': 123
}
}
},
tags={
'string': 'string'
},
jobTemplateId='string',
jobTemplateParameters={
'string': 'string'
},
retryPolicyConfiguration={
'maxAttempts': 123
}
)
string
The name of the job run.
string
[REQUIRED]
The virtual cluster ID for which the job run request is submitted.
string
[REQUIRED]
The client idempotency token of the job run request.
This field is autopopulated if not provided.
string
The execution role ARN for the job run.
string
The Amazon EMR release version to use for the job run.
dict
The job driver for the job run.
sparkSubmitJobDriver (dict) --
The job driver parameters specified for spark submit.
entryPoint (string) -- [REQUIRED]
The entry point of job application.
entryPointArguments (list) --
The arguments for job application.
(string) --
sparkSubmitParameters (string) --
The Spark submit parameters that are used for job runs.
sparkSqlJobDriver (dict) --
The job driver for job type.
entryPoint (string) --
The SQL file to be executed.
sparkSqlParameters (string) --
The Spark parameters to be included in the Spark SQL command.
dict
The configuration overrides for the job run.
applicationConfiguration (list) --
The configurations for the application running by the job run.
(dict) --
A configuration specification to be used when provisioning virtual clusters, which can include configurations for applications and software bundled with Amazon EMR on EKS. A configuration consists of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file.
classification (string) -- [REQUIRED]
The classification within a configuration.
properties (dict) --
A set of properties specified within a configuration classification.
(string) --
(string) --
configurations (list) --
A list of additional configurations to apply within a configuration object.
monitoringConfiguration (dict) --
The configurations for monitoring.
managedLogs (dict) --
The entity that controls configuration for managed logs.
allowAWSToRetainLogs (string) --
Determines whether Amazon Web Services can retain logs.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
persistentAppUI (string) --
Monitoring configurations for the persistent application UI.
cloudWatchMonitoringConfiguration (dict) --
Monitoring configurations for CloudWatch.
logGroupName (string) -- [REQUIRED]
The name of the log group for log publishing.
logStreamNamePrefix (string) --
The specified name prefix for log streams.
s3MonitoringConfiguration (dict) --
Amazon S3 configuration for monitoring log publishing.
logUri (string) -- [REQUIRED]
Amazon S3 destination URI for log publishing.
encryptionKeyArn (string) --
The Amazon resource name (ARN) of the encryption key for logs.
containerLogRotationConfiguration (dict) --
Enable or disable container log rotation.
rotationSize (string) -- [REQUIRED]
The file size at which to rotate logs. Minimum of 2KB, Maximum of 2GB.
maxFilesToKeep (integer) -- [REQUIRED]
The number of files to keep in container after rotation.
dict
The tags assigned to job runs.
(string) --
(string) --
string
The job template ID to be used to start the job run.
dict
The values of job template parameters to start a job run.
(string) --
(string) --
dict
The retry policy configuration for the job run.
maxAttempts (integer) -- [REQUIRED]
The maximum number of attempts on the job's driver.
dict
Response Syntax
{
'id': 'string',
'name': 'string',
'arn': 'string',
'virtualClusterId': 'string'
}
Response Structure
(dict) --
id (string) --
This output displays the started job run ID.
name (string) --
This output displays the name of the started job run.
arn (string) --
This output lists the ARN of job run.
virtualClusterId (string) --
This output displays the virtual cluster ID for which the job run was submitted.