AWS API Changes

2020/08/28 - Amazon Elastic MapReduce - 4 new api methods

Changes Amazon EMR adds support for ICMP, port -1, in Block Public Access Exceptions and API access for EMR Notebooks execution. You can now non-interactively execute EMR Notebooks and pass input parameters.

ListNotebookExecutions (new)

Link ¶

Provides summaries of all notebook executions. You can filter the list based on multiple criteria such as status, time range, and editor id. Returns a maximum of 50 notebook executions and a marker to track the paging of a longer notebook execution list across multiple ListNotebookExecution calls.

See also: AWS API Documentation

Request Syntax

client.list_notebook_executions(
    EditorId='string',
    Status='START_PENDING'|'STARTING'|'RUNNING'|'FINISHING'|'FINISHED'|'FAILING'|'FAILED'|'STOP_PENDING'|'STOPPING'|'STOPPED',
    From=datetime(2015, 1, 1),
    To=datetime(2015, 1, 1),
    Marker='string'
)

type EditorId

string

param EditorId

The unique ID of the editor associated with the notebook execution.

type Status

string

param Status

The status filter for listing notebook executions.

START_PENDING indicates that the cluster has received the execution request but execution has not begun.
STARTING indicates that the execution is starting on the cluster.
RUNNING indicates that the execution is being processed by the cluster.
FINISHING indicates that execution processing is in the final stages.
FINISHED indicates that the execution has completed without error.
FAILING indicates that the execution is failing and will not finish successfully.
FAILED indicates that the execution failed.
STOP_PENDING indicates that the cluster has received a StopNotebookExecution request and the stop is pending.
STOPPING indicates that the cluster is in the process of stopping the execution as a result of a StopNotebookExecution request.
STOPPED indicates that the execution stopped because of a StopNotebookExecution request.

type From

datetime

param From

The beginning of time range filter for listing notebook executions. The default is the timestamp of 30 days ago.

type To

datetime

param To

The end of time range filter for listing notebook executions. The default is the current timestamp.

type Marker

string

param Marker

The pagination token, returned by a previous ListNotebookExecutions call, that indicates the start of the list for this ListNotebookExecutions call.

rtype

dict

returns

Response Syntax

{
    'NotebookExecutions': [
        {
            'NotebookExecutionId': 'string',
            'EditorId': 'string',
            'NotebookExecutionName': 'string',
            'Status': 'START_PENDING'|'STARTING'|'RUNNING'|'FINISHING'|'FINISHED'|'FAILING'|'FAILED'|'STOP_PENDING'|'STOPPING'|'STOPPED',
            'StartTime': datetime(2015, 1, 1),
            'EndTime': datetime(2015, 1, 1)
        },
    ],
    'Marker': 'string'
}

Response Structure

(dict) --
- NotebookExecutions (list) --
  
  A list of notebook executions.
  - (dict) --
    - NotebookExecutionId (string) --
      
      The unique identifier of the notebook execution.
    - EditorId (string) --
      
      The unique identifier of the editor associated with the notebook execution.
    - NotebookExecutionName (string) --
      
      The name of the notebook execution.
    - Status (string) --
      
      The status of the notebook execution.
      - START_PENDING indicates that the cluster has received the execution request but execution has not begun.
      - STARTING indicates that the execution is starting on the cluster.
      - RUNNING indicates that the execution is being processed by the cluster.
      - FINISHING indicates that execution processing is in the final stages.
      - FINISHED indicates that the execution has completed without error.
      - FAILING indicates that the execution is failing and will not finish successfully.
      - FAILED indicates that the execution failed.
      - STOP_PENDING indicates that the cluster has received a StopNotebookExecution request and the stop is pending.
      - STOPPING indicates that the cluster is in the process of stopping the execution as a result of a StopNotebookExecution request.
      - STOPPED indicates that the execution stopped because of a StopNotebookExecution request.
    - StartTime (datetime) --
      
      The timestamp when notebook execution started.
    - EndTime (datetime) --
      
      The timestamp when notebook execution started.
- Marker (string) --
  
  A pagination token that a subsequent ListNotebookExecutions can use to determine the next set of results to retrieve.

StopNotebookExecution (new)

Link ¶

Stops a notebook execution.

See also: AWS API Documentation

Request Syntax

client.stop_notebook_execution(
    NotebookExecutionId='string'
)

type NotebookExecutionId

string

param NotebookExecutionId

[REQUIRED]

The unique identifier of the notebook execution.

returns

None

DescribeNotebookExecution (new)

Link ¶

Provides details of a notebook execution.

See also: AWS API Documentation

Request Syntax

client.describe_notebook_execution(
    NotebookExecutionId='string'
)

type NotebookExecutionId

string

param NotebookExecutionId

[REQUIRED]

The unique identifier of the notebook execution.

rtype

dict

returns

Response Syntax

{
    'NotebookExecution': {
        'NotebookExecutionId': 'string',
        'EditorId': 'string',
        'ExecutionEngine': {
            'Id': 'string',
            'Type': 'EMR',
            'MasterInstanceSecurityGroupId': 'string'
        },
        'NotebookExecutionName': 'string',
        'NotebookParams': 'string',
        'Status': 'START_PENDING'|'STARTING'|'RUNNING'|'FINISHING'|'FINISHED'|'FAILING'|'FAILED'|'STOP_PENDING'|'STOPPING'|'STOPPED',
        'StartTime': datetime(2015, 1, 1),
        'EndTime': datetime(2015, 1, 1),
        'Arn': 'string',
        'OutputNotebookURI': 'string',
        'LastStateChangeReason': 'string',
        'NotebookInstanceSecurityGroupId': 'string',
        'Tags': [
            {
                'Key': 'string',
                'Value': 'string'
            },
        ]
    }
}

Response Structure

(dict) --
- NotebookExecution (dict) --
  
  Properties of the notebook execution.
  - NotebookExecutionId (string) --
    
    The unique identifier of a notebook execution.
  - EditorId (string) --
    
    The unique identifier of the EMR Notebook that is used for the notebook execution.
  - ExecutionEngine (dict) --
    
    The execution engine, such as an EMR cluster, used to run the EMR notebook and perform the notebook execution.
    - Id (string) --
      
      The unique identifier of the execution engine. For an EMR cluster, this is the cluster ID.
    - Type (string) --
      
      The type of execution engine. A value of EMR specifies an EMR cluster.
    - MasterInstanceSecurityGroupId (string) --
      
      An optional unique ID of an EC2 security group to associate with the master instance of the EMR cluster for this notebook execution. For more information see Specifying EC2 Security Groups for EMR Notebooks in the EMR Management Guide .
  - NotebookExecutionName (string) --
    
    A name for the notebook execution.
  - NotebookParams (string) --
    
    Input parameters in JSON format passed to the EMR Notebook at runtime for execution.
  - Status (string) --
    
    The status of the notebook execution.
    - START_PENDING indicates that the cluster has received the execution request but execution has not begun.
    - STARTING indicates that the execution is starting on the cluster.
    - RUNNING indicates that the execution is being processed by the cluster.
    - FINISHING indicates that execution processing is in the final stages.
    - FINISHED indicates that the execution has completed without error.
    - FAILING indicates that the execution is failing and will not finish successfully.
    - FAILED indicates that the execution failed.
    - STOP_PENDING indicates that the cluster has received a StopNotebookExecution request and the stop is pending.
    - STOPPING indicates that the cluster is in the process of stopping the execution as a result of a StopNotebookExecution request.
    - STOPPED indicates that the execution stopped because of a StopNotebookExecution request.
  - StartTime (datetime) --
    
    The timestamp when notebook execution started.
  - EndTime (datetime) --
    
    The timestamp when notebook execution ended.
  - Arn (string) --
    
    The Amazon Resource Name (ARN) of the notebook execution.
  - OutputNotebookURI (string) --
    
    The location of the notebook execution's output file in Amazon S3.
  - LastStateChangeReason (string) --
    
    The reason for the latest status change of the notebook execution.
  - NotebookInstanceSecurityGroupId (string) --
    
    The unique identifier of the EC2 security group associated with the EMR Notebook instance. For more information see Specifying EC2 Security Groups for EMR Notebooks in the EMR Management Guide .
  - Tags (list) --
    
    A list of tags associated with a notebook execution. Tags are user-defined key value pairs that consist of a required key string with a maximum of 128 characters and an optional value string with a maximum of 256 characters.
    - (dict) --
      
      A key/value pair containing user-defined metadata that you can associate with an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
      - Key (string) --
        
        A user-defined key, which is the minimum required information for a valid tag. For more information, see Tag.
      - Value (string) --
        
        A user-defined value, which is optional in a tag. For more information, see Tag Clusters.

StartNotebookExecution (new)

Link ¶

Starts a notebook execution.

See also: AWS API Documentation

Request Syntax

client.start_notebook_execution(
    EditorId='string',
    RelativePath='string',
    NotebookExecutionName='string',
    NotebookParams='string',
    ExecutionEngine={
        'Id': 'string',
        'Type': 'EMR',
        'MasterInstanceSecurityGroupId': 'string'
    },
    ServiceRole='string',
    NotebookInstanceSecurityGroupId='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)

type EditorId

string

param EditorId

[REQUIRED]

The unique identifier of the EMR Notebook to use for notebook execution.

type RelativePath

string

param RelativePath

[REQUIRED]

The path and file name of the notebook file for this execution, relative to the path specified for the EMR Notebook. For example, if you specify a path of s3://MyBucket/MyNotebooks when you create an EMR Notebook for a notebook with an ID of e-ABCDEFGHIJK1234567890ABCD (the EditorID of this request), and you specify a RelativePath of my_notebook_executions/notebook_execution.ipynb , the location of the file for the notebook execution is s3://MyBucket/MyNotebooks/e-ABCDEFGHIJK1234567890ABCD/my_notebook_executions/notebook_execution.ipynb .

type NotebookExecutionName

string

param NotebookExecutionName

An optional name for the notebook execution.

type NotebookParams

string

param NotebookParams

Input parameters in JSON format passed to the EMR Notebook at runtime for execution.

type ExecutionEngine

dict

param ExecutionEngine

[REQUIRED]

Specifies the execution engine (cluster) that runs the notebook execution.

Id (string) -- [REQUIRED]

The unique identifier of the execution engine. For an EMR cluster, this is the cluster ID.
Type (string) --

The type of execution engine. A value of EMR specifies an EMR cluster.
MasterInstanceSecurityGroupId (string) --

An optional unique ID of an EC2 security group to associate with the master instance of the EMR cluster for this notebook execution. For more information see Specifying EC2 Security Groups for EMR Notebooks in the EMR Management Guide .

type ServiceRole

string

param ServiceRole

[REQUIRED]

The name or ARN of the IAM role that is used as the service role for Amazon EMR (the EMR role) for the notebook execution.

type NotebookInstanceSecurityGroupId

string

param NotebookInstanceSecurityGroupId

The unique identifier of the Amazon EC2 security group to associate with the EMR Notebook for this notebook execution.

type Tags

list

param Tags

A list of tags associated with a notebook execution. Tags are user-defined key value pairs that consist of a required key string with a maximum of 128 characters and an optional value string with a maximum of 256 characters.

(dict) --

A key/value pair containing user-defined metadata that you can associate with an Amazon EMR resource. Tags make it easier to associate clusters in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see Tag Clusters.
- Key (string) --
  
  A user-defined key, which is the minimum required information for a valid tag. For more information, see Tag.
- Value (string) --
  
  A user-defined value, which is optional in a tag. For more information, see Tag Clusters.

rtype

dict

returns

Response Syntax

{
    'NotebookExecutionId': 'string'
}

Response Structure

(dict) --
- NotebookExecutionId (string) --
  
  The unique identifier of the notebook execution.