AWS API Changes

2017/10/24 - AWS Glue - 1 new4 updated api methods

Changes Update glue client to latest version

BatchStopJobRun (new)

Link ¶

Stops a batch of job runs for a given job.

Request Syntax

client.batch_stop_job_run(
    JobName='string',
    JobRunIds=[
        'string',
    ]
)

type JobName:

string

param JobName:

[REQUIRED]

The name of the job whose job runs are to be stopped.

type JobRunIds:

list

param JobRunIds:

[REQUIRED]

A list of job run Ids of the given job to be stopped.

(string) --

rtype:

dict

returns:

Response Syntax

{
    'SuccessfulSubmissions': [
        {
            'JobName': 'string',
            'JobRunId': 'string'
        },
    ],
    'Errors': [
        {
            'JobName': 'string',
            'JobRunId': 'string',
            'ErrorDetail': {
                'ErrorCode': 'string',
                'ErrorMessage': 'string'
            }
        },
    ]
}

Response Structure

(dict) --
- SuccessfulSubmissions (list) --
  
  A list of job runs which are successfully submitted for stopping.
  - (dict) --
    
    Details about the job run which is submitted successfully for stopping.
    - JobName (string) --
      
      The name of the job.
    - JobRunId (string) --
      
      The job run Id.
- Errors (list) --
  
  A list containing the job run Ids and details of the error that occurred for each job run while submitting to stop.
  - (dict) --
    
    Details about the job run and the error that occurred while trying to submit it for stopping.
    - JobName (string) --
      
      The name of the job.
    - JobRunId (string) --
      
      The job run Id.
    - ErrorDetail (dict) --
      
      The details of the error that occurred.
      - ErrorCode (string) --
        
        The code associated with this error.
      - ErrorMessage (string) --
        
        A message describing the error.

CreateDevEndpoint (updated)

Link ¶
Changes (response)

{'ZeppelinRemoteSparkInterpreterPort': 'integer'}

Creates a new DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.create_dev_endpoint(
    EndpointName='string',
    RoleArn='string',
    SecurityGroupIds=[
        'string',
    ],
    SubnetId='string',
    PublicKey='string',
    NumberOfNodes=123,
    ExtraPythonLibsS3Path='string',
    ExtraJarsS3Path='string'
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

The name to be assigned to the new DevEndpoint.

type RoleArn:

string

param RoleArn:

[REQUIRED]

The IAM role for the DevEndpoint.

type SecurityGroupIds:

list

param SecurityGroupIds:

Security group IDs for the security groups to be used by the new DevEndpoint.

(string) --

type SubnetId:

string

param SubnetId:

The subnet ID for the new DevEndpoint to use.

type PublicKey:

string

param PublicKey:

[REQUIRED]

The public key to use for authentication.

type NumberOfNodes:

integer

param NumberOfNodes:

The number of AWS Glue Data Processing Units (DPUs) to allocate to this DevEndpoint.

type ExtraPythonLibsS3Path:

string

param ExtraPythonLibsS3Path:

Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.

Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.

type ExtraJarsS3Path:

string

param ExtraJarsS3Path:

Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.

rtype:

dict

returns:

Response Syntax

{
    'EndpointName': 'string',
    'Status': 'string',
    'SecurityGroupIds': [
        'string',
    ],
    'SubnetId': 'string',
    'RoleArn': 'string',
    'YarnEndpointAddress': 'string',
    'ZeppelinRemoteSparkInterpreterPort': 123,
    'NumberOfNodes': 123,
    'AvailabilityZone': 'string',
    'VpcId': 'string',
    'ExtraPythonLibsS3Path': 'string',
    'ExtraJarsS3Path': 'string',
    'FailureReason': 'string',
    'CreatedTimestamp': datetime(2015, 1, 1)
}

Response Structure

(dict) --
- EndpointName (string) --
  
  The name assigned to the new DevEndpoint.
- Status (string) --
  
  The current status of the new DevEndpoint.
- SecurityGroupIds (list) --
  
  The security groups assigned to the new DevEndpoint.
  - (string) --
- SubnetId (string) --
  
  The subnet ID assigned to the new DevEndpoint.
- RoleArn (string) --
  
  The AWS ARN of the role assigned to the new DevEndpoint.
- YarnEndpointAddress (string) --
  
  The address of the YARN endpoint used by this DevEndpoint.
- ZeppelinRemoteSparkInterpreterPort (integer) --
  
  The Apache Zeppelin port for the remote Apache Spark interpreter.
- NumberOfNodes (integer) --
  
  The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
- AvailabilityZone (string) --
  
  The AWS availability zone where this DevEndpoint is located.
- VpcId (string) --
  
  The ID of the VPC used by this DevEndpoint.
- ExtraPythonLibsS3Path (string) --
  
  Path(s) to one or more Python libraries in an S3 bucket that will be loaded in your DevEndpoint.
- ExtraJarsS3Path (string) --
  
  Path to one or more Java Jars in an S3 bucket that will be loaded in your DevEndpoint.
- FailureReason (string) --
  
  The reason for a current failure in this DevEndpoint.
- CreatedTimestamp (datetime) --
  
  The point in time at which this DevEndpoint was created.

GetDevEndpoint (updated)

Link ¶
Changes (response)

{'DevEndpoint': {'ZeppelinRemoteSparkInterpreterPort': 'integer'}}

Retrieves information about a specified DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.get_dev_endpoint(
    EndpointName='string'
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

Name of the DevEndpoint for which to retrieve information.

rtype:

dict

returns:

Response Syntax

{
    'DevEndpoint': {
        'EndpointName': 'string',
        'RoleArn': 'string',
        'SecurityGroupIds': [
            'string',
        ],
        'SubnetId': 'string',
        'YarnEndpointAddress': 'string',
        'ZeppelinRemoteSparkInterpreterPort': 123,
        'PublicAddress': 'string',
        'Status': 'string',
        'NumberOfNodes': 123,
        'AvailabilityZone': 'string',
        'VpcId': 'string',
        'ExtraPythonLibsS3Path': 'string',
        'ExtraJarsS3Path': 'string',
        'FailureReason': 'string',
        'LastUpdateStatus': 'string',
        'CreatedTimestamp': datetime(2015, 1, 1),
        'LastModifiedTimestamp': datetime(2015, 1, 1),
        'PublicKey': 'string'
    }
}

Response Structure

(dict) --
- DevEndpoint (dict) --
  
  A DevEndpoint definition.
  - EndpointName (string) --
    
    The name of the DevEndpoint.
  - RoleArn (string) --
    
    The AWS ARN of the IAM role used in this DevEndpoint.
  - SecurityGroupIds (list) --
    
    A list of security group identifiers used in this DevEndpoint.
    - (string) --
  - SubnetId (string) --
    
    The subnet ID for this DevEndpoint.
  - YarnEndpointAddress (string) --
    
    The YARN endpoint address used by this DevEndpoint.
  - ZeppelinRemoteSparkInterpreterPort (integer) --
    
    The Apache Zeppelin port for the remote Apache Spark interpreter.
  - PublicAddress (string) --
    
    The public address used by this DevEndpoint.
  - Status (string) --
    
    The current status of this DevEndpoint.
  - NumberOfNodes (integer) --
    
    The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
  - AvailabilityZone (string) --
    
    The AWS availability zone where this DevEndpoint is located.
  - VpcId (string) --
    
    The ID of the virtual private cloud (VPC) used by this DevEndpoint.
  - ExtraPythonLibsS3Path (string) --
    
    Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.
    
    Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
  - ExtraJarsS3Path (string) --
    
    Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.
    
    Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.
  - FailureReason (string) --
    
    The reason for a current failure in this DevEndpoint.
  - LastUpdateStatus (string) --
    
    The status of the last update.
  - CreatedTimestamp (datetime) --
    
    The point in time at which this DevEndpoint was created.
  - LastModifiedTimestamp (datetime) --
    
    The point in time at which this DevEndpoint was last modified.
  - PublicKey (string) --
    
    The public key to be used by this DevEndpoint for authentication.

GetDevEndpoints (updated)

Link ¶
Changes (response)

{'DevEndpoints': {'ZeppelinRemoteSparkInterpreterPort': 'integer'}}

Retrieves all the DevEndpoints in this AWS account.

See also: AWS API Documentation

Request Syntax

client.get_dev_endpoints(
    MaxResults=123,
    NextToken='string'
)

type MaxResults:

integer

param MaxResults:

The maximum size of information to return.

type NextToken:

string

param NextToken:

A continuation token, if this is a continuation call.

rtype:

dict

returns:

Response Syntax

{
    'DevEndpoints': [
        {
            'EndpointName': 'string',
            'RoleArn': 'string',
            'SecurityGroupIds': [
                'string',
            ],
            'SubnetId': 'string',
            'YarnEndpointAddress': 'string',
            'ZeppelinRemoteSparkInterpreterPort': 123,
            'PublicAddress': 'string',
            'Status': 'string',
            'NumberOfNodes': 123,
            'AvailabilityZone': 'string',
            'VpcId': 'string',
            'ExtraPythonLibsS3Path': 'string',
            'ExtraJarsS3Path': 'string',
            'FailureReason': 'string',
            'LastUpdateStatus': 'string',
            'CreatedTimestamp': datetime(2015, 1, 1),
            'LastModifiedTimestamp': datetime(2015, 1, 1),
            'PublicKey': 'string'
        },
    ],
    'NextToken': 'string'
}

Response Structure

(dict) --
- DevEndpoints (list) --
  
  A list of DevEndpoint definitions.
  - (dict) --
    
    A development endpoint where a developer can remotely debug ETL scripts.
    - EndpointName (string) --
      
      The name of the DevEndpoint.
    - RoleArn (string) --
      
      The AWS ARN of the IAM role used in this DevEndpoint.
    - SecurityGroupIds (list) --
      
      A list of security group identifiers used in this DevEndpoint.
      - (string) --
    - SubnetId (string) --
      
      The subnet ID for this DevEndpoint.
    - YarnEndpointAddress (string) --
      
      The YARN endpoint address used by this DevEndpoint.
    - ZeppelinRemoteSparkInterpreterPort (integer) --
      
      The Apache Zeppelin port for the remote Apache Spark interpreter.
    - PublicAddress (string) --
      
      The public address used by this DevEndpoint.
    - Status (string) --
      
      The current status of this DevEndpoint.
    - NumberOfNodes (integer) --
      
      The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
    - AvailabilityZone (string) --
      
      The AWS availability zone where this DevEndpoint is located.
    - VpcId (string) --
      
      The ID of the virtual private cloud (VPC) used by this DevEndpoint.
    - ExtraPythonLibsS3Path (string) --
      
      Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.
      
      Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
    - ExtraJarsS3Path (string) --
      
      Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.
      
      Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.
    - FailureReason (string) --
      
      The reason for a current failure in this DevEndpoint.
    - LastUpdateStatus (string) --
      
      The status of the last update.
    - CreatedTimestamp (datetime) --
      
      The point in time at which this DevEndpoint was created.
    - LastModifiedTimestamp (datetime) --
      
      The point in time at which this DevEndpoint was last modified.
    - PublicKey (string) --
      
      The public key to be used by this DevEndpoint for authentication.
- NextToken (string) --
  
  A continuation token, if not all DevEndpoint definitions have yet been returned.

UpdateDevEndpoint (updated)

Link ¶
Changes (request)

{'UpdateEtlLibraries': 'boolean'}

Updates a specified DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.update_dev_endpoint(
    EndpointName='string',
    PublicKey='string',
    CustomLibraries={
        'ExtraPythonLibsS3Path': 'string',
        'ExtraJarsS3Path': 'string'
    },
    UpdateEtlLibraries=True|False
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the DevEndpoint to be updated.

type PublicKey:

string

param PublicKey:

The public key for the DevEndpoint to use.

type CustomLibraries:

dict

param CustomLibraries:

Custom Python or Java libraries to be loaded in the DevEndpoint.

ExtraPythonLibsS3Path (string) --

Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.

Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
ExtraJarsS3Path (string) --

Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.

Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.

type UpdateEtlLibraries:

boolean

param UpdateEtlLibraries:

True if the list of custom libraries to be loaded in the development endpoint needs to be updated, or False otherwise.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --