AWS API Changes

2018/07/30 - AWS Glue - 4 updated api methods

Changes Update glue client to latest version

CreateDevEndpoint (updated)

Link ¶
Changes (request)

{'PublicKeys': ['string']}

Creates a new DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.create_dev_endpoint(
    EndpointName='string',
    RoleArn='string',
    SecurityGroupIds=[
        'string',
    ],
    SubnetId='string',
    PublicKey='string',
    PublicKeys=[
        'string',
    ],
    NumberOfNodes=123,
    ExtraPythonLibsS3Path='string',
    ExtraJarsS3Path='string'
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

The name to be assigned to the new DevEndpoint.

type RoleArn:

string

param RoleArn:

[REQUIRED]

The IAM role for the DevEndpoint.

type SecurityGroupIds:

list

param SecurityGroupIds:

Security group IDs for the security groups to be used by the new DevEndpoint.

(string) --

type SubnetId:

string

param SubnetId:

The subnet ID for the new DevEndpoint to use.

type PublicKey:

string

param PublicKey:

The public key to be used by this DevEndpoint for authentication. This attribute is provided for backward compatibility, as the recommended attribute to use is public keys.

type PublicKeys:

list

param PublicKeys:

A list of public keys to be used by the DevEndpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.

(string) --

type NumberOfNodes:

integer

param NumberOfNodes:

The number of AWS Glue Data Processing Units (DPUs) to allocate to this DevEndpoint.

type ExtraPythonLibsS3Path:

string

param ExtraPythonLibsS3Path:

Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.

Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.

type ExtraJarsS3Path:

string

param ExtraJarsS3Path:

Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.

rtype:

dict

returns:

Response Syntax

{
    'EndpointName': 'string',
    'Status': 'string',
    'SecurityGroupIds': [
        'string',
    ],
    'SubnetId': 'string',
    'RoleArn': 'string',
    'YarnEndpointAddress': 'string',
    'ZeppelinRemoteSparkInterpreterPort': 123,
    'NumberOfNodes': 123,
    'AvailabilityZone': 'string',
    'VpcId': 'string',
    'ExtraPythonLibsS3Path': 'string',
    'ExtraJarsS3Path': 'string',
    'FailureReason': 'string',
    'CreatedTimestamp': datetime(2015, 1, 1)
}

Response Structure

(dict) --
- EndpointName (string) --
  
  The name assigned to the new DevEndpoint.
- Status (string) --
  
  The current status of the new DevEndpoint.
- SecurityGroupIds (list) --
  
  The security groups assigned to the new DevEndpoint.
  - (string) --
- SubnetId (string) --
  
  The subnet ID assigned to the new DevEndpoint.
- RoleArn (string) --
  
  The AWS ARN of the role assigned to the new DevEndpoint.
- YarnEndpointAddress (string) --
  
  The address of the YARN endpoint used by this DevEndpoint.
- ZeppelinRemoteSparkInterpreterPort (integer) --
  
  The Apache Zeppelin port for the remote Apache Spark interpreter.
- NumberOfNodes (integer) --
  
  The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
- AvailabilityZone (string) --
  
  The AWS availability zone where this DevEndpoint is located.
- VpcId (string) --
  
  The ID of the VPC used by this DevEndpoint.
- ExtraPythonLibsS3Path (string) --
  
  Path(s) to one or more Python libraries in an S3 bucket that will be loaded in your DevEndpoint.
- ExtraJarsS3Path (string) --
  
  Path to one or more Java Jars in an S3 bucket that will be loaded in your DevEndpoint.
- FailureReason (string) --
  
  The reason for a current failure in this DevEndpoint.
- CreatedTimestamp (datetime) --
  
  The point in time at which this DevEndpoint was created.

GetDevEndpoint (updated)

Link ¶
Changes (response)

{'DevEndpoint': {'PublicKeys': ['string']}}

Retrieves information about a specified DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.get_dev_endpoint(
    EndpointName='string'
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

Name of the DevEndpoint for which to retrieve information.

rtype:

dict

returns:

Response Syntax

{
    'DevEndpoint': {
        'EndpointName': 'string',
        'RoleArn': 'string',
        'SecurityGroupIds': [
            'string',
        ],
        'SubnetId': 'string',
        'YarnEndpointAddress': 'string',
        'PrivateAddress': 'string',
        'ZeppelinRemoteSparkInterpreterPort': 123,
        'PublicAddress': 'string',
        'Status': 'string',
        'NumberOfNodes': 123,
        'AvailabilityZone': 'string',
        'VpcId': 'string',
        'ExtraPythonLibsS3Path': 'string',
        'ExtraJarsS3Path': 'string',
        'FailureReason': 'string',
        'LastUpdateStatus': 'string',
        'CreatedTimestamp': datetime(2015, 1, 1),
        'LastModifiedTimestamp': datetime(2015, 1, 1),
        'PublicKey': 'string',
        'PublicKeys': [
            'string',
        ]
    }
}

Response Structure

(dict) --
- DevEndpoint (dict) --
  
  A DevEndpoint definition.
  - EndpointName (string) --
    
    The name of the DevEndpoint.
  - RoleArn (string) --
    
    The AWS ARN of the IAM role used in this DevEndpoint.
  - SecurityGroupIds (list) --
    
    A list of security group identifiers used in this DevEndpoint.
    - (string) --
  - SubnetId (string) --
    
    The subnet ID for this DevEndpoint.
  - YarnEndpointAddress (string) --
    
    The YARN endpoint address used by this DevEndpoint.
  - PrivateAddress (string) --
    
    A private DNS to access the DevEndpoint within a VPC, if the DevEndpoint is created within one.
  - ZeppelinRemoteSparkInterpreterPort (integer) --
    
    The Apache Zeppelin port for the remote Apache Spark interpreter.
  - PublicAddress (string) --
    
    The public VPC address used by this DevEndpoint.
  - Status (string) --
    
    The current status of this DevEndpoint.
  - NumberOfNodes (integer) --
    
    The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
  - AvailabilityZone (string) --
    
    The AWS availability zone where this DevEndpoint is located.
  - VpcId (string) --
    
    The ID of the virtual private cloud (VPC) used by this DevEndpoint.
  - ExtraPythonLibsS3Path (string) --
    
    Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.
    
    Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
  - ExtraJarsS3Path (string) --
    
    Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.
    
    Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.
  - FailureReason (string) --
    
    The reason for a current failure in this DevEndpoint.
  - LastUpdateStatus (string) --
    
    The status of the last update.
  - CreatedTimestamp (datetime) --
    
    The point in time at which this DevEndpoint was created.
  - LastModifiedTimestamp (datetime) --
    
    The point in time at which this DevEndpoint was last modified.
  - PublicKey (string) --
    
    The public key to be used by this DevEndpoint for authentication. This attribute is provided for backward compatibility, as the recommended attribute to use is public keys.
  - PublicKeys (list) --
    
    A list of public keys to be used by the DevEndpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.
    
    Note
    
    If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys: call the UpdateDevEndpoint API with the public key content in the deletePublicKeys attribute, and the list of new keys in the addPublicKeys attribute.
    - (string) --

GetDevEndpoints (updated)

Link ¶
Changes (response)

{'DevEndpoints': {'PublicKeys': ['string']}}

Retrieves all the DevEndpoints in this AWS account.

See also: AWS API Documentation

Request Syntax

client.get_dev_endpoints(
    MaxResults=123,
    NextToken='string'
)

type MaxResults:

integer

param MaxResults:

The maximum size of information to return.

type NextToken:

string

param NextToken:

A continuation token, if this is a continuation call.

rtype:

dict

returns:

Response Syntax

{
    'DevEndpoints': [
        {
            'EndpointName': 'string',
            'RoleArn': 'string',
            'SecurityGroupIds': [
                'string',
            ],
            'SubnetId': 'string',
            'YarnEndpointAddress': 'string',
            'PrivateAddress': 'string',
            'ZeppelinRemoteSparkInterpreterPort': 123,
            'PublicAddress': 'string',
            'Status': 'string',
            'NumberOfNodes': 123,
            'AvailabilityZone': 'string',
            'VpcId': 'string',
            'ExtraPythonLibsS3Path': 'string',
            'ExtraJarsS3Path': 'string',
            'FailureReason': 'string',
            'LastUpdateStatus': 'string',
            'CreatedTimestamp': datetime(2015, 1, 1),
            'LastModifiedTimestamp': datetime(2015, 1, 1),
            'PublicKey': 'string',
            'PublicKeys': [
                'string',
            ]
        },
    ],
    'NextToken': 'string'
}

Response Structure

(dict) --
- DevEndpoints (list) --
  
  A list of DevEndpoint definitions.
  - (dict) --
    
    A development endpoint where a developer can remotely debug ETL scripts.
    - EndpointName (string) --
      
      The name of the DevEndpoint.
    - RoleArn (string) --
      
      The AWS ARN of the IAM role used in this DevEndpoint.
    - SecurityGroupIds (list) --
      
      A list of security group identifiers used in this DevEndpoint.
      - (string) --
    - SubnetId (string) --
      
      The subnet ID for this DevEndpoint.
    - YarnEndpointAddress (string) --
      
      The YARN endpoint address used by this DevEndpoint.
    - PrivateAddress (string) --
      
      A private DNS to access the DevEndpoint within a VPC, if the DevEndpoint is created within one.
    - ZeppelinRemoteSparkInterpreterPort (integer) --
      
      The Apache Zeppelin port for the remote Apache Spark interpreter.
    - PublicAddress (string) --
      
      The public VPC address used by this DevEndpoint.
    - Status (string) --
      
      The current status of this DevEndpoint.
    - NumberOfNodes (integer) --
      
      The number of AWS Glue Data Processing Units (DPUs) allocated to this DevEndpoint.
    - AvailabilityZone (string) --
      
      The AWS availability zone where this DevEndpoint is located.
    - VpcId (string) --
      
      The ID of the virtual private cloud (VPC) used by this DevEndpoint.
    - ExtraPythonLibsS3Path (string) --
      
      Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.
      
      Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
    - ExtraJarsS3Path (string) --
      
      Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.
      
      Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.
    - FailureReason (string) --
      
      The reason for a current failure in this DevEndpoint.
    - LastUpdateStatus (string) --
      
      The status of the last update.
    - CreatedTimestamp (datetime) --
      
      The point in time at which this DevEndpoint was created.
    - LastModifiedTimestamp (datetime) --
      
      The point in time at which this DevEndpoint was last modified.
    - PublicKey (string) --
      
      The public key to be used by this DevEndpoint for authentication. This attribute is provided for backward compatibility, as the recommended attribute to use is public keys.
    - PublicKeys (list) --
      
      A list of public keys to be used by the DevEndpoints for authentication. The use of this attribute is preferred over a single public key because the public keys allow you to have a different private key per client.
      
      Note
      
      If you previously created an endpoint with a public key, you must remove that key to be able to set a list of public keys: call the UpdateDevEndpoint API with the public key content in the deletePublicKeys attribute, and the list of new keys in the addPublicKeys attribute.
      - (string) --
- NextToken (string) --
  
  A continuation token, if not all DevEndpoint definitions have yet been returned.

UpdateDevEndpoint (updated)

Link ¶
Changes (request)

{'AddPublicKeys': ['string'], 'DeletePublicKeys': ['string']}

Updates a specified DevEndpoint.

See also: AWS API Documentation

Request Syntax

client.update_dev_endpoint(
    EndpointName='string',
    PublicKey='string',
    AddPublicKeys=[
        'string',
    ],
    DeletePublicKeys=[
        'string',
    ],
    CustomLibraries={
        'ExtraPythonLibsS3Path': 'string',
        'ExtraJarsS3Path': 'string'
    },
    UpdateEtlLibraries=True|False
)

type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the DevEndpoint to be updated.

type PublicKey:

string

param PublicKey:

The public key for the DevEndpoint to use.

type AddPublicKeys:

list

param AddPublicKeys:

The list of public keys for the DevEndpoint to use.

(string) --

type DeletePublicKeys:

list

param DeletePublicKeys:

The list of public keys to be deleted from the DevEndpoint.

(string) --

type CustomLibraries:

dict

param CustomLibraries:

Custom Python or Java libraries to be loaded in the DevEndpoint.

ExtraPythonLibsS3Path (string) --

Path(s) to one or more Python libraries in an S3 bucket that should be loaded in your DevEndpoint. Multiple values must be complete paths separated by a comma.

Please note that only pure Python libraries can currently be used on a DevEndpoint. Libraries that rely on C extensions, such as the pandas Python data analysis library, are not yet supported.
ExtraJarsS3Path (string) --

Path to one or more Java Jars in an S3 bucket that should be loaded in your DevEndpoint.

Please note that only pure Java/Scala libraries can currently be used on a DevEndpoint.

type UpdateEtlLibraries:

boolean

param UpdateEtlLibraries:

True if the list of custom libraries to be loaded in the development endpoint needs to be updated, or False otherwise.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --