AWS API Changes

2019/03/26 - AWS Glue - 4 updated api methods

Changes Update glue client to latest version

CreateClassifier (updated)

Link ¶
Changes (request)

{'CsvClassifier': {'AllowSingleColumn': 'boolean',
                   'ContainsHeader': 'UNKNOWN | PRESENT | ABSENT',
                   'Delimiter': 'string',
                   'DisableValueTrimming': 'boolean',
                   'Header': ['string'],
                   'Name': 'string',
                   'QuoteSymbol': 'string'}}

Creates a classifier in the user's account. This may be a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field of the request is present.

See also: AWS API Documentation

Request Syntax

client.create_classifier(
    GrokClassifier={
        'Classification': 'string',
        'Name': 'string',
        'GrokPattern': 'string',
        'CustomPatterns': 'string'
    },
    XMLClassifier={
        'Classification': 'string',
        'Name': 'string',
        'RowTag': 'string'
    },
    JsonClassifier={
        'Name': 'string',
        'JsonPath': 'string'
    },
    CsvClassifier={
        'Name': 'string',
        'Delimiter': 'string',
        'QuoteSymbol': 'string',
        'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
        'Header': [
            'string',
        ],
        'DisableValueTrimming': True|False,
        'AllowSingleColumn': True|False
    }
)

type GrokClassifier:

dict

param GrokClassifier:

A GrokClassifier object specifying the classifier to create.

Classification (string) -- [REQUIRED]

An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
Name (string) -- [REQUIRED]

The name of the new classifier.
GrokPattern (string) -- [REQUIRED]

The grok pattern used by this classifier.
CustomPatterns (string) --

Optional custom grok patterns used by this classifier.

type XMLClassifier:

dict

param XMLClassifier:

An XMLClassifier object specifying the classifier to create.

Classification (string) -- [REQUIRED]

An identifier of the data format that the classifier matches.
Name (string) -- [REQUIRED]

The name of the classifier.
RowTag (string) --

The XML tag designating the element that contains each record in an XML document being parsed. Note that this cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).

type JsonClassifier:

dict

param JsonClassifier:

A JsonClassifier object specifying the classifier to create.

Name (string) -- [REQUIRED]

The name of the classifier.
JsonPath (string) -- [REQUIRED]

A JsonPath string defining the JSON data for the classifier to classify. AWS Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.

type CsvClassifier:

dict

param CsvClassifier:

A CsvClassifier object specifying the classifier to create.

Name (string) -- [REQUIRED]

The name of the classifier.
Delimiter (string) --

A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --

A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
ContainsHeader (string) --

Indicates whether the CSV file contains a header.
Header (list) --

A list of strings representing column names.
- (string) --
DisableValueTrimming (boolean) --

Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --

Enables the processing of files that contain only one column.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --

GetClassifier (updated)

Link ¶
Changes (response)

{'Classifier': {'CsvClassifier': {'AllowSingleColumn': 'boolean',
                                  'ContainsHeader': 'UNKNOWN | PRESENT | '
                                                    'ABSENT',
                                  'CreationTime': 'timestamp',
                                  'Delimiter': 'string',
                                  'DisableValueTrimming': 'boolean',
                                  'Header': ['string'],
                                  'LastUpdated': 'timestamp',
                                  'Name': 'string',
                                  'QuoteSymbol': 'string',
                                  'Version': 'long'}}}

Retrieve a classifier by name.

See also: AWS API Documentation

Request Syntax

client.get_classifier(
    Name='string'
)

type Name:

string

param Name:

[REQUIRED]

Name of the classifier to retrieve.

rtype:

dict

returns:

Response Syntax

{
    'Classifier': {
        'GrokClassifier': {
            'Name': 'string',
            'Classification': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'GrokPattern': 'string',
            'CustomPatterns': 'string'
        },
        'XMLClassifier': {
            'Name': 'string',
            'Classification': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'RowTag': 'string'
        },
        'JsonClassifier': {
            'Name': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'JsonPath': 'string'
        },
        'CsvClassifier': {
            'Name': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'Delimiter': 'string',
            'QuoteSymbol': 'string',
            'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
            'Header': [
                'string',
            ],
            'DisableValueTrimming': True|False,
            'AllowSingleColumn': True|False
        }
    }
}

Response Structure

(dict) --
- Classifier (dict) --
  
  The requested classifier.
  - GrokClassifier (dict) --
    
    A GrokClassifier object.
    - Name (string) --
      
      The name of the classifier.
    - Classification (string) --
      
      An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
    - CreationTime (datetime) --
      
      The time this classifier was registered.
    - LastUpdated (datetime) --
      
      The time this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - GrokPattern (string) --
      
      The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifers.
    - CustomPatterns (string) --
      
      Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifers.
  - XMLClassifier (dict) --
    
    An XMLClassifier object.
    - Name (string) --
      
      The name of the classifier.
    - Classification (string) --
      
      An identifier of the data format that the classifier matches.
    - CreationTime (datetime) --
      
      The time this classifier was registered.
    - LastUpdated (datetime) --
      
      The time this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - RowTag (string) --
      
      The XML tag designating the element that contains each record in an XML document being parsed. Note that this cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
  - JsonClassifier (dict) --
    
    A JsonClassifier object.
    - Name (string) --
      
      The name of the classifier.
    - CreationTime (datetime) --
      
      The time this classifier was registered.
    - LastUpdated (datetime) --
      
      The time this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - JsonPath (string) --
      
      A JsonPath string defining the JSON data for the classifier to classify. AWS Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
  - CsvClassifier (dict) --
    
    A CSVClassifier object.
    - Name (string) --
      
      The name of the classifier.
    - CreationTime (datetime) --
      
      The time this classifier was registered.
    - LastUpdated (datetime) --
      
      The time this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - Delimiter (string) --
      
      A custom symbol to denote what separates each column entry in the row.
    - QuoteSymbol (string) --
      
      A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
    - ContainsHeader (string) --
      
      Indicates whether the CSV file contains a header.
    - Header (list) --
      
      A list of strings representing column names.
      - (string) --
    - DisableValueTrimming (boolean) --
      
      Specifies not to trim values before identifying the type of column values. The default value is true.
    - AllowSingleColumn (boolean) --
      
      Enables the processing of files that contain only one column.

GetClassifiers (updated)

Link ¶
Changes (response)

{'Classifiers': {'CsvClassifier': {'AllowSingleColumn': 'boolean',
                                   'ContainsHeader': 'UNKNOWN | PRESENT | '
                                                     'ABSENT',
                                   'CreationTime': 'timestamp',
                                   'Delimiter': 'string',
                                   'DisableValueTrimming': 'boolean',
                                   'Header': ['string'],
                                   'LastUpdated': 'timestamp',
                                   'Name': 'string',
                                   'QuoteSymbol': 'string',
                                   'Version': 'long'}}}

Lists all classifier objects in the Data Catalog.

See also: AWS API Documentation

Request Syntax

client.get_classifiers(
    MaxResults=123,
    NextToken='string'
)

type MaxResults:

integer

param MaxResults:

Size of the list to return (optional).

type NextToken:

string

param NextToken:

An optional continuation token.

rtype:

dict

returns:

Response Syntax

{
    'Classifiers': [
        {
            'GrokClassifier': {
                'Name': 'string',
                'Classification': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'GrokPattern': 'string',
                'CustomPatterns': 'string'
            },
            'XMLClassifier': {
                'Name': 'string',
                'Classification': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'RowTag': 'string'
            },
            'JsonClassifier': {
                'Name': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'JsonPath': 'string'
            },
            'CsvClassifier': {
                'Name': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'Delimiter': 'string',
                'QuoteSymbol': 'string',
                'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
                'Header': [
                    'string',
                ],
                'DisableValueTrimming': True|False,
                'AllowSingleColumn': True|False
            }
        },
    ],
    'NextToken': 'string'
}

Response Structure

(dict) --
- Classifiers (list) --
  
  The requested list of classifier objects.
  - (dict) --
    
    Classifiers are triggered during a crawl task. A classifier checks whether a given file is in a format it can handle, and if it is, the classifier creates a schema in the form of a StructType object that matches that data format.
    
    You can use the standard classifiers that AWS Glue supplies, or you can write your own classifiers to best categorize your data sources and specify the appropriate schemas to use for them. A classifier can be a grok classifier, an XML classifier, a JSON classifier, or a custom CSV classifier as specified in one of the fields in the Classifier object.
    - GrokClassifier (dict) --
      
      A GrokClassifier object.
      - Name (string) --
        
        The name of the classifier.
      - Classification (string) --
        
        An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
      - CreationTime (datetime) --
        
        The time this classifier was registered.
      - LastUpdated (datetime) --
        
        The time this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - GrokPattern (string) --
        
        The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifers.
      - CustomPatterns (string) --
        
        Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifers.
    - XMLClassifier (dict) --
      
      An XMLClassifier object.
      - Name (string) --
        
        The name of the classifier.
      - Classification (string) --
        
        An identifier of the data format that the classifier matches.
      - CreationTime (datetime) --
        
        The time this classifier was registered.
      - LastUpdated (datetime) --
        
        The time this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - RowTag (string) --
        
        The XML tag designating the element that contains each record in an XML document being parsed. Note that this cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
    - JsonClassifier (dict) --
      
      A JsonClassifier object.
      - Name (string) --
        
        The name of the classifier.
      - CreationTime (datetime) --
        
        The time this classifier was registered.
      - LastUpdated (datetime) --
        
        The time this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - JsonPath (string) --
        
        A JsonPath string defining the JSON data for the classifier to classify. AWS Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
    - CsvClassifier (dict) --
      
      A CSVClassifier object.
      - Name (string) --
        
        The name of the classifier.
      - CreationTime (datetime) --
        
        The time this classifier was registered.
      - LastUpdated (datetime) --
        
        The time this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - Delimiter (string) --
        
        A custom symbol to denote what separates each column entry in the row.
      - QuoteSymbol (string) --
        
        A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
      - ContainsHeader (string) --
        
        Indicates whether the CSV file contains a header.
      - Header (list) --
        
        A list of strings representing column names.
        
        (string) --
      - DisableValueTrimming (boolean) --
        
        Specifies not to trim values before identifying the type of column values. The default value is true.
      - AllowSingleColumn (boolean) --
        
        Enables the processing of files that contain only one column.
- NextToken (string) --
  
  A continuation token.

UpdateClassifier (updated)

Link ¶
Changes (request)

{'CsvClassifier': {'AllowSingleColumn': 'boolean',
                   'ContainsHeader': 'UNKNOWN | PRESENT | ABSENT',
                   'Delimiter': 'string',
                   'DisableValueTrimming': 'boolean',
                   'Header': ['string'],
                   'Name': 'string',
                   'QuoteSymbol': 'string'}}

Modifies an existing classifier (a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field is present).

See also: AWS API Documentation

Request Syntax

client.update_classifier(
    GrokClassifier={
        'Name': 'string',
        'Classification': 'string',
        'GrokPattern': 'string',
        'CustomPatterns': 'string'
    },
    XMLClassifier={
        'Name': 'string',
        'Classification': 'string',
        'RowTag': 'string'
    },
    JsonClassifier={
        'Name': 'string',
        'JsonPath': 'string'
    },
    CsvClassifier={
        'Name': 'string',
        'Delimiter': 'string',
        'QuoteSymbol': 'string',
        'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
        'Header': [
            'string',
        ],
        'DisableValueTrimming': True|False,
        'AllowSingleColumn': True|False
    }
)

type GrokClassifier:

dict

param GrokClassifier:

A GrokClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the GrokClassifier.
Classification (string) --

An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
GrokPattern (string) --

The grok pattern used by this classifier.
CustomPatterns (string) --

Optional custom grok patterns used by this classifier.

type XMLClassifier:

dict

param XMLClassifier:

An XMLClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
Classification (string) --

An identifier of the data format that the classifier matches.
RowTag (string) --

The XML tag designating the element that contains each record in an XML document being parsed. Note that this cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).

type JsonClassifier:

dict

param JsonClassifier:

A JsonClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
JsonPath (string) --

A JsonPath string defining the JSON data for the classifier to classify. AWS Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.

type CsvClassifier:

dict

param CsvClassifier:

A CsvClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
Delimiter (string) --

A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --

A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
ContainsHeader (string) --

Indicates whether the CSV file contains a header.
Header (list) --

A list of strings representing column names.
- (string) --
DisableValueTrimming (boolean) --

Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --

Enables the processing of files that contain only one column.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --