AWS API Changes

2023/08/15 - AWS Glue - 4 updated api methods

Changes AWS Glue Crawlers can now accept SerDe overrides from a custom csv classifier. The two SerDe options are LazySimpleSerDe and OpenCSVSerDe. In case, the user wants crawler to do the selection, "None" can be selected for this purpose.

CreateClassifier (updated)

Link ¶
Changes (request)

{'CsvClassifier': {'Serde': 'OpenCSVSerDe | LazySimpleSerDe | None'}}

Creates a classifier in the user's account. This can be a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field of the request is present.

See also: AWS API Documentation

Request Syntax

client.create_classifier(
    GrokClassifier={
        'Classification': 'string',
        'Name': 'string',
        'GrokPattern': 'string',
        'CustomPatterns': 'string'
    },
    XMLClassifier={
        'Classification': 'string',
        'Name': 'string',
        'RowTag': 'string'
    },
    JsonClassifier={
        'Name': 'string',
        'JsonPath': 'string'
    },
    CsvClassifier={
        'Name': 'string',
        'Delimiter': 'string',
        'QuoteSymbol': 'string',
        'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
        'Header': [
            'string',
        ],
        'DisableValueTrimming': True|False,
        'AllowSingleColumn': True|False,
        'CustomDatatypeConfigured': True|False,
        'CustomDatatypes': [
            'string',
        ],
        'Serde': 'OpenCSVSerDe'|'LazySimpleSerDe'|'None'
    }
)

type GrokClassifier:

dict

param GrokClassifier:

A GrokClassifier object specifying the classifier to create.

Classification (string) -- [REQUIRED]

An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
Name (string) -- [REQUIRED]

The name of the new classifier.
GrokPattern (string) -- [REQUIRED]

The grok pattern used by this classifier.
CustomPatterns (string) --

Optional custom grok patterns used by this classifier.

type XMLClassifier:

dict

param XMLClassifier:

An XMLClassifier object specifying the classifier to create.

Classification (string) -- [REQUIRED]

An identifier of the data format that the classifier matches.
Name (string) -- [REQUIRED]

The name of the classifier.
RowTag (string) --

The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).

type JsonClassifier:

dict

param JsonClassifier:

A JsonClassifier object specifying the classifier to create.

Name (string) -- [REQUIRED]

The name of the classifier.
JsonPath (string) -- [REQUIRED]

A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.

type CsvClassifier:

dict

param CsvClassifier:

A CsvClassifier object specifying the classifier to create.

Name (string) -- [REQUIRED]

The name of the classifier.
Delimiter (string) --

A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --

A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
ContainsHeader (string) --

Indicates whether the CSV file contains a header.
Header (list) --

A list of strings representing column names.
- (string) --
DisableValueTrimming (boolean) --

Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --

Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --

Enables the configuration of custom datatypes.
CustomDatatypes (list) --

Creates a list of supported custom datatypes.
- (string) --
Serde (string) --

Sets the SerDe for processing CSV in the classifier, which will be applied in the Data Catalog. Valid values are OpenCSVSerDe, LazySimpleSerDe, and None. You can specify the None value when you want the crawler to do the detection.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --

GetClassifier (updated)

Link ¶
Changes (response)

{'Classifier': {'CsvClassifier': {'Serde': 'OpenCSVSerDe | LazySimpleSerDe | '
                                           'None'}}}

Retrieve a classifier by name.

See also: AWS API Documentation

Request Syntax

client.get_classifier(
    Name='string'
)

type Name:

string

param Name:

[REQUIRED]

Name of the classifier to retrieve.

rtype:

dict

returns:

Response Syntax

{
    'Classifier': {
        'GrokClassifier': {
            'Name': 'string',
            'Classification': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'GrokPattern': 'string',
            'CustomPatterns': 'string'
        },
        'XMLClassifier': {
            'Name': 'string',
            'Classification': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'RowTag': 'string'
        },
        'JsonClassifier': {
            'Name': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'JsonPath': 'string'
        },
        'CsvClassifier': {
            'Name': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastUpdated': datetime(2015, 1, 1),
            'Version': 123,
            'Delimiter': 'string',
            'QuoteSymbol': 'string',
            'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
            'Header': [
                'string',
            ],
            'DisableValueTrimming': True|False,
            'AllowSingleColumn': True|False,
            'CustomDatatypeConfigured': True|False,
            'CustomDatatypes': [
                'string',
            ],
            'Serde': 'OpenCSVSerDe'|'LazySimpleSerDe'|'None'
        }
    }
}

Response Structure

(dict) --
- Classifier (dict) --
  
  The requested classifier.
  - GrokClassifier (dict) --
    
    A classifier that uses grok.
    - Name (string) --
      
      The name of the classifier.
    - Classification (string) --
      
      An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
    - CreationTime (datetime) --
      
      The time that this classifier was registered.
    - LastUpdated (datetime) --
      
      The time that this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - GrokPattern (string) --
      
      The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifiers.
    - CustomPatterns (string) --
      
      Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifiers.
  - XMLClassifier (dict) --
    
    A classifier for XML content.
    - Name (string) --
      
      The name of the classifier.
    - Classification (string) --
      
      An identifier of the data format that the classifier matches.
    - CreationTime (datetime) --
      
      The time that this classifier was registered.
    - LastUpdated (datetime) --
      
      The time that this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - RowTag (string) --
      
      The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
  - JsonClassifier (dict) --
    
    A classifier for JSON content.
    - Name (string) --
      
      The name of the classifier.
    - CreationTime (datetime) --
      
      The time that this classifier was registered.
    - LastUpdated (datetime) --
      
      The time that this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - JsonPath (string) --
      
      A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
  - CsvClassifier (dict) --
    
    A classifier for comma-separated values (CSV).
    - Name (string) --
      
      The name of the classifier.
    - CreationTime (datetime) --
      
      The time that this classifier was registered.
    - LastUpdated (datetime) --
      
      The time that this classifier was last updated.
    - Version (integer) --
      
      The version of this classifier.
    - Delimiter (string) --
      
      A custom symbol to denote what separates each column entry in the row.
    - QuoteSymbol (string) --
      
      A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
    - ContainsHeader (string) --
      
      Indicates whether the CSV file contains a header.
    - Header (list) --
      
      A list of strings representing column names.
      - (string) --
    - DisableValueTrimming (boolean) --
      
      Specifies not to trim values before identifying the type of column values. The default value is true.
    - AllowSingleColumn (boolean) --
      
      Enables the processing of files that contain only one column.
    - CustomDatatypeConfigured (boolean) --
      
      Enables the custom datatype to be configured.
    - CustomDatatypes (list) --
      
      A list of custom datatypes including "BINARY", "BOOLEAN", "DATE", "DECIMAL", "DOUBLE", "FLOAT", "INT", "LONG", "SHORT", "STRING", "TIMESTAMP".
      - (string) --
    - Serde (string) --
      
      Sets the SerDe for processing CSV in the classifier, which will be applied in the Data Catalog. Valid values are OpenCSVSerDe, LazySimpleSerDe, and None. You can specify the None value when you want the crawler to do the detection.

GetClassifiers (updated)

Link ¶
Changes (response)

{'Classifiers': {'CsvClassifier': {'Serde': 'OpenCSVSerDe | LazySimpleSerDe | '
                                            'None'}}}

Lists all classifier objects in the Data Catalog.

See also: AWS API Documentation

Request Syntax

client.get_classifiers(
    MaxResults=123,
    NextToken='string'
)

type MaxResults:

integer

param MaxResults:

The size of the list to return (optional).

type NextToken:

string

param NextToken:

An optional continuation token.

rtype:

dict

returns:

Response Syntax

{
    'Classifiers': [
        {
            'GrokClassifier': {
                'Name': 'string',
                'Classification': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'GrokPattern': 'string',
                'CustomPatterns': 'string'
            },
            'XMLClassifier': {
                'Name': 'string',
                'Classification': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'RowTag': 'string'
            },
            'JsonClassifier': {
                'Name': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'JsonPath': 'string'
            },
            'CsvClassifier': {
                'Name': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastUpdated': datetime(2015, 1, 1),
                'Version': 123,
                'Delimiter': 'string',
                'QuoteSymbol': 'string',
                'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
                'Header': [
                    'string',
                ],
                'DisableValueTrimming': True|False,
                'AllowSingleColumn': True|False,
                'CustomDatatypeConfigured': True|False,
                'CustomDatatypes': [
                    'string',
                ],
                'Serde': 'OpenCSVSerDe'|'LazySimpleSerDe'|'None'
            }
        },
    ],
    'NextToken': 'string'
}

Response Structure

(dict) --
- Classifiers (list) --
  
  The requested list of classifier objects.
  - (dict) --
    
    Classifiers are triggered during a crawl task. A classifier checks whether a given file is in a format it can handle. If it is, the classifier creates a schema in the form of a StructType object that matches that data format.
    
    You can use the standard classifiers that Glue provides, or you can write your own classifiers to best categorize your data sources and specify the appropriate schemas to use for them. A classifier can be a grok classifier, an XML classifier, a JSON classifier, or a custom CSV classifier, as specified in one of the fields in the Classifier object.
    - GrokClassifier (dict) --
      
      A classifier that uses grok.
      - Name (string) --
        
        The name of the classifier.
      - Classification (string) --
        
        An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
      - CreationTime (datetime) --
        
        The time that this classifier was registered.
      - LastUpdated (datetime) --
        
        The time that this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - GrokPattern (string) --
        
        The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifiers.
      - CustomPatterns (string) --
        
        Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifiers.
    - XMLClassifier (dict) --
      
      A classifier for XML content.
      - Name (string) --
        
        The name of the classifier.
      - Classification (string) --
        
        An identifier of the data format that the classifier matches.
      - CreationTime (datetime) --
        
        The time that this classifier was registered.
      - LastUpdated (datetime) --
        
        The time that this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - RowTag (string) --
        
        The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
    - JsonClassifier (dict) --
      
      A classifier for JSON content.
      - Name (string) --
        
        The name of the classifier.
      - CreationTime (datetime) --
        
        The time that this classifier was registered.
      - LastUpdated (datetime) --
        
        The time that this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - JsonPath (string) --
        
        A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
    - CsvClassifier (dict) --
      
      A classifier for comma-separated values (CSV).
      - Name (string) --
        
        The name of the classifier.
      - CreationTime (datetime) --
        
        The time that this classifier was registered.
      - LastUpdated (datetime) --
        
        The time that this classifier was last updated.
      - Version (integer) --
        
        The version of this classifier.
      - Delimiter (string) --
        
        A custom symbol to denote what separates each column entry in the row.
      - QuoteSymbol (string) --
        
        A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
      - ContainsHeader (string) --
        
        Indicates whether the CSV file contains a header.
      - Header (list) --
        
        A list of strings representing column names.
        
        (string) --
      - DisableValueTrimming (boolean) --
        
        Specifies not to trim values before identifying the type of column values. The default value is true.
      - AllowSingleColumn (boolean) --
        
        Enables the processing of files that contain only one column.
      - CustomDatatypeConfigured (boolean) --
        
        Enables the custom datatype to be configured.
      - CustomDatatypes (list) --
        
        A list of custom datatypes including "BINARY", "BOOLEAN", "DATE", "DECIMAL", "DOUBLE", "FLOAT", "INT", "LONG", "SHORT", "STRING", "TIMESTAMP".
        
        (string) --
      - Serde (string) --
        
        Sets the SerDe for processing CSV in the classifier, which will be applied in the Data Catalog. Valid values are OpenCSVSerDe, LazySimpleSerDe, and None. You can specify the None value when you want the crawler to do the detection.
- NextToken (string) --
  
  A continuation token.

UpdateClassifier (updated)

Link ¶
Changes (request)

{'CsvClassifier': {'Serde': 'OpenCSVSerDe | LazySimpleSerDe | None'}}

Modifies an existing classifier (a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field is present).

See also: AWS API Documentation

Request Syntax

client.update_classifier(
    GrokClassifier={
        'Name': 'string',
        'Classification': 'string',
        'GrokPattern': 'string',
        'CustomPatterns': 'string'
    },
    XMLClassifier={
        'Name': 'string',
        'Classification': 'string',
        'RowTag': 'string'
    },
    JsonClassifier={
        'Name': 'string',
        'JsonPath': 'string'
    },
    CsvClassifier={
        'Name': 'string',
        'Delimiter': 'string',
        'QuoteSymbol': 'string',
        'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
        'Header': [
            'string',
        ],
        'DisableValueTrimming': True|False,
        'AllowSingleColumn': True|False,
        'CustomDatatypeConfigured': True|False,
        'CustomDatatypes': [
            'string',
        ],
        'Serde': 'OpenCSVSerDe'|'LazySimpleSerDe'|'None'
    }
)

type GrokClassifier:

dict

param GrokClassifier:

A GrokClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the GrokClassifier.
Classification (string) --

An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
GrokPattern (string) --

The grok pattern used by this classifier.
CustomPatterns (string) --

Optional custom grok patterns used by this classifier.

type XMLClassifier:

dict

param XMLClassifier:

An XMLClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
Classification (string) --

An identifier of the data format that the classifier matches.
RowTag (string) --

The XML tag designating the element that contains each record in an XML document being parsed. This cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).

type JsonClassifier:

dict

param JsonClassifier:

A JsonClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
JsonPath (string) --

A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.

type CsvClassifier:

dict

param CsvClassifier:

A CsvClassifier object with updated fields.

Name (string) -- [REQUIRED]

The name of the classifier.
Delimiter (string) --

A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --

A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
ContainsHeader (string) --

Indicates whether the CSV file contains a header.
Header (list) --

A list of strings representing column names.
- (string) --
DisableValueTrimming (boolean) --

Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --

Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --

Specifies the configuration of custom datatypes.
CustomDatatypes (list) --

Specifies a list of supported custom datatypes.
- (string) --
Serde (string) --

Sets the SerDe for processing CSV in the classifier, which will be applied in the Data Catalog. Valid values are OpenCSVSerDe, LazySimpleSerDe, and None. You can specify the None value when you want the crawler to do the detection.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

(dict) --