2022/10/27 - AWS Glue - 4 updated api methods
Changes Added support for custom datatypes when using custom csv classifier.
{'CsvClassifier': {'CustomDatatypeConfigured': 'boolean',
'CustomDatatypes': ['string']}}
Creates a classifier in the user's account. This can be a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field of the request is present.
See also: AWS API Documentation
Request Syntax
client.create_classifier(
GrokClassifier={
'Classification': 'string',
'Name': 'string',
'GrokPattern': 'string',
'CustomPatterns': 'string'
},
XMLClassifier={
'Classification': 'string',
'Name': 'string',
'RowTag': 'string'
},
JsonClassifier={
'Name': 'string',
'JsonPath': 'string'
},
CsvClassifier={
'Name': 'string',
'Delimiter': 'string',
'QuoteSymbol': 'string',
'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
'Header': [
'string',
],
'DisableValueTrimming': True|False,
'AllowSingleColumn': True|False,
'CustomDatatypeConfigured': True|False,
'CustomDatatypes': [
'string',
]
}
)
dict
A GrokClassifier object specifying the classifier to create.
Classification (string) -- [REQUIRED]
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
Name (string) -- [REQUIRED]
The name of the new classifier.
GrokPattern (string) -- [REQUIRED]
The grok pattern used by this classifier.
CustomPatterns (string) --
Optional custom grok patterns used by this classifier.
dict
An XMLClassifier object specifying the classifier to create.
Classification (string) -- [REQUIRED]
An identifier of the data format that the classifier matches.
Name (string) -- [REQUIRED]
The name of the classifier.
RowTag (string) --
The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
dict
A JsonClassifier object specifying the classifier to create.
Name (string) -- [REQUIRED]
The name of the classifier.
JsonPath (string) -- [REQUIRED]
A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
dict
A CsvClassifier object specifying the classifier to create.
Name (string) -- [REQUIRED]
The name of the classifier.
Delimiter (string) --
A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --
A custom symbol to denote what combines content into a single column value. Must be different from the column delimiter.
ContainsHeader (string) --
Indicates whether the CSV file contains a header.
Header (list) --
A list of strings representing column names.
(string) --
DisableValueTrimming (boolean) --
Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --
Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --
Enables the configuration of custom datatypes.
CustomDatatypes (list) --
Creates a list of supported custom datatypes.
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --
{'Classifier': {'CsvClassifier': {'CustomDatatypeConfigured': 'boolean',
'CustomDatatypes': ['string']}}}
Retrieve a classifier by name.
See also: AWS API Documentation
Request Syntax
client.get_classifier(
Name='string'
)
string
[REQUIRED]
Name of the classifier to retrieve.
dict
Response Syntax
{
'Classifier': {
'GrokClassifier': {
'Name': 'string',
'Classification': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'GrokPattern': 'string',
'CustomPatterns': 'string'
},
'XMLClassifier': {
'Name': 'string',
'Classification': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'RowTag': 'string'
},
'JsonClassifier': {
'Name': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'JsonPath': 'string'
},
'CsvClassifier': {
'Name': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'Delimiter': 'string',
'QuoteSymbol': 'string',
'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
'Header': [
'string',
],
'DisableValueTrimming': True|False,
'AllowSingleColumn': True|False,
'CustomDatatypeConfigured': True|False,
'CustomDatatypes': [
'string',
]
}
}
}
Response Structure
(dict) --
Classifier (dict) --
The requested classifier.
GrokClassifier (dict) --
A classifier that uses grok.
Name (string) --
The name of the classifier.
Classification (string) --
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
GrokPattern (string) --
The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifiers.
CustomPatterns (string) --
Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifiers.
XMLClassifier (dict) --
A classifier for XML content.
Name (string) --
The name of the classifier.
Classification (string) --
An identifier of the data format that the classifier matches.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
RowTag (string) --
The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
JsonClassifier (dict) --
A classifier for JSON content.
Name (string) --
The name of the classifier.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
JsonPath (string) --
A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
CsvClassifier (dict) --
A classifier for comma-separated values (CSV).
Name (string) --
The name of the classifier.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
Delimiter (string) --
A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --
A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
ContainsHeader (string) --
Indicates whether the CSV file contains a header.
Header (list) --
A list of strings representing column names.
(string) --
DisableValueTrimming (boolean) --
Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --
Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --
Enables the custom datatype to be configured.
CustomDatatypes (list) --
A list of custom datatypes including "BINARY", "BOOLEAN", "DATE", "DECIMAL", "DOUBLE", "FLOAT", "INT", "LONG", "SHORT", "STRING", "TIMESTAMP".
(string) --
{'Classifiers': {'CsvClassifier': {'CustomDatatypeConfigured': 'boolean',
'CustomDatatypes': ['string']}}}
Lists all classifier objects in the Data Catalog.
See also: AWS API Documentation
Request Syntax
client.get_classifiers(
MaxResults=123,
NextToken='string'
)
integer
The size of the list to return (optional).
string
An optional continuation token.
dict
Response Syntax
{
'Classifiers': [
{
'GrokClassifier': {
'Name': 'string',
'Classification': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'GrokPattern': 'string',
'CustomPatterns': 'string'
},
'XMLClassifier': {
'Name': 'string',
'Classification': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'RowTag': 'string'
},
'JsonClassifier': {
'Name': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'JsonPath': 'string'
},
'CsvClassifier': {
'Name': 'string',
'CreationTime': datetime(2015, 1, 1),
'LastUpdated': datetime(2015, 1, 1),
'Version': 123,
'Delimiter': 'string',
'QuoteSymbol': 'string',
'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
'Header': [
'string',
],
'DisableValueTrimming': True|False,
'AllowSingleColumn': True|False,
'CustomDatatypeConfigured': True|False,
'CustomDatatypes': [
'string',
]
}
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Classifiers (list) --
The requested list of classifier objects.
(dict) --
Classifiers are triggered during a crawl task. A classifier checks whether a given file is in a format it can handle. If it is, the classifier creates a schema in the form of a StructType object that matches that data format.
You can use the standard classifiers that Glue provides, or you can write your own classifiers to best categorize your data sources and specify the appropriate schemas to use for them. A classifier can be a grok classifier, an XML classifier, a JSON classifier, or a custom CSV classifier, as specified in one of the fields in the Classifier object.
GrokClassifier (dict) --
A classifier that uses grok.
Name (string) --
The name of the classifier.
Classification (string) --
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, and so on.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
GrokPattern (string) --
The grok pattern applied to a data store by this classifier. For more information, see built-in patterns in Writing Custom Classifiers.
CustomPatterns (string) --
Optional custom grok patterns defined by this classifier. For more information, see custom patterns in Writing Custom Classifiers.
XMLClassifier (dict) --
A classifier for XML content.
Name (string) --
The name of the classifier.
Classification (string) --
An identifier of the data format that the classifier matches.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
RowTag (string) --
The XML tag designating the element that contains each record in an XML document being parsed. This can't identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
JsonClassifier (dict) --
A classifier for JSON content.
Name (string) --
The name of the classifier.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
JsonPath (string) --
A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
CsvClassifier (dict) --
A classifier for comma-separated values (CSV).
Name (string) --
The name of the classifier.
CreationTime (datetime) --
The time that this classifier was registered.
LastUpdated (datetime) --
The time that this classifier was last updated.
Version (integer) --
The version of this classifier.
Delimiter (string) --
A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --
A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
ContainsHeader (string) --
Indicates whether the CSV file contains a header.
Header (list) --
A list of strings representing column names.
(string) --
DisableValueTrimming (boolean) --
Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --
Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --
Enables the custom datatype to be configured.
CustomDatatypes (list) --
A list of custom datatypes including "BINARY", "BOOLEAN", "DATE", "DECIMAL", "DOUBLE", "FLOAT", "INT", "LONG", "SHORT", "STRING", "TIMESTAMP".
(string) --
NextToken (string) --
A continuation token.
{'CsvClassifier': {'CustomDatatypeConfigured': 'boolean',
'CustomDatatypes': ['string']}}
Modifies an existing classifier (a GrokClassifier, an XMLClassifier, a JsonClassifier, or a CsvClassifier, depending on which field is present).
See also: AWS API Documentation
Request Syntax
client.update_classifier(
GrokClassifier={
'Name': 'string',
'Classification': 'string',
'GrokPattern': 'string',
'CustomPatterns': 'string'
},
XMLClassifier={
'Name': 'string',
'Classification': 'string',
'RowTag': 'string'
},
JsonClassifier={
'Name': 'string',
'JsonPath': 'string'
},
CsvClassifier={
'Name': 'string',
'Delimiter': 'string',
'QuoteSymbol': 'string',
'ContainsHeader': 'UNKNOWN'|'PRESENT'|'ABSENT',
'Header': [
'string',
],
'DisableValueTrimming': True|False,
'AllowSingleColumn': True|False,
'CustomDatatypeConfigured': True|False,
'CustomDatatypes': [
'string',
]
}
)
dict
A GrokClassifier object with updated fields.
Name (string) -- [REQUIRED]
The name of the GrokClassifier.
Classification (string) --
An identifier of the data format that the classifier matches, such as Twitter, JSON, Omniture logs, Amazon CloudWatch Logs, and so on.
GrokPattern (string) --
The grok pattern used by this classifier.
CustomPatterns (string) --
Optional custom grok patterns used by this classifier.
dict
An XMLClassifier object with updated fields.
Name (string) -- [REQUIRED]
The name of the classifier.
Classification (string) --
An identifier of the data format that the classifier matches.
RowTag (string) --
The XML tag designating the element that contains each record in an XML document being parsed. This cannot identify a self-closing element (closed by />). An empty row element that contains only attributes can be parsed as long as it ends with a closing tag (for example, <row item_a="A" item_b="B"></row> is okay, but <row item_a="A" item_b="B" /> is not).
dict
A JsonClassifier object with updated fields.
Name (string) -- [REQUIRED]
The name of the classifier.
JsonPath (string) --
A JsonPath string defining the JSON data for the classifier to classify. Glue supports a subset of JsonPath, as described in Writing JsonPath Custom Classifiers.
dict
A CsvClassifier object with updated fields.
Name (string) -- [REQUIRED]
The name of the classifier.
Delimiter (string) --
A custom symbol to denote what separates each column entry in the row.
QuoteSymbol (string) --
A custom symbol to denote what combines content into a single column value. It must be different from the column delimiter.
ContainsHeader (string) --
Indicates whether the CSV file contains a header.
Header (list) --
A list of strings representing column names.
(string) --
DisableValueTrimming (boolean) --
Specifies not to trim values before identifying the type of column values. The default value is true.
AllowSingleColumn (boolean) --
Enables the processing of files that contain only one column.
CustomDatatypeConfigured (boolean) --
Specifies the configuration of custom datatypes.
CustomDatatypes (list) --
Specifies a list of supported custom datatypes.
(string) --
dict
Response Syntax
{}
Response Structure
(dict) --