2022/11/28 - Amazon CloudWatch Logs - 3 new7 updated api methods
Changes Updates to support CloudWatch Logs data protection and CloudWatch cross-account observability
Creates a data protection policy for the specified log group. A data protection policy can help safeguard sensitive data that's ingested by the log group by auditing and masking the sensitive log data.
By default, when a user views a log event that includes masked data, the sensitive data is replaced by asterisks. A user who has the logs:Unmask permission can use a GetLogEvents or FilterLogEvents operation with the unmask parameter set to true to view the unmasked log events. Users with the logs:Unmask can also view unmasked data in the CloudWatch Logs console by running a CloudWatch Logs Insights query with the unmask query command.
For more information, including a list of types of data that can be audited and masked, see Protect sensitive log data with masking.
See also: AWS API Documentation
Request Syntax
client.put_data_protection_policy(
logGroupIdentifier='string',
policyDocument='string'
)
string
[REQUIRED]
Specify either the log group name or log group ARN.
string
[REQUIRED]
Specify the data protection policy, in JSON.
This policy must include two JSON blocks:
The first block must include both a DataIdentifer array and an Operation property with an Audit action. The DataIdentifer array lists the types of sensitive data that you want to mask. For more information about the available options, see Types of data that you can mask. The Operation property with an Audit action is required to find the sensitive data terms. This Audit action must contain a FindingsDestination object. You can optionally use that FindingsDestination object to list one or more destinations to send audit findings to. If you specify destinations such as log groups, Kinesis Data Firehose streams, and S3 buckets, they must already exist.
The second block must include both a DataIdentifer array and an Operation property with an Deidentify action. The DataIdentifer array must exactly match the DataIdentifer array in the first block of the policy. The Operation property with the Deidentify action is what actually masks the data, and it must contain the "MaskConfig": {} object. The "MaskConfig": {} object must be empty.
For an example data protection policy, see the Examples section on this page.
dict
Response Syntax
{
'logGroupIdentifier': 'string',
'policyDocument': 'string',
'lastUpdatedTime': 123
}
Response Structure
(dict) --
logGroupIdentifier (string) --
The log group name or ARN that you specified in your request.
policyDocument (string) --
The data protection policy used for this log group.
lastUpdatedTime (integer) --
The date and time that this policy was most recently updated.
Returns information about a log group data protection policy.
See also: AWS API Documentation
Request Syntax
client.get_data_protection_policy(
logGroupIdentifier='string'
)
string
[REQUIRED]
The name or ARN of the log group that contains the data protection policy that you want to see.
dict
Response Syntax
{
'logGroupIdentifier': 'string',
'policyDocument': 'string',
'lastUpdatedTime': 123
}
Response Structure
(dict) --
logGroupIdentifier (string) --
The log group name or ARN that you specified in your request.
policyDocument (string) --
The data protection policy document for this log group.
lastUpdatedTime (integer) --
The date and time that this policy was most recently updated.
Deletes the data protection policy from the specified log group.
For more information about data protection policies, see PutDataProtectionPolicy.
See also: AWS API Documentation
Request Syntax
client.delete_data_protection_policy(
logGroupIdentifier='string'
)
string
[REQUIRED]
The name or ARN of the log group that you want to delete the data protection policy for.
None
{'accountIdentifiers': ['string'],
'includeLinkedAccounts': 'boolean',
'logGroupNamePattern': 'string'}
Response {'logGroups': {'dataProtectionStatus': 'ACTIVATED | DELETED | ARCHIVED | '
'DISABLED'}}
Lists the specified log groups. You can list all your log groups or filter the results by prefix. The results are ASCII-sorted by log group name.
CloudWatch Logs doesn’t support IAM policies that control access to the DescribeLogGroups action by using the aws:ResourceTag/key-name condition key. Other CloudWatch Logs actions do support the use of the aws:ResourceTag/key-name condition key to control access. For more information about using tags to control access, see Controlling access to Amazon Web Services resources using tags.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account and view data from the linked source accounts. For more information, see CloudWatch cross-account observability.
See also: AWS API Documentation
Request Syntax
client.describe_log_groups(
accountIdentifiers=[
'string',
],
logGroupNamePrefix='string',
logGroupNamePattern='string',
nextToken='string',
limit=123,
includeLinkedAccounts=True|False
)
list
When includeLinkedAccounts is set to True, use this parameter to specify the list of accounts to search. You can specify as many as 20 account IDs in the array.
(string) --
string
The prefix to match.
string
If you specify a string for this parameter, the operation returns only log groups that have names that match the string based on a case-sensitive substring search. For example, if you specify Foo, log groups named FooBar, aws/Foo, and GroupFoo would match, but foo, F/o/o and Froo would not match.
string
The token for the next set of items to return. (You received this token from a previous call.)
integer
The maximum number of items returned. If you don't specify a value, the default is up to 50 items.
boolean
If you are using a monitoring account, set this to True to have the operation return log groups in the accounts listed in accountIdentifiers.
If this parameter is set to true and accountIdentifiers contains a null value, the operation returns all log groups in the monitoring account and all log groups in all source accounts that are linked to the monitoring account.
dict
Response Syntax
{
'logGroups': [
{
'logGroupName': 'string',
'creationTime': 123,
'retentionInDays': 123,
'metricFilterCount': 123,
'arn': 'string',
'storedBytes': 123,
'kmsKeyId': 'string',
'dataProtectionStatus': 'ACTIVATED'|'DELETED'|'ARCHIVED'|'DISABLED'
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
logGroups (list) --
The log groups.
If the retentionInDays value is not included for a log group, then that log group's events do not expire.
(dict) --
Represents a log group.
logGroupName (string) --
The name of the log group.
creationTime (integer) --
The creation time of the log group, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
retentionInDays (integer) --
The number of days to retain the log events in the specified log group. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, and 3653.
To set a log group so that its log events do not expire, use DeleteRetentionPolicy.
metricFilterCount (integer) --
The number of metric filters.
arn (string) --
The Amazon Resource Name (ARN) of the log group.
storedBytes (integer) --
The number of bytes stored.
kmsKeyId (string) --
The Amazon Resource Name (ARN) of the KMS key to use when encrypting log data.
dataProtectionStatus (string) --
Displays whether this log group has a protection policy, or whether it had one in the past. For more information, see PutDataProtectionPolicy.
nextToken (string) --
The token for the next set of items to return. The token expires after 24 hours.
{'logGroupIdentifier': 'string'}
Lists the log streams for the specified log group. You can list all the log streams or filter the results by prefix. You can also control how the results are ordered.
This operation has a limit of five transactions per second, after which transactions are throttled.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account and view data from the linked source accounts. For more information, see CloudWatch cross-account observability.
See also: AWS API Documentation
Request Syntax
client.describe_log_streams(
logGroupName='string',
logGroupIdentifier='string',
logStreamNamePrefix='string',
orderBy='LogStreamName'|'LastEventTime',
descending=True|False,
nextToken='string',
limit=123
)
string
[REQUIRED]
The name of the log group.
string
Specify either the name or ARN of the log group to view. If the log group is in a source account and you are using a monitoring account, you must use the log group ARN.
If you specify values for both logGroupName and logGroupIdentifier, the action returns an InvalidParameterException error.
string
The prefix to match.
If orderBy is LastEventTime, you cannot specify this parameter.
string
If the value is LogStreamName, the results are ordered by log stream name. If the value is LastEventTime, the results are ordered by the event time. The default value is LogStreamName.
If you order the results by event time, you cannot specify the logStreamNamePrefix parameter.
lastEventTimestamp represents the time of the most recent log event in the log stream in CloudWatch Logs. This number is expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. lastEventTimestamp updates on an eventual consistency basis. It typically updates in less than an hour from ingestion, but in rare situations might take longer.
boolean
If the value is true, results are returned in descending order. If the value is to false, results are returned in ascending order. The default value is false.
string
The token for the next set of items to return. (You received this token from a previous call.)
integer
The maximum number of items returned. If you don't specify a value, the default is up to 50 items.
dict
Response Syntax
{
'logStreams': [
{
'logStreamName': 'string',
'creationTime': 123,
'firstEventTimestamp': 123,
'lastEventTimestamp': 123,
'lastIngestionTime': 123,
'uploadSequenceToken': 'string',
'arn': 'string',
'storedBytes': 123
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
logStreams (list) --
The log streams.
(dict) --
Represents a log stream, which is a sequence of log events from a single emitter of logs.
logStreamName (string) --
The name of the log stream.
creationTime (integer) --
The creation time of the stream, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
firstEventTimestamp (integer) --
The time of the first event, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
lastEventTimestamp (integer) --
The time of the most recent log event in the log stream in CloudWatch Logs. This number is expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. The lastEventTime value updates on an eventual consistency basis. It typically updates in less than an hour from ingestion, but in rare situations might take longer.
lastIngestionTime (integer) --
The ingestion time, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
uploadSequenceToken (string) --
The sequence token.
arn (string) --
The Amazon Resource Name (ARN) of the log stream.
storedBytes (integer) --
The number of bytes stored.
Important: As of June 17, 2019, this parameter is no longer supported for log streams, and is always reported as zero. This change applies only to log streams. The storedBytes parameter for log groups is not affected.
nextToken (string) --
The token for the next set of items to return. The token expires after 24 hours.
{'logGroupIdentifier': 'string', 'unmask': 'boolean'}
Lists log events from the specified log group. You can list all the log events or filter the results using a filter pattern, a time range, and the name of the log stream.
You must have the logs;FilterLogEvents permission to perform this operation.
By default, this operation returns as many log events as can fit in 1 MB (up to 10,000 log events) or all the events found within the specified time range. If the results include a token, that means there are more log events available. You can get additional results by specifying the token in a subsequent call. This operation can return empty results while there are more log events available through the token.
The returned log events are sorted by event timestamp, the timestamp when the event was ingested by CloudWatch Logs, and the ID of the PutLogEvents request.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account and view data from the linked source accounts. For more information, see CloudWatch cross-account observability.
See also: AWS API Documentation
Request Syntax
client.filter_log_events(
logGroupName='string',
logGroupIdentifier='string',
logStreamNames=[
'string',
],
logStreamNamePrefix='string',
startTime=123,
endTime=123,
filterPattern='string',
nextToken='string',
limit=123,
interleaved=True|False,
unmask=True|False
)
string
[REQUIRED]
The name of the log group to search.
string
Specify either the name or ARN of the log group to view log events from. If the log group is in a source account and you are using a monitoring account, you must use the log group ARN.
If you specify values for both logGroupName and logGroupIdentifier, the action returns an InvalidParameterException error.
list
Filters the results to only logs from the log streams in this list.
If you specify a value for both logStreamNamePrefix and logStreamNames, the action returns an InvalidParameterException error.
(string) --
string
Filters the results to include only events from log streams that have names starting with this prefix.
If you specify a value for both logStreamNamePrefix and logStreamNames, but the value for logStreamNamePrefix does not match any log stream names specified in logStreamNames, the action returns an InvalidParameterException error.
integer
The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp before this time are not returned.
integer
The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp later than this time are not returned.
string
The filter pattern to use. For more information, see Filter and Pattern Syntax.
If not provided, all the events are matched.
string
The token for the next set of events to return. (You received this token from a previous call.)
integer
The maximum number of events to return. The default is 10,000 events.
boolean
If the value is true, the operation attempts to provide responses that contain events from multiple log streams within the log group, interleaved in a single response. If the value is false, all the matched log events in the first log stream are searched first, then those in the next log stream, and so on.
Important As of June 17, 2019, this parameter is ignored and the value is assumed to be true. The response from this operation always interleaves events from multiple log streams within a log group.
boolean
Specify true to display the log event fields with all sensitive data unmasked and visible. The default is false.
To use this operation with this parameter, you must be signed into an account with the logs:Unmask permission.
dict
Response Syntax
{
'events': [
{
'logStreamName': 'string',
'timestamp': 123,
'message': 'string',
'ingestionTime': 123,
'eventId': 'string'
},
],
'searchedLogStreams': [
{
'logStreamName': 'string',
'searchedCompletely': True|False
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
events (list) --
The matched events.
(dict) --
Represents a matched event.
logStreamName (string) --
The name of the log stream to which this event belongs.
timestamp (integer) --
The time the event occurred, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
message (string) --
The data contained in the log event.
ingestionTime (integer) --
The time the event was ingested, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
eventId (string) --
The ID of the event.
searchedLogStreams (list) --
Important As of May 15, 2020, this parameter is no longer supported. This parameter returns an empty list.
Indicates which log streams have been searched and whether each has been searched completely.
(dict) --
Represents the search status of a log stream.
logStreamName (string) --
The name of the log stream.
searchedCompletely (boolean) --
Indicates whether all the events in this log stream were searched.
nextToken (string) --
The token to use when requesting the next set of items. The token expires after 24 hours.
{'logGroupIdentifier': 'string', 'unmask': 'boolean'}
Lists log events from the specified log stream. You can list all of the log events or filter using a time range.
By default, this operation returns as many log events as can fit in a response size of 1MB (up to 10,000 log events). You can get additional log events by specifying one of the tokens in a subsequent call. This operation can return empty results while there are more log events available through the token.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account and view data from the linked source accounts. For more information, see CloudWatch cross-account observability.
See also: AWS API Documentation
Request Syntax
client.get_log_events(
logGroupName='string',
logGroupIdentifier='string',
logStreamName='string',
startTime=123,
endTime=123,
nextToken='string',
limit=123,
startFromHead=True|False,
unmask=True|False
)
string
[REQUIRED]
The name of the log group.
string
Specify either the name or ARN of the log group to view events from. If the log group is in a source account and you are using a monitoring account, you must use the log group ARN.
If you specify values for both logGroupName and logGroupIdentifier, the action returns an InvalidParameterException error.
string
[REQUIRED]
The name of the log stream.
integer
The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to this time or later than this time are included. Events with a timestamp earlier than this time are not included.
integer
The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to or later than this time are not included.
string
The token for the next set of items to return. (You received this token from a previous call.)
integer
The maximum number of log events returned. If you don't specify a limit, the default is as many log events as can fit in a response size of 1 MB (up to 10,000 log events).
boolean
If the value is true, the earliest log events are returned first. If the value is false, the latest log events are returned first. The default value is false.
If you are using a previous nextForwardToken value as the nextToken in this operation, you must specify true for startFromHead.
boolean
Specify true to display the log event fields with all sensitive data unmasked and visible. The default is false.
To use this operation with this parameter, you must be signed into an account with the logs:Unmask permission.
dict
Response Syntax
{
'events': [
{
'timestamp': 123,
'message': 'string',
'ingestionTime': 123
},
],
'nextForwardToken': 'string',
'nextBackwardToken': 'string'
}
Response Structure
(dict) --
events (list) --
The events.
(dict) --
Represents a log event.
timestamp (integer) --
The time the event occurred, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
message (string) --
The data contained in the log event.
ingestionTime (integer) --
The time the event was ingested, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
nextForwardToken (string) --
The token for the next set of items in the forward direction. The token expires after 24 hours. If you have reached the end of the stream, it returns the same token you passed in.
nextBackwardToken (string) --
The token for the next set of items in the backward direction. The token expires after 24 hours. This token is not null. If you have reached the end of the stream, it returns the same token you passed in.
{'logGroupIdentifier': 'string'}
Returns a list of the fields that are included in log events in the specified log group. Includes the percentage of log events that contain each field. The search is limited to a time period that you specify.
In the results, fields that start with @ are fields generated by CloudWatch Logs. For example, @timestamp is the timestamp of each log event. For more information about the fields that are generated by CloudWatch logs, see Supported Logs and Discovered Fields.
The response results are sorted by the frequency percentage, starting with the highest percentage.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account and view data from the linked source accounts. For more information, see CloudWatch cross-account observability.
See also: AWS API Documentation
Request Syntax
client.get_log_group_fields(
logGroupName='string',
time=123,
logGroupIdentifier='string'
)
string
[REQUIRED]
The name of the log group to search.
If you specify values for both logGroupName and logGroupIdentifier, the action returns an InvalidParameterException error.
integer
The time to set as the center of the query. If you specify time, the 15 minutes before this time are queries. If you omit time, the 8 minutes before and 8 minutes after this time are searched.
The time value is specified as epoch time, which is the number of seconds since January 1, 1970, 00:00:00 UTC.
string
Specify either the name or ARN of the log group to view. If the log group is in a source account and you are using a monitoring account, you must specify the ARN.
If you specify values for both logGroupName and logGroupIdentifier, the action returns an InvalidParameterException error.
dict
Response Syntax
{
'logGroupFields': [
{
'name': 'string',
'percent': 123
},
]
}
Response Structure
(dict) --
logGroupFields (list) --
The array of fields found in the query. Each object in the array contains the name of the field, along with the percentage of time it appeared in the log events that were queried.
(dict) --
The fields contained in log events found by a GetLogGroupFields operation, along with the percentage of queried log events in which each field appears.
name (string) --
The name of a log field.
percent (integer) --
The percentage of log events queried that contained the field.
{'unmask': 'boolean'}
Retrieves all of the fields and values of a single log event. All fields are retrieved, even if the original query that produced the logRecordPointer retrieved only a subset of fields. Fields are returned as field name/field value pairs.
The full unparsed log event is returned within @message.
See also: AWS API Documentation
Request Syntax
client.get_log_record(
logRecordPointer='string',
unmask=True|False
)
string
[REQUIRED]
The pointer corresponding to the log event record you want to retrieve. You get this from the response of a GetQueryResults operation. In that response, the value of the @ptr field for a log event is the value to use as logRecordPointer to retrieve that complete log event record.
boolean
Specify true to display the log event fields with all sensitive data unmasked and visible. The default is false.
To use this operation with this parameter, you must be signed into an account with the logs:Unmask permission.
dict
Response Syntax
{
'logRecord': {
'string': 'string'
}
}
Response Structure
(dict) --
logRecord (dict) --
The requested log event, as a JSON string.
(string) --
(string) --
{'logGroupIdentifiers': ['string']}
Schedules a query of a log group using CloudWatch Logs Insights. You specify the log group and time range to query and the query string to use.
For more information, see CloudWatch Logs Insights Query Syntax.
Queries time out after 15 minutes of runtime. If your queries are timing out, reduce the time range being searched or partition your query into a number of queries.
If you are using CloudWatch cross-account observability, you can use this operation in a monitoring account to start a query in a linked source account. For more information, see CloudWatch cross-account observability. For a cross-account StartQuery operation, the query definition must be defined in the monitoring account.
You can have up to 20 concurrent CloudWatch Logs insights queries, including queries that have been added to dashboards.
See also: AWS API Documentation
Request Syntax
client.start_query(
logGroupName='string',
logGroupNames=[
'string',
],
logGroupIdentifiers=[
'string',
],
startTime=123,
endTime=123,
queryString='string',
limit=123
)
string
The log group on which to perform the query.
list
The list of log groups to be queried. You can include up to 50 log groups.
(string) --
list
The list of log groups to query. You can include up to 50 log groups.
You can specify them by the log group name or ARN. If a log group that you're querying is in a source account and you're using a monitoring account, you must specify the ARN of the log group here. The query definition must also be defined in the monitoring account.
If you specify an ARN, the ARN can't end with an asterisk (*).
A StartQuery operation must include exactly one of the following parameters: logGroupName, logGroupNames or logGroupIdentifiers.
(string) --
integer
[REQUIRED]
The beginning of the time range to query. The range is inclusive, so the specified start time is included in the query. Specified as epoch time, the number of seconds since January 1, 1970, 00:00:00 UTC.
integer
[REQUIRED]
The end of the time range to query. The range is inclusive, so the specified end time is included in the query. Specified as epoch time, the number of seconds since January 1, 1970, 00:00:00 UTC.
string
[REQUIRED]
The query string to use. For more information, see CloudWatch Logs Insights Query Syntax.
integer
The maximum number of log events to return in the query. If the query string uses the fields command, only the specified fields and their values are returned. The default is 1000.
dict
Response Syntax
{
'queryId': 'string'
}
Response Structure
(dict) --
queryId (string) --
The unique ID of the query.