Amazon Elastic Compute Cloud

2021/10/13 - Amazon Elastic Compute Cloud - 2 updated api methods

Changes  This release adds support for additional VPC Flow Logs delivery options to S3, such as Apache Parquet formatted files, Hourly partitions and Hive-compatible S3 prefixes

CreateFlowLogs (updated) Link ¶
Changes (request)
{'DestinationOptions': {'FileFormat': 'plain-text | parquet',
                        'HiveCompatiblePartitions': 'boolean',
                        'PerHourPartition': 'boolean'}}

Creates one or more flow logs to capture information about IP traffic for a specific network interface, subnet, or VPC.

Flow log data for a monitored network interface is recorded as flow log records, which are log events consisting of fields that describe the traffic flow. For more information, see Flow log records in the Amazon Virtual Private Cloud User Guide .

When publishing to CloudWatch Logs, flow log records are published to a log group, and each network interface has a unique log stream in the log group. When publishing to Amazon S3, flow log records for all of the monitored network interfaces are published to a single log file object that is stored in the specified bucket.

For more information, see VPC Flow Logs in the Amazon Virtual Private Cloud User Guide .

See also: AWS API Documentation

Request Syntax

client.create_flow_logs(
    DryRun=True|False,
    ClientToken='string',
    DeliverLogsPermissionArn='string',
    LogGroupName='string',
    ResourceIds=[
        'string',
    ],
    ResourceType='VPC'|'Subnet'|'NetworkInterface',
    TrafficType='ACCEPT'|'REJECT'|'ALL',
    LogDestinationType='cloud-watch-logs'|'s3',
    LogDestination='string',
    LogFormat='string',
    TagSpecifications=[
        {
            'ResourceType': 'capacity-reservation'|'client-vpn-endpoint'|'customer-gateway'|'carrier-gateway'|'dedicated-host'|'dhcp-options'|'egress-only-internet-gateway'|'elastic-ip'|'elastic-gpu'|'export-image-task'|'export-instance-task'|'fleet'|'fpga-image'|'host-reservation'|'image'|'import-image-task'|'import-snapshot-task'|'instance'|'instance-event-window'|'internet-gateway'|'ipv4pool-ec2'|'ipv6pool-ec2'|'key-pair'|'launch-template'|'local-gateway'|'local-gateway-route-table'|'local-gateway-virtual-interface'|'local-gateway-virtual-interface-group'|'local-gateway-route-table-vpc-association'|'local-gateway-route-table-virtual-interface-group-association'|'natgateway'|'network-acl'|'network-interface'|'network-insights-analysis'|'network-insights-path'|'placement-group'|'prefix-list'|'replace-root-volume-task'|'reserved-instances'|'route-table'|'security-group'|'security-group-rule'|'snapshot'|'spot-fleet-request'|'spot-instances-request'|'subnet'|'traffic-mirror-filter'|'traffic-mirror-session'|'traffic-mirror-target'|'transit-gateway'|'transit-gateway-attachment'|'transit-gateway-connect-peer'|'transit-gateway-multicast-domain'|'transit-gateway-route-table'|'volume'|'vpc'|'vpc-endpoint'|'vpc-endpoint-service'|'vpc-peering-connection'|'vpn-connection'|'vpn-gateway'|'vpc-flow-log',
            'Tags': [
                {
                    'Key': 'string',
                    'Value': 'string'
                },
            ]
        },
    ],
    MaxAggregationInterval=123,
    DestinationOptions={
        'FileFormat': 'plain-text'|'parquet',
        'HiveCompatiblePartitions': True|False,
        'PerHourPartition': True|False
    }
)
type DryRun

boolean

param DryRun

Checks whether you have the required permissions for the action, without actually making the request, and provides an error response. If you have the required permissions, the error response is DryRunOperation . Otherwise, it is UnauthorizedOperation .

type ClientToken

string

param ClientToken

Unique, case-sensitive identifier that you provide to ensure the idempotency of the request. For more information, see How to ensure idempotency.

type DeliverLogsPermissionArn

string

param DeliverLogsPermissionArn

The ARN for the IAM role that permits Amazon EC2 to publish flow logs to a CloudWatch Logs log group in your account.

If you specify LogDestinationType as s3 , do not specify DeliverLogsPermissionArn or LogGroupName .

type LogGroupName

string

param LogGroupName

The name of a new or existing CloudWatch Logs log group where Amazon EC2 publishes your flow logs.

If you specify LogDestinationType as s3 , do not specify DeliverLogsPermissionArn or LogGroupName .

type ResourceIds

list

param ResourceIds

[REQUIRED]

The ID of the subnet, network interface, or VPC for which you want to create a flow log.

Constraints: Maximum of 1000 resources

  • (string) --

type ResourceType

string

param ResourceType

[REQUIRED]

The type of resource for which to create the flow log. For example, if you specified a VPC ID for the ResourceId property, specify VPC for this property.

type TrafficType

string

param TrafficType

[REQUIRED]

The type of traffic to log. You can log traffic that the resource accepts or rejects, or all traffic.

type LogDestinationType

string

param LogDestinationType

The type of destination to which the flow log data is to be published. Flow log data can be published to CloudWatch Logs or Amazon S3. To publish flow log data to CloudWatch Logs, specify cloud-watch-logs . To publish flow log data to Amazon S3, specify s3 .

If you specify LogDestinationType as s3 , do not specify DeliverLogsPermissionArn or LogGroupName .

Default: cloud-watch-logs

type LogDestination

string

param LogDestination

The destination to which the flow log data is to be published. Flow log data can be published to a CloudWatch Logs log group or an Amazon S3 bucket. The value specified for this parameter depends on the value specified for LogDestinationType .

If LogDestinationType is not specified or cloud-watch-logs , specify the Amazon Resource Name (ARN) of the CloudWatch Logs log group. For example, to publish to a log group called my-logs , specify arn:aws:logs:us-east-1:123456789012:log-group:my-logs . Alternatively, use LogGroupName instead.

If LogDestinationType is s3 , specify the ARN of the Amazon S3 bucket. You can also specify a subfolder in the bucket. To specify a subfolder in the bucket, use the following ARN format: bucket_ARN/subfolder_name/ . For example, to specify a subfolder named my-logs in a bucket named my-bucket , use the following ARN: arn:aws:s3:::my-bucket/my-logs/ . You cannot use AWSLogs as a subfolder name. This is a reserved term.

type LogFormat

string

param LogFormat

The fields to include in the flow log record, in the order in which they should appear. For a list of available fields, see Flow log records. If you omit this parameter, the flow log is created using the default format. If you specify this parameter, you must specify at least one field.

Specify the fields using the ${field-id} format, separated by spaces. For the CLI, surround this parameter value with single quotes on Linux or double quotes on Windows.

type TagSpecifications

list

param TagSpecifications

The tags to apply to the flow logs.

  • (dict) --

    The tags to apply to a resource when the resource is being created.

    • ResourceType (string) --

      The type of resource to tag on creation.

    • Tags (list) --

      The tags to apply to the resource.

      • (dict) --

        Describes a tag.

        • Key (string) --

          The key of the tag.

          Constraints: Tag keys are case-sensitive and accept a maximum of 127 Unicode characters. May not begin with aws: .

        • Value (string) --

          The value of the tag.

          Constraints: Tag values are case-sensitive and accept a maximum of 255 Unicode characters.

type MaxAggregationInterval

integer

param MaxAggregationInterval

The maximum interval of time during which a flow of packets is captured and aggregated into a flow log record. You can specify 60 seconds (1 minute) or 600 seconds (10 minutes).

When a network interface is attached to a Nitro-based instance, the aggregation interval is always 60 seconds or less, regardless of the value that you specify.

Default: 600

type DestinationOptions

dict

param DestinationOptions

The destination options.

  • FileFormat (string) --

    The format for the flow log. The default is plain-text .

  • HiveCompatiblePartitions (boolean) --

    Indicates whether to use Hive-compatible prefixes for flow logs stored in Amazon S3. The default is false .

  • PerHourPartition (boolean) --

    Indicates whether to partition the flow log per hour. This reduces the cost and response time for queries. The default is false .

rtype

dict

returns

Response Syntax

{
    'ClientToken': 'string',
    'FlowLogIds': [
        'string',
    ],
    'Unsuccessful': [
        {
            'Error': {
                'Code': 'string',
                'Message': 'string'
            },
            'ResourceId': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • ClientToken (string) --

      Unique, case-sensitive identifier that you provide to ensure the idempotency of the request.

    • FlowLogIds (list) --

      The IDs of the flow logs.

      • (string) --

    • Unsuccessful (list) --

      Information about the flow logs that could not be created successfully.

      • (dict) --

        Information about items that were not successfully processed in a batch call.

        • Error (dict) --

          Information about the error.

          • Code (string) --

            The error code.

          • Message (string) --

            The error message accompanying the error code.

        • ResourceId (string) --

          The ID of the resource.

DescribeFlowLogs (updated) Link ¶
Changes (response)
{'FlowLogs': {'DestinationOptions': {'FileFormat': 'plain-text | parquet',
                                     'HiveCompatiblePartitions': 'boolean',
                                     'PerHourPartition': 'boolean'}}}

Describes one or more flow logs. To view the information in your flow logs (the log streams for the network interfaces), you must use the CloudWatch Logs console or the CloudWatch Logs API.

See also: AWS API Documentation

Request Syntax

client.describe_flow_logs(
    DryRun=True|False,
    Filter=[
        {
            'Name': 'string',
            'Values': [
                'string',
            ]
        },
    ],
    FlowLogIds=[
        'string',
    ],
    MaxResults=123,
    NextToken='string'
)
type DryRun

boolean

param DryRun

Checks whether you have the required permissions for the action, without actually making the request, and provides an error response. If you have the required permissions, the error response is DryRunOperation . Otherwise, it is UnauthorizedOperation .

type Filter

list

param Filter

One or more filters.

  • deliver-log-status - The status of the logs delivery ( SUCCESS | FAILED ).

  • log-destination-type - The type of destination to which the flow log publishes data. Possible destination types include cloud-watch-logs and s3 .

  • flow-log-id - The ID of the flow log.

  • log-group-name - The name of the log group.

  • resource-id - The ID of the VPC, subnet, or network interface.

  • traffic-type - The type of traffic ( ACCEPT | REJECT | ALL ).

  • tag :<key> - The key/value combination of a tag assigned to the resource. Use the tag key in the filter name and the tag value as the filter value. For example, to find all resources that have a tag with the key Owner and the value TeamA , specify tag:Owner for the filter name and TeamA for the filter value.

  • tag-key - The key of a tag assigned to the resource. Use this filter to find all resources assigned a tag with a specific key, regardless of the tag value.

  • (dict) --

    A filter name and value pair that is used to return a more specific list of results from a describe operation. Filters can be used to match a set of resources by specific criteria, such as tags, attributes, or IDs.

    • Name (string) --

      The name of the filter. Filter names are case-sensitive.

    • Values (list) --

      The filter values. Filter values are case-sensitive.

      • (string) --

type FlowLogIds

list

param FlowLogIds

One or more flow log IDs.

Constraint: Maximum of 1000 flow log IDs.

  • (string) --

type MaxResults

integer

param MaxResults

The maximum number of results to return with a single call. To retrieve the remaining results, make another call with the returned nextToken value.

type NextToken

string

param NextToken

The token for the next page of results.

rtype

dict

returns

Response Syntax

{
    'FlowLogs': [
        {
            'CreationTime': datetime(2015, 1, 1),
            'DeliverLogsErrorMessage': 'string',
            'DeliverLogsPermissionArn': 'string',
            'DeliverLogsStatus': 'string',
            'FlowLogId': 'string',
            'FlowLogStatus': 'string',
            'LogGroupName': 'string',
            'ResourceId': 'string',
            'TrafficType': 'ACCEPT'|'REJECT'|'ALL',
            'LogDestinationType': 'cloud-watch-logs'|'s3',
            'LogDestination': 'string',
            'LogFormat': 'string',
            'Tags': [
                {
                    'Key': 'string',
                    'Value': 'string'
                },
            ],
            'MaxAggregationInterval': 123,
            'DestinationOptions': {
                'FileFormat': 'plain-text'|'parquet',
                'HiveCompatiblePartitions': True|False,
                'PerHourPartition': True|False
            }
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • FlowLogs (list) --

      Information about the flow logs.

      • (dict) --

        Describes a flow log.

        • CreationTime (datetime) --

          The date and time the flow log was created.

        • DeliverLogsErrorMessage (string) --

          Information about the error that occurred. Rate limited indicates that CloudWatch Logs throttling has been applied for one or more network interfaces, or that you've reached the limit on the number of log groups that you can create. Access error indicates that the IAM role associated with the flow log does not have sufficient permissions to publish to CloudWatch Logs. Unknown error indicates an internal error.

        • DeliverLogsPermissionArn (string) --

          The ARN of the IAM role that posts logs to CloudWatch Logs.

        • DeliverLogsStatus (string) --

          The status of the logs delivery ( SUCCESS | FAILED ).

        • FlowLogId (string) --

          The flow log ID.

        • FlowLogStatus (string) --

          The status of the flow log ( ACTIVE ).

        • LogGroupName (string) --

          The name of the flow log group.

        • ResourceId (string) --

          The ID of the resource on which the flow log was created.

        • TrafficType (string) --

          The type of traffic captured for the flow log.

        • LogDestinationType (string) --

          The type of destination to which the flow log data is published. Flow log data can be published to CloudWatch Logs or Amazon S3.

        • LogDestination (string) --

          The destination to which the flow log data is published. Flow log data can be published to an CloudWatch Logs log group or an Amazon S3 bucket. If the flow log publishes to CloudWatch Logs, this element indicates the Amazon Resource Name (ARN) of the CloudWatch Logs log group to which the data is published. If the flow log publishes to Amazon S3, this element indicates the ARN of the Amazon S3 bucket to which the data is published.

        • LogFormat (string) --

          The format of the flow log record.

        • Tags (list) --

          The tags for the flow log.

          • (dict) --

            Describes a tag.

            • Key (string) --

              The key of the tag.

              Constraints: Tag keys are case-sensitive and accept a maximum of 127 Unicode characters. May not begin with aws: .

            • Value (string) --

              The value of the tag.

              Constraints: Tag values are case-sensitive and accept a maximum of 255 Unicode characters.

        • MaxAggregationInterval (integer) --

          The maximum interval of time, in seconds, during which a flow of packets is captured and aggregated into a flow log record.

          When a network interface is attached to a Nitro-based instance, the aggregation interval is always 60 seconds (1 minute) or less, regardless of the specified value.

          Valid Values: 60 | 600

        • DestinationOptions (dict) --

          The destination options.

          • FileFormat (string) --

            The format for the flow log.

          • HiveCompatiblePartitions (boolean) --

            Indicates whether to use Hive-compatible prefixes for flow logs stored in Amazon S3.

          • PerHourPartition (boolean) --

            Indicates whether to partition the flow log per hour.

    • NextToken (string) --

      The token to use to retrieve the next page of results. This value is null when there are no more results to return.