Managed Streaming for Kafka Connect

2022/03/01 - Managed Streaming for Kafka Connect - 1 new2 updated api methods

Changes  Adds operation for custom plugin deletion (DeleteCustomPlugin) and adds new StateDescription field to DescribeCustomPlugin and DescribeConnector responses to return errors from asynchronous resource creation.

DeleteCustomPlugin (new) Link ¶

Deletes a custom plugin.

See also: AWS API Documentation

Request Syntax

client.delete_custom_plugin(
    customPluginArn='string'
)
type customPluginArn:

string

param customPluginArn:

[REQUIRED]

The Amazon Resource Name (ARN) of the custom plugin that you want to delete.

rtype:

dict

returns:

Response Syntax

{
    'customPluginArn': 'string',
    'customPluginState': 'CREATING'|'CREATE_FAILED'|'ACTIVE'|'UPDATING'|'UPDATE_FAILED'|'DELETING'
}

Response Structure

  • (dict) --

    • customPluginArn (string) --

      The Amazon Resource Name (ARN) of the custom plugin that you requested to delete.

    • customPluginState (string) --

      The state of the custom plugin.

DescribeConnector (updated) Link ¶
Changes (response)
{'stateDescription': {'code': 'string', 'message': 'string'}}

Returns summary information about the connector.

See also: AWS API Documentation

Request Syntax

client.describe_connector(
    connectorArn='string'
)
type connectorArn:

string

param connectorArn:

[REQUIRED]

The Amazon Resource Name (ARN) of the connector that you want to describe.

rtype:

dict

returns:

Response Syntax

{
    'capacity': {
        'autoScaling': {
            'maxWorkerCount': 123,
            'mcuCount': 123,
            'minWorkerCount': 123,
            'scaleInPolicy': {
                'cpuUtilizationPercentage': 123
            },
            'scaleOutPolicy': {
                'cpuUtilizationPercentage': 123
            }
        },
        'provisionedCapacity': {
            'mcuCount': 123,
            'workerCount': 123
        }
    },
    'connectorArn': 'string',
    'connectorConfiguration': {
        'string': 'string'
    },
    'connectorDescription': 'string',
    'connectorName': 'string',
    'connectorState': 'RUNNING'|'CREATING'|'UPDATING'|'DELETING'|'FAILED',
    'creationTime': datetime(2015, 1, 1),
    'currentVersion': 'string',
    'kafkaCluster': {
        'apacheKafkaCluster': {
            'bootstrapServers': 'string',
            'vpc': {
                'securityGroups': [
                    'string',
                ],
                'subnets': [
                    'string',
                ]
            }
        }
    },
    'kafkaClusterClientAuthentication': {
        'authenticationType': 'NONE'|'IAM'
    },
    'kafkaClusterEncryptionInTransit': {
        'encryptionType': 'PLAINTEXT'|'TLS'
    },
    'kafkaConnectVersion': 'string',
    'logDelivery': {
        'workerLogDelivery': {
            'cloudWatchLogs': {
                'enabled': True|False,
                'logGroup': 'string'
            },
            'firehose': {
                'deliveryStream': 'string',
                'enabled': True|False
            },
            's3': {
                'bucket': 'string',
                'enabled': True|False,
                'prefix': 'string'
            }
        }
    },
    'plugins': [
        {
            'customPlugin': {
                'customPluginArn': 'string',
                'revision': 123
            }
        },
    ],
    'serviceExecutionRoleArn': 'string',
    'stateDescription': {
        'code': 'string',
        'message': 'string'
    },
    'workerConfiguration': {
        'revision': 123,
        'workerConfigurationArn': 'string'
    }
}

Response Structure

  • (dict) --

    • capacity (dict) --

      Information about the capacity of the connector, whether it is auto scaled or provisioned.

      • autoScaling (dict) --

        Describes the connector's auto scaling capacity.

        • maxWorkerCount (integer) --

          The maximum number of workers allocated to the connector.

        • mcuCount (integer) --

          The number of microcontroller units (MCUs) allocated to each connector worker. The valid values are 1,2,4,8.

        • minWorkerCount (integer) --

          The minimum number of workers allocated to the connector.

        • scaleInPolicy (dict) --

          The sacle-in policy for the connector.

          • cpuUtilizationPercentage (integer) --

            Specifies the CPU utilization percentage threshold at which you want connector scale in to be triggered.

        • scaleOutPolicy (dict) --

          The sacle-out policy for the connector.>

          • cpuUtilizationPercentage (integer) --

            The CPU utilization percentage threshold at which you want connector scale out to be triggered.

      • provisionedCapacity (dict) --

        Describes a connector's provisioned capacity.

        • mcuCount (integer) --

          The number of microcontroller units (MCUs) allocated to each connector worker. The valid values are 1,2,4,8.

        • workerCount (integer) --

          The number of workers that are allocated to the connector.

    • connectorArn (string) --

      The Amazon Resource Name (ARN) of the connector.

    • connectorConfiguration (dict) --

      A map of keys to values that represent the configuration for the connector.

      • (string) --

        • (string) --

    • connectorDescription (string) --

      A summary description of the connector.

    • connectorName (string) --

      The name of the connector.

    • connectorState (string) --

      The state of the connector.

    • creationTime (datetime) --

      The time the connector was created.

    • currentVersion (string) --

      The current version of the connector.

    • kafkaCluster (dict) --

      The Apache Kafka cluster that the connector is connected to.

      • apacheKafkaCluster (dict) --

        The Apache Kafka cluster to which the connector is connected.

        • bootstrapServers (string) --

          The bootstrap servers of the cluster.

        • vpc (dict) --

          Details of an Amazon VPC which has network connectivity to the Apache Kafka cluster.

          • securityGroups (list) --

            The security groups for the connector.

            • (string) --

          • subnets (list) --

            The subnets for the connector.

            • (string) --

    • kafkaClusterClientAuthentication (dict) --

      The type of client authentication used to connect to the Apache Kafka cluster. The value is NONE when no client authentication is used.

      • authenticationType (string) --

        The type of client authentication used to connect to the Apache Kafka cluster. Value NONE means that no client authentication is used.

    • kafkaClusterEncryptionInTransit (dict) --

      Details of encryption in transit to the Apache Kafka cluster.

      • encryptionType (string) --

        The type of encryption in transit to the Apache Kafka cluster.

    • kafkaConnectVersion (string) --

      The version of Kafka Connect. It has to be compatible with both the Apache Kafka cluster's version and the plugins.

    • logDelivery (dict) --

      Details about delivering logs to Amazon CloudWatch Logs.

      • workerLogDelivery (dict) --

        The workers can send worker logs to different destination types. This configuration specifies the details of these destinations.

        • cloudWatchLogs (dict) --

          Details about delivering logs to Amazon CloudWatch Logs.

          • enabled (boolean) --

            Whether log delivery to Amazon CloudWatch Logs is enabled.

          • logGroup (string) --

            The name of the CloudWatch log group that is the destination for log delivery.

        • firehose (dict) --

          Details about delivering logs to Amazon Kinesis Data Firehose.

          • deliveryStream (string) --

            The name of the Kinesis Data Firehose delivery stream that is the destination for log delivery.

          • enabled (boolean) --

            Specifies whether connector logs get delivered to Amazon Kinesis Data Firehose.

        • s3 (dict) --

          Details about delivering logs to Amazon S3.

          • bucket (string) --

            The name of the S3 bucket that is the destination for log delivery.

          • enabled (boolean) --

            Specifies whether connector logs get sent to the specified Amazon S3 destination.

          • prefix (string) --

            The S3 prefix that is the destination for log delivery.

    • plugins (list) --

      Specifies which plugins were used for this connector.

      • (dict) --

        The description of the plugin.

        • customPlugin (dict) --

          Details about a custom plugin.

          • customPluginArn (string) --

            The Amazon Resource Name (ARN) of the custom plugin.

          • revision (integer) --

            The revision of the custom plugin.

    • serviceExecutionRoleArn (string) --

      The Amazon Resource Name (ARN) of the IAM role used by the connector to access Amazon Web Services resources.

    • stateDescription (dict) --

      Details about the state of a connector.

      • code (string) --

        A code that describes the state of a resource.

      • message (string) --

        A message that describes the state of a resource.

    • workerConfiguration (dict) --

      Specifies which worker configuration was used for the connector.

      • revision (integer) --

        The revision of the worker configuration.

      • workerConfigurationArn (string) --

        The Amazon Resource Name (ARN) of the worker configuration.

DescribeCustomPlugin (updated) Link ¶
Changes (response)
{'stateDescription': {'code': 'string', 'message': 'string'}}

A summary description of the custom plugin.

See also: AWS API Documentation

Request Syntax

client.describe_custom_plugin(
    customPluginArn='string'
)
type customPluginArn:

string

param customPluginArn:

[REQUIRED]

Returns information about a custom plugin.

rtype:

dict

returns:

Response Syntax

{
    'creationTime': datetime(2015, 1, 1),
    'customPluginArn': 'string',
    'customPluginState': 'CREATING'|'CREATE_FAILED'|'ACTIVE'|'UPDATING'|'UPDATE_FAILED'|'DELETING',
    'description': 'string',
    'latestRevision': {
        'contentType': 'JAR'|'ZIP',
        'creationTime': datetime(2015, 1, 1),
        'description': 'string',
        'fileDescription': {
            'fileMd5': 'string',
            'fileSize': 123
        },
        'location': {
            's3Location': {
                'bucketArn': 'string',
                'fileKey': 'string',
                'objectVersion': 'string'
            }
        },
        'revision': 123
    },
    'name': 'string',
    'stateDescription': {
        'code': 'string',
        'message': 'string'
    }
}

Response Structure

  • (dict) --

    • creationTime (datetime) --

      The time that the custom plugin was created.

    • customPluginArn (string) --

      The Amazon Resource Name (ARN) of the custom plugin.

    • customPluginState (string) --

      The state of the custom plugin.

    • description (string) --

      The description of the custom plugin.

    • latestRevision (dict) --

      The latest successfully created revision of the custom plugin. If there are no successfully created revisions, this field will be absent.

      • contentType (string) --

        The format of the plugin file.

      • creationTime (datetime) --

        The time that the custom plugin was created.

      • description (string) --

        The description of the custom plugin.

      • fileDescription (dict) --

        Details about the custom plugin file.

        • fileMd5 (string) --

          The hex-encoded MD5 checksum of the custom plugin file. You can use it to validate the file.

        • fileSize (integer) --

          The size in bytes of the custom plugin file. You can use it to validate the file.

      • location (dict) --

        Information about the location of the custom plugin.

        • s3Location (dict) --

          The S3 bucket Amazon Resource Name (ARN), file key, and object version of the plugin file stored in Amazon S3.

          • bucketArn (string) --

            The Amazon Resource Name (ARN) of an S3 bucket.

          • fileKey (string) --

            The file key for an object in an S3 bucket.

          • objectVersion (string) --

            The version of an object in an S3 bucket.

      • revision (integer) --

        The revision of the custom plugin.

    • name (string) --

      The name of the custom plugin.

    • stateDescription (dict) --

      Details about the state of a custom plugin.

      • code (string) --

        A code that describes the state of a resource.

      • message (string) --

        A message that describes the state of a resource.