2024/09/09 - Managed Streaming for Kafka - 2 updated api methods
Changes Amazon MSK Replicator can now replicate data to identically named topics between MSK clusters within the same AWS Region or across different AWS Regions.
{'ReplicationInfoList': {'TopicReplication': {'TopicNameConfiguration': {'Type': 'PREFIXED_WITH_SOURCE_CLUSTER_ALIAS '
'| '
'IDENTICAL'}}}}
Creates the replicator.
See also: AWS API Documentation
Request Syntax
client.create_replicator(
Description='string',
KafkaClusters=[
{
'AmazonMskCluster': {
'MskClusterArn': 'string'
},
'VpcConfig': {
'SecurityGroupIds': [
'string',
],
'SubnetIds': [
'string',
]
}
},
],
ReplicationInfoList=[
{
'ConsumerGroupReplication': {
'ConsumerGroupsToExclude': [
'string',
],
'ConsumerGroupsToReplicate': [
'string',
],
'DetectAndCopyNewConsumerGroups': True|False,
'SynchroniseConsumerGroupOffsets': True|False
},
'SourceKafkaClusterArn': 'string',
'TargetCompressionType': 'NONE'|'GZIP'|'SNAPPY'|'LZ4'|'ZSTD',
'TargetKafkaClusterArn': 'string',
'TopicReplication': {
'CopyAccessControlListsForTopics': True|False,
'CopyTopicConfigurations': True|False,
'DetectAndCopyNewTopics': True|False,
'StartingPosition': {
'Type': 'LATEST'|'EARLIEST'
},
'TopicNameConfiguration': {
'Type': 'PREFIXED_WITH_SOURCE_CLUSTER_ALIAS'|'IDENTICAL'
},
'TopicsToExclude': [
'string',
],
'TopicsToReplicate': [
'string',
]
}
},
],
ReplicatorName='string',
ServiceExecutionRoleArn='string',
Tags={
'string': 'string'
}
)
string
A summary description of the replicator.
list
[REQUIRED]
Kafka Clusters to use in setting up sources / targets for replication.
(dict) --
Information about Kafka Cluster to be used as source / target for replication.
AmazonMskCluster (dict) -- [REQUIRED]
Details of an Amazon MSK Cluster.
MskClusterArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of an Amazon MSK cluster.
VpcConfig (dict) -- [REQUIRED]
Details of an Amazon VPC which has network connectivity to the Apache Kafka cluster.
SecurityGroupIds (list) --
The security groups to attach to the ENIs for the broker nodes.
(string) --
SubnetIds (list) -- [REQUIRED]
The list of subnets in the client VPC to connect to.
(string) --
list
[REQUIRED]
A list of replication configurations, where each configuration targets a given source cluster to target cluster replication flow.
(dict) --
Specifies configuration for replication between a source and target Kafka cluster.
ConsumerGroupReplication (dict) -- [REQUIRED]
Configuration relating to consumer group replication.
ConsumerGroupsToExclude (list) --
List of regular expression patterns indicating the consumer groups that should not be replicated.
(string) --
ConsumerGroupsToReplicate (list) -- [REQUIRED]
List of regular expression patterns indicating the consumer groups to copy.
(string) --
DetectAndCopyNewConsumerGroups (boolean) --
Enables synchronization of consumer groups to target cluster.
SynchroniseConsumerGroupOffsets (boolean) --
Enables synchronization of consumer group offsets to target cluster. The translated offsets will be written to topic __consumer_offsets.
SourceKafkaClusterArn (string) -- [REQUIRED]
The ARN of the source Kafka cluster.
TargetCompressionType (string) -- [REQUIRED]
The compression type to use when producing records to target cluster.
TargetKafkaClusterArn (string) -- [REQUIRED]
The ARN of the target Kafka cluster.
TopicReplication (dict) -- [REQUIRED]
Configuration relating to topic replication.
CopyAccessControlListsForTopics (boolean) --
Whether to periodically configure remote topic ACLs to match their corresponding upstream topics.
CopyTopicConfigurations (boolean) --
Whether to periodically configure remote topics to match their corresponding upstream topics.
DetectAndCopyNewTopics (boolean) --
Whether to periodically check for new topics and partitions.
StartingPosition (dict) --
Configuration for specifying the position in the topics to start replicating from.
Type (string) --
The type of replication starting position.
TopicNameConfiguration (dict) --
Configuration for specifying replicated topic names should be the same as their corresponding upstream topics or prefixed with source cluster alias.
Type (string) --
The type of replicated topic name.
TopicsToExclude (list) --
List of regular expression patterns indicating the topics that should not be replicated.
(string) --
TopicsToReplicate (list) -- [REQUIRED]
List of regular expression patterns indicating the topics to copy.
(string) --
string
[REQUIRED]
The name of the replicator. Alpha-numeric characters with '-' are allowed.
string
[REQUIRED]
The ARN of the IAM role used by the replicator to access resources in the customer's account (e.g source and target clusters)
dict
List of tags to attach to created Replicator.
(string) --
(string) --
dict
Response Syntax
{
'ReplicatorArn': 'string',
'ReplicatorName': 'string',
'ReplicatorState': 'RUNNING'|'CREATING'|'UPDATING'|'DELETING'|'FAILED'
}
Response Structure
(dict) --
HTTP Status Code 200: OK.
ReplicatorArn (string) --
The Amazon Resource Name (ARN) of the replicator.
ReplicatorName (string) --
Name of the replicator provided by the customer.
ReplicatorState (string) --
State of the replicator.
{'ReplicationInfoList': {'TopicReplication': {'TopicNameConfiguration': {'Type': 'PREFIXED_WITH_SOURCE_CLUSTER_ALIAS '
'| '
'IDENTICAL'}}}}
Describes a replicator.
See also: AWS API Documentation
Request Syntax
client.describe_replicator(
ReplicatorArn='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) of the replicator to be described.
dict
Response Syntax
{
'CreationTime': datetime(2015, 1, 1),
'CurrentVersion': 'string',
'IsReplicatorReference': True|False,
'KafkaClusters': [
{
'AmazonMskCluster': {
'MskClusterArn': 'string'
},
'KafkaClusterAlias': 'string',
'VpcConfig': {
'SecurityGroupIds': [
'string',
],
'SubnetIds': [
'string',
]
}
},
],
'ReplicationInfoList': [
{
'ConsumerGroupReplication': {
'ConsumerGroupsToExclude': [
'string',
],
'ConsumerGroupsToReplicate': [
'string',
],
'DetectAndCopyNewConsumerGroups': True|False,
'SynchroniseConsumerGroupOffsets': True|False
},
'SourceKafkaClusterAlias': 'string',
'TargetCompressionType': 'NONE'|'GZIP'|'SNAPPY'|'LZ4'|'ZSTD',
'TargetKafkaClusterAlias': 'string',
'TopicReplication': {
'CopyAccessControlListsForTopics': True|False,
'CopyTopicConfigurations': True|False,
'DetectAndCopyNewTopics': True|False,
'StartingPosition': {
'Type': 'LATEST'|'EARLIEST'
},
'TopicNameConfiguration': {
'Type': 'PREFIXED_WITH_SOURCE_CLUSTER_ALIAS'|'IDENTICAL'
},
'TopicsToExclude': [
'string',
],
'TopicsToReplicate': [
'string',
]
}
},
],
'ReplicatorArn': 'string',
'ReplicatorDescription': 'string',
'ReplicatorName': 'string',
'ReplicatorResourceArn': 'string',
'ReplicatorState': 'RUNNING'|'CREATING'|'UPDATING'|'DELETING'|'FAILED',
'ServiceExecutionRoleArn': 'string',
'StateInfo': {
'Code': 'string',
'Message': 'string'
},
'Tags': {
'string': 'string'
}
}
Response Structure
(dict) --
HTTP Status Code 200: OK.
CreationTime (datetime) --
The time when the replicator was created.
CurrentVersion (string) --
The current version number of the replicator.
IsReplicatorReference (boolean) --
Whether this resource is a replicator reference.
KafkaClusters (list) --
Kafka Clusters used in setting up sources / targets for replication.
(dict) --
Information about Kafka Cluster used as source / target for replication.
AmazonMskCluster (dict) --
Details of an Amazon MSK Cluster.
MskClusterArn (string) --
The Amazon Resource Name (ARN) of an Amazon MSK cluster.
KafkaClusterAlias (string) --
The alias of the Kafka cluster. Used to prefix names of replicated topics.
VpcConfig (dict) --
Details of an Amazon VPC which has network connectivity to the Apache Kafka cluster.
SecurityGroupIds (list) --
The security groups to attach to the ENIs for the broker nodes.
(string) --
SubnetIds (list) --
The list of subnets in the client VPC to connect to.
(string) --
ReplicationInfoList (list) --
A list of replication configurations, where each configuration targets a given source cluster to target cluster replication flow.
(dict) --
Specifies configuration for replication between a source and target Kafka cluster (sourceKafkaClusterAlias -> targetKafkaClusterAlias)
ConsumerGroupReplication (dict) --
Configuration relating to consumer group replication.
ConsumerGroupsToExclude (list) --
List of regular expression patterns indicating the consumer groups that should not be replicated.
(string) --
ConsumerGroupsToReplicate (list) --
List of regular expression patterns indicating the consumer groups to copy.
(string) --
DetectAndCopyNewConsumerGroups (boolean) --
Enables synchronization of consumer groups to target cluster.
SynchroniseConsumerGroupOffsets (boolean) --
Enables synchronization of consumer group offsets to target cluster. The translated offsets will be written to topic __consumer_offsets.
SourceKafkaClusterAlias (string) --
The alias of the source Kafka cluster.
TargetCompressionType (string) --
The compression type to use when producing records to target cluster.
TargetKafkaClusterAlias (string) --
The alias of the target Kafka cluster.
TopicReplication (dict) --
Configuration relating to topic replication.
CopyAccessControlListsForTopics (boolean) --
Whether to periodically configure remote topic ACLs to match their corresponding upstream topics.
CopyTopicConfigurations (boolean) --
Whether to periodically configure remote topics to match their corresponding upstream topics.
DetectAndCopyNewTopics (boolean) --
Whether to periodically check for new topics and partitions.
StartingPosition (dict) --
Configuration for specifying the position in the topics to start replicating from.
Type (string) --
The type of replication starting position.
TopicNameConfiguration (dict) --
Configuration for specifying replicated topic names should be the same as their corresponding upstream topics or prefixed with source cluster alias.
Type (string) --
The type of replicated topic name.
TopicsToExclude (list) --
List of regular expression patterns indicating the topics that should not be replicated.
(string) --
TopicsToReplicate (list) --
List of regular expression patterns indicating the topics to copy.
(string) --
ReplicatorArn (string) --
The Amazon Resource Name (ARN) of the replicator.
ReplicatorDescription (string) --
The description of the replicator.
ReplicatorName (string) --
The name of the replicator.
ReplicatorResourceArn (string) --
The Amazon Resource Name (ARN) of the replicator resource in the region where the replicator was created.
ReplicatorState (string) --
State of the replicator.
ServiceExecutionRoleArn (string) --
The Amazon Resource Name (ARN) of the IAM role used by the replicator to access resources in the customer's account (e.g source and target clusters)
StateInfo (dict) --
Details about the state of the replicator.
Code (string) -- Code that describes the current state of the replicator.
Message (string) -- Message that describes the state of the replicator.
Tags (dict) --
List of tags attached to the Replicator.
(string) --
(string) --