2026/07/30 - Managed Streaming for Kafka - 5 new api methods
Changes Amazon MSK Express brokers now support streaming tables for Apache Iceberg, continuously materializing Apache Kafka topics as Iceberg tables in Amazon S3 Tables. Express brokers also now support data delivery to Amazon S3 general purpose buckets.
Updates the destination configuration of an existing channel. Exactly one of icebergDestinationUpdate or s3DestinationUpdate must be supplied.
See also: AWS API Documentation
Request Syntax
client.update_channel(
ChannelArn='string',
ClusterArn='string',
IcebergDestinationUpdate={
'DataFreshnessInSeconds': 123
},
S3DestinationUpdate={
'DataFreshnessInSeconds': 123
}
)
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the channel.
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the cluster.
dict
Updates fields on an Apache Iceberg destination. Use only when the channel was created with an Iceberg destination.
DataFreshnessInSeconds (integer) -- [REQUIRED]
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900.
dict
Updates fields on an Amazon S3 destination. Use only when the channel was created with an Amazon S3 destination.
DataFreshnessInSeconds (integer) -- [REQUIRED]
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900.
dict
Response Syntax
{
'ChannelArn': 'string',
'ClusterOperationArn': 'string'
}
Response Structure
(dict) --
Returns the channel ARN and the cluster-operation ARN that tracks the asynchronous update.
ChannelArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the channel.
ClusterOperationArn (string) --
The Amazon Resource Name (ARN) of the cluster operation.
Returns the current configuration and state of a channel.
See also: AWS API Documentation
Request Syntax
client.describe_channel(
ChannelArn='string',
ClusterArn='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the channel.
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the cluster.
dict
Response Syntax
{
'ChannelArn': 'string',
'ChannelName': 'string',
'EncryptionConfiguration': {
'KmsKeyArn': 'string'
},
'IcebergDestinationConfiguration': {
'AppendOnly': True|False,
'Catalog': {
'CatalogArn': 'string',
'WarehouseLocation': 'string'
},
'DataFreshnessInSeconds': 123,
'DeadLetterQueueS3': {
'BucketArn': 'string',
'ErrorOutputPrefix': 'string',
'ExpectedBucketOwner': 'string'
},
'DestinationTableList': [
{
'DestinationDatabaseName': 'string',
'DestinationTableName': 'string',
'PartitionSpec': {
'PartitionStrategy': 'TIME_HOUR',
'SourceList': [
{
'SourceName': 'string'
},
]
}
},
],
'SchemaEvolution': {
'EnableSchemaEvolution': True|False
},
'ServiceExecutionRoleArn': 'string',
'TableCreation': {
'EnableTableCreation': True|False
},
'CompressionType': 'ZSTD'|'SNAPPY'
},
'S3DestinationConfiguration': {
'DataFreshnessInSeconds': 123,
'DeadLetterQueueS3': {
'BucketArn': 'string',
'ErrorOutputPrefix': 'string',
'ExpectedBucketOwner': 'string'
},
'ServiceExecutionRoleArn': 'string',
'Storage': {
'BucketArn': 'string',
'CompressionType': 'NONE'|'GZIP'|'ZSTD',
'OutputPrefix': 'string',
'OutputKeyTemplate': 'string',
'StorageClass': 'STANDARD'|'INTELLIGENT_TIERING'|'GLACIER_IR',
'ExpectedBucketOwner': 'string'
}
},
'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'FAILED'|'SUSPENDING'|'SUSPENDED',
'DestinationType': 'ICEBERG'|'S3',
'CreationTime': datetime(2015, 1, 1),
'TopicConfigurationList': [
{
'RecordConverter': {
'ValueConverter': 'BYTE_ARRAY'|'JSON'|'JSON_SCHEMA_GSR'|'STRING'
},
'RecordSchema': {
'GsrArn': 'string'
},
'TopicArn': 'string'
},
],
'LoggingInfo': {
'CloudWatchLogs': {
'Enabled': True|False,
'LogGroup': 'string'
},
'Firehose': {
'DeliveryStream': 'string',
'Enabled': True|False
},
'S3': {
'Bucket': 'string',
'Enabled': True|False,
'Prefix': 'string'
}
},
'StateInfo': {
'Code': 'string',
'Message': 'string'
},
'ClusterOperationArn': 'string',
'Tags': {
'string': 'string'
}
}
Response Structure
(dict) --
Successful response.
ChannelArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the channel.
ChannelName (string) --
The name of the channel.
EncryptionConfiguration (dict) --
The encryption configuration applied to the channel.
KmsKeyArn (string) --
The Amazon Resource Name (ARN) of the AWS KMS key used to encrypt the data.
IcebergDestinationConfiguration (dict) --
The Apache Iceberg destination for the channel, if configured.
AppendOnly (boolean) --
Whether the destination is append-only. Must be true; updates and deletes are not supported.
Catalog (dict) --
The AWS Glue Data Catalog and S3 Tables warehouse used by the destination.
CatalogArn (string) --
The Amazon Resource Name (ARN) of the federated AWS Glue Data Catalog that projects the S3 Tables bucket. If omitted, MSK derives the catalog ARN from warehouseLocation.
WarehouseLocation (string) --
The Amazon Resource Name (ARN) of the S3 Tables bucket that backs the Apache Iceberg warehouse.
DataFreshnessInSeconds (integer) --
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
DeadLetterQueueS3 (dict) --
The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
BucketArn (string) --
The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
ErrorOutputPrefix (string) --
An optional prefix prepended to every dead-letter Amazon S3 object key.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
DestinationTableList (list) --
The destination Iceberg tables. Currently exactly one table must be specified.
(dict) --
Configuration of an Apache Iceberg destination table.
DestinationDatabaseName (string) --
The name of the destination namespace (database) in the AWS Glue Data Catalog.
DestinationTableName (string) --
The name of the destination Apache Iceberg table.
PartitionSpec (dict) --
The partition specification for the destination table.
PartitionStrategy (string) --
The partitioning strategy applied to records written to the table.
SourceList (list) --
The source columns used by the partitioning strategy. For TIME_HOUR, must contain exactly one source column whose value is a timestamp.
(dict) --
A source column used by an Apache Iceberg destination table's partition specification.
SourceName (string) --
Source name.
SchemaEvolution (dict) --
Configuration controlling whether the destination table's schema is evolved to match incoming records.
EnableSchemaEvolution (boolean) --
Whether to allow MSK to evolve the destination table's schema. Must be false for the current release.
ServiceExecutionRoleArn (string) --
The Amazon Resource Name (ARN) of the IAM role that MSK assumes to access the destination table, the AWS Glue Data Catalog, and the dead-letter Amazon S3 bucket.
TableCreation (dict) --
Configuration controlling whether MSK creates the destination table if it does not already exist.
EnableTableCreation (boolean) --
Whether MSK creates the destination table on the customer's behalf. Must be true for the current release.
CompressionType (string) --
The compression codec for Iceberg table data files. Defaults to ZSTD.
S3DestinationConfiguration (dict) --
The Amazon S3 destination for the channel, if configured.
DataFreshnessInSeconds (integer) --
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
DeadLetterQueueS3 (dict) --
The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
BucketArn (string) --
The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
ErrorOutputPrefix (string) --
An optional prefix prepended to every dead-letter Amazon S3 object key.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
ServiceExecutionRoleArn (string) --
The Amazon Resource Name (ARN) of the IAM role that MSK assumes to write to the destination Amazon S3 bucket and the dead-letter bucket.
Storage (dict) --
The Amazon S3 bucket, prefix, and storage class for delivered records.
BucketArn (string) --
The Amazon Resource Name (ARN) of the destination Amazon S3 bucket.
CompressionType (string) --
The compression codec applied to delivered Amazon S3 objects.
OutputPrefix (string) --
An optional prefix prepended to every Amazon S3 object key written by the channel.
OutputKeyTemplate (string) --
An optional template that controls the Amazon S3 object key for each delivered record. Supports the placeholders !{partition-id}, !{sequence-number}, and !{kafka-offset}.
StorageClass (string) --
The Amazon S3 storage class for delivered objects.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the Amazon S3 bucket.
Status (string) --
The current lifecycle state of the channel.
DestinationType (string) --
The type of destination configured for the channel.
CreationTime (datetime) --
The time when the channel was created.
TopicConfigurationList (list) --
The list of topic configurations for the channel.
(dict) --
Configuration of an Apache Kafka topic that feeds a channel.
RecordConverter (dict) --
Configuration that controls how Apache Kafka record values are deserialized for the destination.
ValueConverter (string) --
The deserialization format applied to Apache Kafka record values.
RecordSchema (dict) --
The schema used to validate records when the value converter requires one (for example, JSON_SCHEMA_GSR).
GsrArn (string) --
The Amazon Resource Name (ARN) of the AWS Glue Schema Registry schema (not registry) used to validate records for the destination Apache Iceberg table.
TopicArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the topic.
LoggingInfo (dict) --
The destinations to which the channel publishes operational logs.
CloudWatchLogs (dict) --
Details of the CloudWatch Logs destination for Channel logs.
Enabled (boolean) --
LogGroup (string) --
Firehose (dict) --
Details of the Kinesis Data Firehose delivery stream that is the destination for Channel logs.
DeliveryStream (string) --
Enabled (boolean) --
S3 (dict) --
Details of the Amazon S3 destination for Channel logs.
Bucket (string) --
Enabled (boolean) --
Prefix (string) --
StateInfo (dict) --
Additional context for the current channel state, populated when the channel is in FAILED.
Code (string) --
A short, machine-readable code identifying the failure cause.
Message (string) --
A human-readable message describing the failure.
ClusterOperationArn (string) --
The Amazon Resource Name (ARN) of the in-flight cluster operation. Returned only while the channel is in CREATING, UPDATING, or DELETING.
Tags (dict) --
The tags attached to the channel.
(string) --
(string) --
Creates a Channel that streams records from an Amazon MSK Express cluster topic to Amazon S3 or Apache Iceberg.
See also: AWS API Documentation
Request Syntax
client.create_channel(
ChannelName='string',
ClusterArn='string',
EncryptionConfiguration={
'KmsKeyArn': 'string'
},
IcebergDestinationConfiguration={
'AppendOnly': True|False,
'Catalog': {
'CatalogArn': 'string',
'WarehouseLocation': 'string'
},
'DataFreshnessInSeconds': 123,
'DeadLetterQueueS3': {
'BucketArn': 'string',
'ErrorOutputPrefix': 'string',
'ExpectedBucketOwner': 'string'
},
'DestinationTableList': [
{
'DestinationDatabaseName': 'string',
'DestinationTableName': 'string',
'PartitionSpec': {
'PartitionStrategy': 'TIME_HOUR',
'SourceList': [
{
'SourceName': 'string'
},
]
}
},
],
'SchemaEvolution': {
'EnableSchemaEvolution': True|False
},
'ServiceExecutionRoleArn': 'string',
'TableCreation': {
'EnableTableCreation': True|False
},
'CompressionType': 'ZSTD'|'SNAPPY'
},
S3DestinationConfiguration={
'DataFreshnessInSeconds': 123,
'DeadLetterQueueS3': {
'BucketArn': 'string',
'ErrorOutputPrefix': 'string',
'ExpectedBucketOwner': 'string'
},
'ServiceExecutionRoleArn': 'string',
'Storage': {
'BucketArn': 'string',
'CompressionType': 'NONE'|'GZIP'|'ZSTD',
'OutputPrefix': 'string',
'OutputKeyTemplate': 'string',
'StorageClass': 'STANDARD'|'INTELLIGENT_TIERING'|'GLACIER_IR',
'ExpectedBucketOwner': 'string'
}
},
Tags={
'string': 'string'
},
TopicConfigurationList=[
{
'RecordConverter': {
'ValueConverter': 'BYTE_ARRAY'|'JSON'|'JSON_SCHEMA_GSR'|'STRING'
},
'RecordSchema': {
'GsrArn': 'string'
},
'TopicArn': 'string'
},
],
LoggingInfo={
'CloudWatchLogs': {
'Enabled': True|False,
'LogGroup': 'string'
},
'Firehose': {
'DeliveryStream': 'string',
'Enabled': True|False
},
'S3': {
'Bucket': 'string',
'Enabled': True|False,
'Prefix': 'string'
}
}
)
string
[REQUIRED]
The name of the channel. Must be unique within the cluster.
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the cluster.
dict
The encryption configuration applied to the channel.
KmsKeyArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the AWS KMS key used to encrypt the data.
dict
The Apache Iceberg destination for the channel. Mutually exclusive with s3DestinationConfiguration.
AppendOnly (boolean) -- [REQUIRED]
Whether the destination is append-only. Must be true; updates and deletes are not supported.
Catalog (dict) --
The AWS Glue Data Catalog and S3 Tables warehouse used by the destination.
CatalogArn (string) --
The Amazon Resource Name (ARN) of the federated AWS Glue Data Catalog that projects the S3 Tables bucket. If omitted, MSK derives the catalog ARN from warehouseLocation.
WarehouseLocation (string) --
The Amazon Resource Name (ARN) of the S3 Tables bucket that backs the Apache Iceberg warehouse.
DataFreshnessInSeconds (integer) --
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
DeadLetterQueueS3 (dict) -- [REQUIRED]
The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
BucketArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
ErrorOutputPrefix (string) --
An optional prefix prepended to every dead-letter Amazon S3 object key.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
DestinationTableList (list) -- [REQUIRED]
The destination Iceberg tables. Currently exactly one table must be specified.
(dict) --
Configuration of an Apache Iceberg destination table.
DestinationDatabaseName (string) --
The name of the destination namespace (database) in the AWS Glue Data Catalog.
DestinationTableName (string) --
The name of the destination Apache Iceberg table.
PartitionSpec (dict) --
The partition specification for the destination table.
PartitionStrategy (string) -- [REQUIRED]
The partitioning strategy applied to records written to the table.
SourceList (list) --
The source columns used by the partitioning strategy. For TIME_HOUR, must contain exactly one source column whose value is a timestamp.
(dict) --
A source column used by an Apache Iceberg destination table's partition specification.
SourceName (string) --
Source name.
SchemaEvolution (dict) -- [REQUIRED]
Configuration controlling whether the destination table's schema is evolved to match incoming records.
EnableSchemaEvolution (boolean) --
Whether to allow MSK to evolve the destination table's schema. Must be false for the current release.
ServiceExecutionRoleArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the IAM role that MSK assumes to access the destination table, the AWS Glue Data Catalog, and the dead-letter Amazon S3 bucket.
TableCreation (dict) -- [REQUIRED]
Configuration controlling whether MSK creates the destination table if it does not already exist.
EnableTableCreation (boolean) --
Whether MSK creates the destination table on the customer's behalf. Must be true for the current release.
CompressionType (string) --
The compression codec for Iceberg table data files. Defaults to ZSTD.
dict
The Amazon S3 destination for the channel. Mutually exclusive with icebergDestinationConfiguration.
DataFreshnessInSeconds (integer) --
The maximum time, in seconds, that records buffer in MSK before being flushed to the destination. Allowed range: 300 to 900. Default: 600.
DeadLetterQueueS3 (dict) -- [REQUIRED]
The Amazon S3 bucket and prefix where MSK writes records that fail to deliver.
BucketArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the dead-letter Amazon S3 bucket.
ErrorOutputPrefix (string) --
An optional prefix prepended to every dead-letter Amazon S3 object key.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the dead-letter Amazon S3 bucket.
ServiceExecutionRoleArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the IAM role that MSK assumes to write to the destination Amazon S3 bucket and the dead-letter bucket.
Storage (dict) -- [REQUIRED]
The Amazon S3 bucket, prefix, and storage class for delivered records.
BucketArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the destination Amazon S3 bucket.
CompressionType (string) -- [REQUIRED]
The compression codec applied to delivered Amazon S3 objects.
OutputPrefix (string) --
An optional prefix prepended to every Amazon S3 object key written by the channel.
OutputKeyTemplate (string) --
An optional template that controls the Amazon S3 object key for each delivered record. Supports the placeholders !{partition-id}, !{sequence-number}, and !{kafka-offset}.
StorageClass (string) -- [REQUIRED]
The Amazon S3 storage class for delivered objects.
ExpectedBucketOwner (string) --
Optional 12-digit AWS account ID expected to own the Amazon S3 bucket.
dict
The tags attached to the channel.
(string) --
(string) --
list
[REQUIRED]
The list of topic configurations for the channel. Currently exactly one topic must be specified.
(dict) --
Configuration of an Apache Kafka topic that feeds a channel.
RecordConverter (dict) -- [REQUIRED]
Configuration that controls how Apache Kafka record values are deserialized for the destination.
ValueConverter (string) -- [REQUIRED]
The deserialization format applied to Apache Kafka record values.
RecordSchema (dict) --
The schema used to validate records when the value converter requires one (for example, JSON_SCHEMA_GSR).
GsrArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the AWS Glue Schema Registry schema (not registry) used to validate records for the destination Apache Iceberg table.
TopicArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the topic.
dict
The destinations to which the channel publishes operational logs.
CloudWatchLogs (dict) --
Details of the CloudWatch Logs destination for Channel logs.
Enabled (boolean) -- [REQUIRED]
LogGroup (string) --
Firehose (dict) --
Details of the Kinesis Data Firehose delivery stream that is the destination for Channel logs.
DeliveryStream (string) --
Enabled (boolean) -- [REQUIRED]
S3 (dict) --
Details of the Amazon S3 destination for Channel logs.
Bucket (string) --
Enabled (boolean) -- [REQUIRED]
Prefix (string) --
dict
Response Syntax
{
'ChannelArn': 'string',
'ClusterOperationArn': 'string'
}
Response Structure
(dict) --
200 response
ChannelArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the channel.
ClusterOperationArn (string) --
The Amazon Resource Name (ARN) of the cluster operation.
Returns the list of channels in a cluster.
See also: AWS API Documentation
Request Syntax
client.list_channels(
ClusterArn='string',
MaxResults=123,
NextToken='string',
TopicNameFilter='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the cluster.
integer
Maximum number of channels to return in a single response.
string
If the response of ListChannels is truncated, it returns a nextToken in the response. This nextToken should be sent in the subsequent request to ListChannels.
string
Filters results to channels whose topic name matches the specified value.
dict
Response Syntax
{
'Channels': [
{
'ChannelArn': 'string',
'ChannelName': 'string',
'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'FAILED'|'SUSPENDING'|'SUSPENDED',
'CreationTime': datetime(2015, 1, 1),
'DestinationType': 'ICEBERG'|'S3',
'ClusterOperationArn': 'string'
},
],
'NextToken': 'string'
}
Response Structure
(dict) --
Successful response.
Channels (list) --
The list of channels in the cluster.
(dict) --
Summary information about a channel returned by ListChannels.
ChannelArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the channel.
ChannelName (string) --
The name of the channel.
Status (string) --
The current lifecycle state of the channel.
CreationTime (datetime) --
The time when the channel was created.
DestinationType (string) --
The type of destination configured for the channel.
ClusterOperationArn (string) --
The Amazon Resource Name (ARN) of the in-flight cluster operation. Returned only while the channel is in CREATING, UPDATING, or DELETING.
NextToken (string) --
If the response from ListChannels is truncated, this token is included. Send it as the nextToken parameter on a subsequent ListChannels call to retrieve the next page.
Deletes the channel specified by channelArn from the cluster specified by clusterArn. The channel transitions through DELETING and is removed when the asynchronous delete completes.
See also: AWS API Documentation
Request Syntax
client.delete_channel(
ChannelArn='string',
ClusterArn='string'
)
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the channel.
string
[REQUIRED]
The Amazon Resource Name (ARN) that uniquely identifies the cluster.
dict
Response Syntax
{
'ChannelArn': 'string',
'ClusterOperationArn': 'string'
}
Response Structure
(dict) --
Successful response.
ChannelArn (string) --
The Amazon Resource Name (ARN) that uniquely identifies the channel.
ClusterOperationArn (string) --
The Amazon Resource Name (ARN) of the cluster operation.