Amazon SageMaker Service

2026/11/16 - Amazon SageMaker Service - 8 new11 updated api methods

Changes  Amazon SageMaker adds Synthetic Data Generation, Auto-labeling, Data Quality Evaluation, and Data Transformation capabilities, Cross-Region Inference Routing, KV Aware routing for realtime inference endpoints.

CreateInferenceRouter (new) Link ¶

Creates an inference router. An inference router is a Regional SageMaker AI resource that sends each inference request to one of the endpoints or inference components that you list for a routing target. These destinations can be in any Amazon Web Services account, and in any Amazon Web Services Region in the same partition as the router.

The router's status is Creating until it's ready to use. SageMaker AI checks that each endpoint is a real-time or asynchronous endpoint, as the routing target requires, and rejects one of the wrong type. It checks endpoints in other accounts with your credentials, so the endpoint policy must allow sagemaker:DescribeEndpoint. SageMaker AI accepts an endpoint that it can't check, for example one that doesn't exist. Requests to it fail if it's the wrong type or doesn't exist. To see the status of each destination, call DescribeInferenceRouter and see the RoutingStatus of each destination.

To invoke the router, use InvokeInferenceRouter or InvokeInferenceRouterAsync. For information about inference routers, see Route requests across Regions with inference routers.

See also: AWS API Documentation

Request Syntax

client.create_inference_router(
    InferenceRouterName='string',
    RoutingTargets=[
        {
            'Name': 'string',
            'Destinations': [
                {
                    'ResourceArn': 'string',
                    'RoutingStatus': 'Routable'|'Unroutable'|'Unknown'
                },
            ],
            'AsyncInferenceConfig': {
                'OutputConfig': {
                    'S3OutputPath': 'string',
                    'S3FailurePath': 'string',
                    'KmsKeyId': 'string',
                    'NotificationConfig': {
                        'SuccessTopic': 'string',
                        'ErrorTopic': 'string'
                    }
                },
                'ClientConfig': {
                    'MaxTimeInBacklogSeconds': 123
                }
            },
            'RoutingPreference': 'LocalRegionFirst'|'InstancePoolOrder'
        },
    ],
    ClientToken='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
type InferenceRouterName:

string

param InferenceRouterName:

[REQUIRED]

The name of the inference router. The name must be unique within your account in the current Amazon Web Services Region. The name is case sensitive.

type RoutingTargets:

list

param RoutingTargets:

[REQUIRED]

The routing targets of the inference router. Each routing target is a model that callers name when they invoke the router, and the destinations that serve it.

  • (dict) --

    A model that an inference router serves, and the destinations that can serve it. When you invoke the router, you name the routing target, and the router sends the request to one of its destinations.

    Every destination in a routing target must serve the same model, with the same request and response format.

    • Name (string) -- [REQUIRED]

      The name of the routing target. Callers specify this name in the RoutingTargetName parameter when they invoke the inference router. The name must be unique within the router.

    • Destinations (list) -- [REQUIRED]

      The endpoints and inference components that serve the routing target. Each destination can appear only once in a routing target.

      In an asynchronous routing target, every destination must be an asynchronous inference endpoint. In a real-time routing target, no destination can be one. SageMaker AI checks an endpoint in another account with your credentials, so both your IAM policy and the endpoint policy must allow sagemaker:DescribeEndpoint. If an endpoint can't be checked, for example because it isn't found or can't be read in time, it's accepted, and requests to it fail if it's the wrong mode.

      • (dict) --

        An endpoint or inference component that serves a routing target.

        • ResourceArn (string) -- [REQUIRED]

          The Amazon Resource Name (ARN) of the endpoint or inference component. The destination can be in any account, and in any Amazon Web Services Region in the same partition as the inference router. To use a destination in another account, its owner must allow your account to invoke it through your router, in an endpoint policy.

        • RoutingStatus (string) --

          Whether the inference router can send requests to the destination, as checked when you called DescribeInferenceRouter. This value isn't the destination's own status. SageMaker AI ignores this field in CreateInferenceRouter and UpdateInferenceRouter requests.

          Routable

          The destination is serving and the router can send it requests. A routable destination can still be overloaded.

          Unroutable

          The destination can't serve requests. For example, it's being created or has failed, or it has no capacity and no way to get any. A destination in your account that doesn't exist is also Unroutable.

          Unknown

          SageMaker AI couldn't check the destination, for example because you aren't allowed to describe it or the check timed out. The destination might still be serving.

    • AsyncInferenceConfig (dict) --

      The asynchronous inference settings of the routing target. Specify this setting to make the routing target asynchronous. If you omit it, the routing target serves real-time requests.

      Asynchronous routing targets aren't available in all Amazon Web Services Regions. For information about asynchronous routing targets and Region availability, see Send asynchronous requests through an inference router.

      Because UpdateInferenceRouter replaces all routing targets, include this setting in every update of an asynchronous routing target.

      For information about asynchronous routing targets, see Send asynchronous requests through an inference router.

      • OutputConfig (dict) -- [REQUIRED]

        The Amazon S3 locations for the results of asynchronous requests, and the notification settings.

        • S3OutputPath (string) -- [REQUIRED]

          The Amazon S3 location for the results of successful requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

        • S3FailurePath (string) -- [REQUIRED]

          The Amazon S3 location for the failure details of failed requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

        • KmsKeyId (string) --

          The KMS key that encrypts the results in Amazon S3. You can specify a key ID, key ARN, alias name, or alias ARN. If any destination is in a different account or Amazon Web Services Region from the inference router, specify a key ARN or an alias ARN.

        • NotificationConfig (dict) --

          The Amazon SNS topics that the inference router publishes to when a request completes or fails.

          • SuccessTopic (string) --

            The ARN of the Amazon SNS topic to publish to when a request completes. The topic must be a standard topic, and can be in any Amazon Web Services Region.

          • ErrorTopic (string) --

            The ARN of the Amazon SNS topic to publish to when a request fails, or when the router can't place it on any destination. The topic must be a standard topic, and can be in any Amazon Web Services Region.

      • ClientConfig (dict) --

        How the inference router submits requests to destinations.

        • MaxTimeInBacklogSeconds (integer) --

          The maximum time, in seconds, that a request can wait at a destination before the inference router moves it to another destination. The default is 900 seconds.

    • RoutingPreference (string) --

      How the inference router chooses among the routing target's destinations. The default is LocalRegionFirst.

      LocalRegionFirst

      The router uses destinations in its own Region first. When the request leaves the Region, the router spreads it across healthy destinations in other Regions.

      InstancePoolOrder

      The router prefers destinations by instance type first and by Region second. Every destination must be an endpoint that lists the same two or more instance types, each with a different priority, in the same priority order. This value is supported only for asynchronous routing targets.

      With either value, the router uses destinations whose health or capacity is reduced after the others, but doesn't exclude them.

type ClientToken:

string

param ClientToken:

A unique, case-sensitive identifier that you provide to ensure the idempotency of the request. If you reuse a client token, use the same request parameters. Otherwise, the request fails.

This field is autopopulated if not provided.

type Tags:

list

param Tags:

A list of key-value pairs to apply to the inference router. For information about tagging, see Tagging Amazon Web Services resources in the Amazon Web Services General Reference.

  • (dict) --

    A tag object that consists of a key and an optional value, used to manage metadata for SageMaker Amazon Web Services resources.

    You can add tags to notebook instances, training jobs, hyperparameter tuning jobs, batch transform jobs, models, labeling jobs, work teams, endpoint configurations, and endpoints. For more information on adding tags to SageMaker resources, see AddTags.

    For more information on adding metadata to your Amazon Web Services resources with tagging, see Tagging Amazon Web Services resources. For advice on best practices for managing Amazon Web Services resources with tagging, see Tagging Best Practices: Implement an Effective Amazon Web Services Resource Tagging Strategy.

    • Key (string) -- [REQUIRED]

      The tag key. Tag keys must be unique per resource.

    • Value (string) -- [REQUIRED]

      The tag value.

rtype:

dict

returns:

Response Syntax

{
    'InferenceRouterArn': 'string'
}

Response Structure

  • (dict) --

    • InferenceRouterArn (string) --

      The Amazon Resource Name (ARN) of the inference router.

PutEndpointPolicy (new) Link ¶

Attaches a resource-based policy to an endpoint. With an endpoint policy, you can allow other Amazon Web Services accounts to send requests to your endpoint, or to inference components on it, through an inference router. The policy that you specify replaces the endpoint's existing policy, if it has one.

You can't attach a policy to an endpoint whose status is Deleting or Failed. SageMaker AI rejects a policy that doesn't follow the endpoint policy rules. For information about the policy rules, see Use destinations in other accounts.

Policy changes typically take effect within 30 seconds, and can take up to 5 minutes. If you delete the endpoint, its policy is also deleted, and a new endpoint with the same name has no policy.

See also: AWS API Documentation

Request Syntax

client.put_endpoint_policy(
    EndpointName='string',
    ResourcePolicy='string'
)
type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the endpoint. The endpoint must be in your account and in the current Amazon Web Services Region.

type ResourcePolicy:

string

param ResourcePolicy:

[REQUIRED]

The endpoint policy, as a JSON policy document. The policy can contain only ASCII characters. For information about the policy rules, see Use destinations in other accounts.

rtype:

dict

returns:

Response Syntax

{
    'EndpointArn': 'string'
}

Response Structure

  • (dict) --

    • EndpointArn (string) --

      The Amazon Resource Name (ARN) of the endpoint.

ListInferenceRouters (new) Link ¶

Lists the inference routers in your account in the current Amazon Web Services Region. A router that you just created might not appear in the list right away.

See also: AWS API Documentation

Request Syntax

client.list_inference_routers(
    MaxResults=123,
    NextToken='string'
)
type MaxResults:

integer

param MaxResults:

The maximum number of inference routers to return in the response.

type NextToken:

string

param NextToken:

If the previous response was truncated, you receive this token. To get the next set of inference routers, use it in your next request.

rtype:

dict

returns:

Response Syntax

{
    'InferenceRouters': [
        {
            'InferenceRouterName': 'string',
            'InferenceRouterArn': 'string',
            'CreationTime': datetime(2015, 1, 1),
            'LastModifiedTime': datetime(2015, 1, 1)
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) --

    • InferenceRouters (list) --

      A list of summaries of the inference routers.

      • (dict) --

        A summary of an inference router.

        • InferenceRouterName (string) --

          The name of the inference router.

        • InferenceRouterArn (string) --

          The Amazon Resource Name (ARN) of the inference router.

        • CreationTime (datetime) --

          The date and time that the inference router was created.

        • LastModifiedTime (datetime) --

          The date and time that the inference router was last updated.

    • NextToken (string) --

      If the response is truncated, SageMaker AI returns this token. To get the next set of inference routers, use it in your next request.

DeleteInferenceRouter (new) Link ¶

Deletes an inference router. The endpoints and inference components that the router sends requests to aren't deleted.

If the router is being created, updated, or deleted, you get a ConflictException.

See also: AWS API Documentation

Request Syntax

client.delete_inference_router(
    InferenceRouterName='string'
)
type InferenceRouterName:

string

param InferenceRouterName:

[REQUIRED]

The name of the inference router to delete.

returns:

None

GetEndpointPolicy (new) Link ¶

Returns the resource-based policy that is attached to an endpoint. If the endpoint doesn't have a policy, you get a ValidationException.

See also: AWS API Documentation

Request Syntax

client.get_endpoint_policy(
    EndpointName='string'
)
type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the endpoint. The endpoint must be in your account and in the current Amazon Web Services Region.

rtype:

dict

returns:

Response Syntax

{
    'EndpointName': 'string',
    'EndpointArn': 'string',
    'ResourcePolicy': 'string'
}

Response Structure

  • (dict) --

    • EndpointName (string) --

      The name of the endpoint.

    • EndpointArn (string) --

      The Amazon Resource Name (ARN) of the endpoint.

    • ResourcePolicy (string) --

      The endpoint policy, as a JSON policy document.

UpdateInferenceRouter (new) Link ¶

Updates the routing targets of an inference router. The routing targets that you specify replace all of the router's existing routing targets. To change one routing target, call DescribeInferenceRouter, edit the routing targets, and send all of them.

The update is applied in full or not at all, and each request uses either the previous configuration or the new one. Updates typically take effect within about 2 minutes.

If the router is being created, updated, or deleted, or another request changes it at the same time, you get a ConflictException. Describe the router, and then send your update again.

See also: AWS API Documentation

Request Syntax

client.update_inference_router(
    InferenceRouterName='string',
    RoutingTargets=[
        {
            'Name': 'string',
            'Destinations': [
                {
                    'ResourceArn': 'string',
                    'RoutingStatus': 'Routable'|'Unroutable'|'Unknown'
                },
            ],
            'AsyncInferenceConfig': {
                'OutputConfig': {
                    'S3OutputPath': 'string',
                    'S3FailurePath': 'string',
                    'KmsKeyId': 'string',
                    'NotificationConfig': {
                        'SuccessTopic': 'string',
                        'ErrorTopic': 'string'
                    }
                },
                'ClientConfig': {
                    'MaxTimeInBacklogSeconds': 123
                }
            },
            'RoutingPreference': 'LocalRegionFirst'|'InstancePoolOrder'
        },
    ]
)
type InferenceRouterName:

string

param InferenceRouterName:

[REQUIRED]

The name of the inference router to update.

type RoutingTargets:

list

param RoutingTargets:

[REQUIRED]

The complete list of routing targets for the inference router. To remove the last routing target, delete the router.

  • (dict) --

    A model that an inference router serves, and the destinations that can serve it. When you invoke the router, you name the routing target, and the router sends the request to one of its destinations.

    Every destination in a routing target must serve the same model, with the same request and response format.

    • Name (string) -- [REQUIRED]

      The name of the routing target. Callers specify this name in the RoutingTargetName parameter when they invoke the inference router. The name must be unique within the router.

    • Destinations (list) -- [REQUIRED]

      The endpoints and inference components that serve the routing target. Each destination can appear only once in a routing target.

      In an asynchronous routing target, every destination must be an asynchronous inference endpoint. In a real-time routing target, no destination can be one. SageMaker AI checks an endpoint in another account with your credentials, so both your IAM policy and the endpoint policy must allow sagemaker:DescribeEndpoint. If an endpoint can't be checked, for example because it isn't found or can't be read in time, it's accepted, and requests to it fail if it's the wrong mode.

      • (dict) --

        An endpoint or inference component that serves a routing target.

        • ResourceArn (string) -- [REQUIRED]

          The Amazon Resource Name (ARN) of the endpoint or inference component. The destination can be in any account, and in any Amazon Web Services Region in the same partition as the inference router. To use a destination in another account, its owner must allow your account to invoke it through your router, in an endpoint policy.

        • RoutingStatus (string) --

          Whether the inference router can send requests to the destination, as checked when you called DescribeInferenceRouter. This value isn't the destination's own status. SageMaker AI ignores this field in CreateInferenceRouter and UpdateInferenceRouter requests.

          Routable

          The destination is serving and the router can send it requests. A routable destination can still be overloaded.

          Unroutable

          The destination can't serve requests. For example, it's being created or has failed, or it has no capacity and no way to get any. A destination in your account that doesn't exist is also Unroutable.

          Unknown

          SageMaker AI couldn't check the destination, for example because you aren't allowed to describe it or the check timed out. The destination might still be serving.

    • AsyncInferenceConfig (dict) --

      The asynchronous inference settings of the routing target. Specify this setting to make the routing target asynchronous. If you omit it, the routing target serves real-time requests.

      Asynchronous routing targets aren't available in all Amazon Web Services Regions. For information about asynchronous routing targets and Region availability, see Send asynchronous requests through an inference router.

      Because UpdateInferenceRouter replaces all routing targets, include this setting in every update of an asynchronous routing target.

      For information about asynchronous routing targets, see Send asynchronous requests through an inference router.

      • OutputConfig (dict) -- [REQUIRED]

        The Amazon S3 locations for the results of asynchronous requests, and the notification settings.

        • S3OutputPath (string) -- [REQUIRED]

          The Amazon S3 location for the results of successful requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

        • S3FailurePath (string) -- [REQUIRED]

          The Amazon S3 location for the failure details of failed requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

        • KmsKeyId (string) --

          The KMS key that encrypts the results in Amazon S3. You can specify a key ID, key ARN, alias name, or alias ARN. If any destination is in a different account or Amazon Web Services Region from the inference router, specify a key ARN or an alias ARN.

        • NotificationConfig (dict) --

          The Amazon SNS topics that the inference router publishes to when a request completes or fails.

          • SuccessTopic (string) --

            The ARN of the Amazon SNS topic to publish to when a request completes. The topic must be a standard topic, and can be in any Amazon Web Services Region.

          • ErrorTopic (string) --

            The ARN of the Amazon SNS topic to publish to when a request fails, or when the router can't place it on any destination. The topic must be a standard topic, and can be in any Amazon Web Services Region.

      • ClientConfig (dict) --

        How the inference router submits requests to destinations.

        • MaxTimeInBacklogSeconds (integer) --

          The maximum time, in seconds, that a request can wait at a destination before the inference router moves it to another destination. The default is 900 seconds.

    • RoutingPreference (string) --

      How the inference router chooses among the routing target's destinations. The default is LocalRegionFirst.

      LocalRegionFirst

      The router uses destinations in its own Region first. When the request leaves the Region, the router spreads it across healthy destinations in other Regions.

      InstancePoolOrder

      The router prefers destinations by instance type first and by Region second. Every destination must be an endpoint that lists the same two or more instance types, each with a different priority, in the same priority order. This value is supported only for asynchronous routing targets.

      With either value, the router uses destinations whose health or capacity is reduced after the others, but doesn't exclude them.

rtype:

dict

returns:

Response Syntax

{
    'InferenceRouterArn': 'string'
}

Response Structure

  • (dict) --

    • InferenceRouterArn (string) --

      The Amazon Resource Name (ARN) of the inference router.

DeleteEndpointPolicy (new) Link ¶

Deletes the resource-based policy that is attached to an endpoint. After you delete the policy, other accounts can no longer send requests to the endpoint through an inference router. The endpoint continues to serve requests from your account. If the endpoint doesn't have a policy, the operation succeeds.

The change typically takes effect within 30 seconds, and can take up to 5 minutes. To revoke access for one account only, use PutEndpointPolicy to replace the policy with one that doesn't include that account.

See also: AWS API Documentation

Request Syntax

client.delete_endpoint_policy(
    EndpointName='string'
)
type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the endpoint. The endpoint must be in your account and in the current Amazon Web Services Region.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

  • (dict) --

DescribeInferenceRouter (new) Link ¶

Returns information about an inference router, including the routing status of each destination.

SageMaker AI checks each destination with your permissions when you call this operation, so callers with different permissions can see a different RoutingStatus. To get the router's tags, use ListTags.

See also: AWS API Documentation

Request Syntax

client.describe_inference_router(
    InferenceRouterName='string'
)
type InferenceRouterName:

string

param InferenceRouterName:

[REQUIRED]

The name of the inference router to describe.

rtype:

dict

returns:

Response Syntax

{
    'InferenceRouterName': 'string',
    'InferenceRouterArn': 'string',
    'RoutingTargets': [
        {
            'Name': 'string',
            'Destinations': [
                {
                    'ResourceArn': 'string',
                    'RoutingStatus': 'Routable'|'Unroutable'|'Unknown'
                },
            ],
            'AsyncInferenceConfig': {
                'OutputConfig': {
                    'S3OutputPath': 'string',
                    'S3FailurePath': 'string',
                    'KmsKeyId': 'string',
                    'NotificationConfig': {
                        'SuccessTopic': 'string',
                        'ErrorTopic': 'string'
                    }
                },
                'ClientConfig': {
                    'MaxTimeInBacklogSeconds': 123
                }
            },
            'RoutingPreference': 'LocalRegionFirst'|'InstancePoolOrder'
        },
    ],
    'Status': 'InService'|'Creating'|'Updating'|'Failed'|'Deleting',
    'StatusReason': 'string',
    'CreationTime': datetime(2015, 1, 1),
    'LastModifiedTime': datetime(2015, 1, 1)
}

Response Structure

  • (dict) --

    • InferenceRouterName (string) --

      The name of the inference router.

    • InferenceRouterArn (string) --

      The Amazon Resource Name (ARN) of the inference router.

    • RoutingTargets (list) --

      The routing targets of the inference router. Each destination includes its RoutingStatus. You can send these routing targets back in an UpdateInferenceRouter request without changes.

      • (dict) --

        A model that an inference router serves, and the destinations that can serve it. When you invoke the router, you name the routing target, and the router sends the request to one of its destinations.

        Every destination in a routing target must serve the same model, with the same request and response format.

        • Name (string) --

          The name of the routing target. Callers specify this name in the RoutingTargetName parameter when they invoke the inference router. The name must be unique within the router.

        • Destinations (list) --

          The endpoints and inference components that serve the routing target. Each destination can appear only once in a routing target.

          In an asynchronous routing target, every destination must be an asynchronous inference endpoint. In a real-time routing target, no destination can be one. SageMaker AI checks an endpoint in another account with your credentials, so both your IAM policy and the endpoint policy must allow sagemaker:DescribeEndpoint. If an endpoint can't be checked, for example because it isn't found or can't be read in time, it's accepted, and requests to it fail if it's the wrong mode.

          • (dict) --

            An endpoint or inference component that serves a routing target.

            • ResourceArn (string) --

              The Amazon Resource Name (ARN) of the endpoint or inference component. The destination can be in any account, and in any Amazon Web Services Region in the same partition as the inference router. To use a destination in another account, its owner must allow your account to invoke it through your router, in an endpoint policy.

            • RoutingStatus (string) --

              Whether the inference router can send requests to the destination, as checked when you called DescribeInferenceRouter. This value isn't the destination's own status. SageMaker AI ignores this field in CreateInferenceRouter and UpdateInferenceRouter requests.

              Routable

              The destination is serving and the router can send it requests. A routable destination can still be overloaded.

              Unroutable

              The destination can't serve requests. For example, it's being created or has failed, or it has no capacity and no way to get any. A destination in your account that doesn't exist is also Unroutable.

              Unknown

              SageMaker AI couldn't check the destination, for example because you aren't allowed to describe it or the check timed out. The destination might still be serving.

        • AsyncInferenceConfig (dict) --

          The asynchronous inference settings of the routing target. Specify this setting to make the routing target asynchronous. If you omit it, the routing target serves real-time requests.

          Asynchronous routing targets aren't available in all Amazon Web Services Regions. For information about asynchronous routing targets and Region availability, see Send asynchronous requests through an inference router.

          Because UpdateInferenceRouter replaces all routing targets, include this setting in every update of an asynchronous routing target.

          For information about asynchronous routing targets, see Send asynchronous requests through an inference router.

          • OutputConfig (dict) --

            The Amazon S3 locations for the results of asynchronous requests, and the notification settings.

            • S3OutputPath (string) --

              The Amazon S3 location for the results of successful requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

            • S3FailurePath (string) --

              The Amazon S3 location for the failure details of failed requests. The bucket must be in the inference router's Amazon Web Services Region and owned by the router's account.

            • KmsKeyId (string) --

              The KMS key that encrypts the results in Amazon S3. You can specify a key ID, key ARN, alias name, or alias ARN. If any destination is in a different account or Amazon Web Services Region from the inference router, specify a key ARN or an alias ARN.

            • NotificationConfig (dict) --

              The Amazon SNS topics that the inference router publishes to when a request completes or fails.

              • SuccessTopic (string) --

                The ARN of the Amazon SNS topic to publish to when a request completes. The topic must be a standard topic, and can be in any Amazon Web Services Region.

              • ErrorTopic (string) --

                The ARN of the Amazon SNS topic to publish to when a request fails, or when the router can't place it on any destination. The topic must be a standard topic, and can be in any Amazon Web Services Region.

          • ClientConfig (dict) --

            How the inference router submits requests to destinations.

            • MaxTimeInBacklogSeconds (integer) --

              The maximum time, in seconds, that a request can wait at a destination before the inference router moves it to another destination. The default is 900 seconds.

        • RoutingPreference (string) --

          How the inference router chooses among the routing target's destinations. The default is LocalRegionFirst.

          LocalRegionFirst

          The router uses destinations in its own Region first. When the request leaves the Region, the router spreads it across healthy destinations in other Regions.

          InstancePoolOrder

          The router prefers destinations by instance type first and by Region second. Every destination must be an endpoint that lists the same two or more instance types, each with a different priority, in the same priority order. This value is supported only for asynchronous routing targets.

          With either value, the router uses destinations whose health or capacity is reduced after the others, but doesn't exclude them.

    • Status (string) --

      The status of the inference router.

    • StatusReason (string) --

      If the status is Failed, the reason that the inference router failed.

    • CreationTime (datetime) --

      The date and time that the inference router was created.

    • LastModifiedTime (datetime) --

      The date and time that the inference router was last updated.

CreateEndpointConfig (updated) Link ¶
Changes (request)
{'CrossRegionTrafficConfig': {'MaxCrossRegionPercentPerEndpoint': 'integer',
                              'MaxInvocationsPerSecondPerInstance': 'integer'},
 'ProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}},
 'ShadowProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}}}

Creates an endpoint configuration that SageMaker hosting services uses to deploy models. In the configuration, you identify one or more models, created using the CreateModel API, to deploy and the resources that you want SageMaker to provision. Then you call the CreateEndpoint API.

In the request, you define a ProductionVariant, for each model that you want to deploy. Each ProductionVariant parameter also describes the resources that you want SageMaker to provision. This includes the number and type of ML compute instances to deploy.

If you are hosting multiple models, you also assign a VariantWeight to specify how much traffic you want to allocate to each model. For example, suppose that you want to host two models, A and B, and you assign traffic weight 2 for model A and 1 for model B. SageMaker distributes two-thirds of the traffic to Model A, and one-third to model B.

See also: AWS API Documentation

Request Syntax

client.create_endpoint_config(
    EndpointConfigName='string',
    ProductionVariants=[
        {
            'VariantName': 'string',
            'ModelName': 'string',
            'InitialInstanceCount': 123,
            'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'ModelNameOverride': 'string',
                    'Priority': 123
                },
            ],
            'VariantInstanceProvisionTimeoutInSeconds': 123,
            'InitialVariantWeight': ...,
            'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
            'CoreDumpConfig': {
                'DestinationS3Uri': 'string',
                'KmsKeyId': 'string'
            },
            'ServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'VolumeSizeInGB': 123,
            'ModelDataDownloadTimeoutInSeconds': 123,
            'ContainerStartupHealthCheckTimeoutInSeconds': 123,
            'EnableSSMAccess': True|False,
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'InferenceAmiVersion': 'al2-ami-sagemaker-inference-gpu-2'|'al2-ami-sagemaker-inference-gpu-2-1'|'al2-ami-sagemaker-inference-gpu-3-1'|'al2-ami-sagemaker-inference-neuron-2'|'al2023-ami-sagemaker-inference-gpu-4-1',
            'CapacityReservationConfig': {
                'CapacityReservationPreference': 'capacity-reservations-only',
                'MlReservationArn': 'string'
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    DataCaptureConfig={
        'EnableCapture': True|False,
        'InitialSamplingPercentage': 123,
        'DestinationS3Uri': 'string',
        'KmsKeyId': 'string',
        'CaptureOptions': [
            {
                'CaptureMode': 'Input'|'Output'|'InputAndOutput'
            },
        ],
        'CaptureContentTypeHeader': {
            'CsvContentTypes': [
                'string',
            ],
            'JsonContentTypes': [
                'string',
            ]
        }
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ],
    KmsKeyId='string',
    AsyncInferenceConfig={
        'ClientConfig': {
            'MaxConcurrentInvocationsPerInstance': 123
        },
        'OutputConfig': {
            'KmsKeyId': 'string',
            'S3OutputPath': 'string',
            'NotificationConfig': {
                'SuccessTopic': 'string',
                'ErrorTopic': 'string',
                'IncludeInferenceResponseIn': [
                    'SUCCESS_NOTIFICATION_TOPIC'|'ERROR_NOTIFICATION_TOPIC',
                ]
            },
            'S3FailurePath': 'string'
        }
    },
    ExplainerConfig={
        'ClarifyExplainerConfig': {
            'EnableExplanations': 'string',
            'InferenceConfig': {
                'FeaturesAttribute': 'string',
                'ContentTemplate': 'string',
                'MaxRecordCount': 123,
                'MaxPayloadInMB': 123,
                'ProbabilityIndex': 123,
                'LabelIndex': 123,
                'ProbabilityAttribute': 'string',
                'LabelAttribute': 'string',
                'LabelHeaders': [
                    'string',
                ],
                'FeatureHeaders': [
                    'string',
                ],
                'FeatureTypes': [
                    'numerical'|'categorical'|'text',
                ]
            },
            'ShapConfig': {
                'ShapBaselineConfig': {
                    'MimeType': 'string',
                    'ShapBaseline': 'string',
                    'ShapBaselineUri': 'string'
                },
                'NumberOfSamples': 123,
                'UseLogit': True|False,
                'Seed': 123,
                'TextConfig': {
                    'Language': 'af'|'sq'|'ar'|'hy'|'eu'|'bn'|'bg'|'ca'|'zh'|'hr'|'cs'|'da'|'nl'|'en'|'et'|'fi'|'fr'|'de'|'el'|'gu'|'he'|'hi'|'hu'|'is'|'id'|'ga'|'it'|'kn'|'ky'|'lv'|'lt'|'lb'|'mk'|'ml'|'mr'|'ne'|'nb'|'fa'|'pl'|'pt'|'ro'|'ru'|'sa'|'sr'|'tn'|'si'|'sk'|'sl'|'es'|'sv'|'tl'|'ta'|'tt'|'te'|'tr'|'uk'|'ur'|'yo'|'lij'|'xx',
                    'Granularity': 'token'|'sentence'|'paragraph'
                }
            }
        }
    },
    ShadowProductionVariants=[
        {
            'VariantName': 'string',
            'ModelName': 'string',
            'InitialInstanceCount': 123,
            'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'ModelNameOverride': 'string',
                    'Priority': 123
                },
            ],
            'VariantInstanceProvisionTimeoutInSeconds': 123,
            'InitialVariantWeight': ...,
            'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
            'CoreDumpConfig': {
                'DestinationS3Uri': 'string',
                'KmsKeyId': 'string'
            },
            'ServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'VolumeSizeInGB': 123,
            'ModelDataDownloadTimeoutInSeconds': 123,
            'ContainerStartupHealthCheckTimeoutInSeconds': 123,
            'EnableSSMAccess': True|False,
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'InferenceAmiVersion': 'al2-ami-sagemaker-inference-gpu-2'|'al2-ami-sagemaker-inference-gpu-2-1'|'al2-ami-sagemaker-inference-gpu-3-1'|'al2-ami-sagemaker-inference-neuron-2'|'al2023-ami-sagemaker-inference-gpu-4-1',
            'CapacityReservationConfig': {
                'CapacityReservationPreference': 'capacity-reservations-only',
                'MlReservationArn': 'string'
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    ExecutionRoleArn='string',
    VpcConfig={
        'SecurityGroupIds': [
            'string',
        ],
        'Subnets': [
            'string',
        ]
    },
    EnableNetworkIsolation=True|False,
    MetricsConfig={
        'EnableEnhancedMetrics': True|False,
        'EnableDetailedObservability': True|False,
        'MetricPublishFrequencyInSeconds': 123
    },
    CrossRegionTrafficConfig={
        'MaxInvocationsPerSecondPerInstance': 123,
        'MaxCrossRegionPercentPerEndpoint': 123
    }
)
type EndpointConfigName:

string

param EndpointConfigName:

[REQUIRED]

The name of the endpoint configuration. You specify this name in a CreateEndpoint request.

type ProductionVariants:

list

param ProductionVariants:

[REQUIRED]

An array of ProductionVariant objects, one for each model that you want to host at this endpoint.

  • (dict) --

    Identifies a model that you want to host and the resources chosen to deploy for hosting it. If you are deploying multiple models, tell SageMaker how to distribute traffic among the models by specifying variant weights. For more information on production variants, check Production variants.

    • VariantName (string) -- [REQUIRED]

      The name of the production variant.

    • ModelName (string) --

      The name of the model that you want to host. This is the name that you specified when creating the model.

    • InitialInstanceCount (integer) --

      Number of instances to launch initially.

    • InstanceType (string) --

      The ML compute instance type.

    • InstancePools (list) --

      A list of instance pools for the production variant. Each instance pool specifies an instance type and its priority for provisioning. Use instance pools to configure heterogeneous endpoints that deploy models across multiple instance types.

      • (dict) --

        Specifies an instance type and its priority for a heterogeneous endpoint. Use instance pools to configure a production variant with multiple instance types, enabling the endpoint to provision instances across different types based on priority.

        • InstanceType (string) -- [REQUIRED]

          The ML compute instance type for the instance pool.

        • ModelNameOverride (string) --

          The name of a SageMaker model to use for this instance pool instead of the model specified for the production variant. Use this to deploy a different model optimized for the instance type in this pool.

        • Priority (integer) -- [REQUIRED]

          The priority for the instance pool. SageMaker attempts to provision instances in order of priority, starting with the lowest value. If instances for a higher-priority pool are unavailable, SageMaker attempts to provision from the next pool.

          Valid values: 1 to 5, where 1 is the highest priority.

    • VariantInstanceProvisionTimeoutInSeconds (integer) --

      The timeout value, in seconds, for provisioning instances for the production variant. When SageMaker encounters an insufficient capacity error while provisioning instances, it retries with the next instance pool (if configured) or waits until the timeout expires. This timeout applies only to capacity provisioning and does not include the time for model download or container startup.

      Valid values: 300 to 3600.

    • InitialVariantWeight (float) --

      Determines initial traffic distribution among all of the models that you specify in the endpoint configuration. The traffic to a production variant is determined by the ratio of the VariantWeight to the sum of all VariantWeight values across all ProductionVariants. If unspecified, it defaults to 1.0.

    • AcceleratorType (string) --

      This parameter is no longer supported. Elastic Inference (EI) is no longer available.

      This parameter was used to specify the size of the EI instance to use for the production variant.

    • CoreDumpConfig (dict) --

      Specifies configuration for a core dump from the model container when the process crashes.

      • DestinationS3Uri (string) -- [REQUIRED]

        The Amazon S3 bucket to send the core dump to.

      • KmsKeyId (string) --

        The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the core dump data at rest using Amazon S3 server-side encryption. The KmsKeyId can be any of the following formats:

        • // KMS Key ID "1234abcd-12ab-34cd-56ef-1234567890ab"

        • // Amazon Resource Name (ARN) of a KMS Key "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"

        • // KMS Key Alias "alias/ExampleAlias"

        • // Amazon Resource Name (ARN) of a KMS Key Alias "arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias"

        If you use a KMS key ID or an alias of your KMS key, the SageMaker execution role must include permissions to call kms:Encrypt. If you don't provide a KMS key ID, SageMaker uses the default KMS key for Amazon S3 for your role's account. SageMaker uses server-side encryption with KMS-managed keys for OutputDataConfig. If you use a bucket policy with an s3:PutObject permission that only allows objects with server-side encryption, set the condition key of s3:x-amz-server-side-encryption to "aws:kms". For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.

        The KMS key policy must grant permission to the IAM role that you specify in your CreateEndpoint and UpdateEndpoint requests. For more information, see Using Key Policies in Amazon Web Services KMS in the Amazon Web Services Key Management Service Developer Guide.

    • ServerlessConfig (dict) --

      The serverless configuration for an endpoint. Specifies a serverless endpoint configuration instead of an instance-based endpoint configuration.

      • MemorySizeInMB (integer) -- [REQUIRED]

        The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

      • MaxConcurrency (integer) -- [REQUIRED]

        The maximum number of concurrent invocations your serverless endpoint can process.

      • ProvisionedConcurrency (integer) --

        The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

    • VolumeSizeInGB (integer) --

      The size, in GB, of the ML storage volume attached to individual inference instance associated with the production variant. Currently only Amazon EBS gp2 storage volumes are supported.

    • ModelDataDownloadTimeoutInSeconds (integer) --

      The timeout value, in seconds, to download and extract the model that you want to host from Amazon S3 to the individual inference instance associated with this production variant.

    • ContainerStartupHealthCheckTimeoutInSeconds (integer) --

      The timeout value, in seconds, for your inference container to pass health check by SageMaker Hosting. For more information about health check, see How Your Container Should Respond to Health Check (Ping) Requests.

    • EnableSSMAccess (boolean) --

      You can use this parameter to turn on native Amazon Web Services Systems Manager (SSM) access for a production variant behind an endpoint. By default, SSM access is disabled for all production variants behind an endpoint. You can turn on or turn off SSM access for a production variant behind an existing endpoint by creating a new endpoint configuration and calling UpdateEndpoint.

    • ManagedInstanceScaling (dict) --

      Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

      • Status (string) --

        Indicates whether managed instance scaling is enabled.

      • MinInstanceCount (integer) --

        The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

      • MaxInstanceCount (integer) --

        The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

      • ScaleInPolicy (dict) --

        Configures the scale-in behavior for managed instance scaling.

        • Strategy (string) -- [REQUIRED]

          The strategy for scaling in instances.

          IDLE_RELEASE

          Releases instances that have no hosted inference component copies.

          CONSOLIDATION

          Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

        • MaximumStepSize (integer) --

          The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

          Default value: 1.

        • CooldownInMinutes (integer) --

          The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

          Default value: 20.

    • RoutingConfig (dict) --

      Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

      • RoutingStrategy (string) -- [REQUIRED]

        Sets how the endpoint routes incoming traffic:

        • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

        • RANDOM: The endpoint routes each request to a randomly chosen instance.

        • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

        • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      • PrefixAwareRoutingConfig (dict) --

        The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

        • PrefixLength (integer) --

          The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

          • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

          • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

          The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

        • ConcurrencyThreshold (integer) --

          The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

          On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

          On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

          The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

    • InferenceAmiVersion (string) --

      Specifies an option from a collection of preconfigured Amazon Machine Image (AMI) images. Each image is configured by Amazon Web Services with a set of software and driver versions. Amazon Web Services optimizes these configurations for different machine learning workloads.

      By selecting an AMI version, you can ensure that your inference environment is compatible with specific software requirements, such as CUDA driver versions, Linux kernel versions, or Amazon Web Services Neuron driver versions.

      The AMI version names, and their configurations, are the following:

      al2-ami-sagemaker-inference-gpu-2

      • Accelerator: GPU

      • NVIDIA driver version: 535

      • CUDA version: 12.2

        al2-ami-sagemaker-inference-gpu-2-1

      • Accelerator: GPU

      • NVIDIA driver version: 535

      • CUDA version: 12.2

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2-ami-sagemaker-inference-gpu-3-1

      • Accelerator: GPU

      • NVIDIA driver version: 550

      • CUDA version: 12.4

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2023-ami-sagemaker-inference-gpu-4-1

      • Accelerator: GPU

      • NVIDIA driver version: 580

      • CUDA version: 13.0

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2-ami-sagemaker-inference-neuron-2

      • Accelerator: Inferentia2 and Trainium

      • Neuron driver version: 2.19

    • CapacityReservationConfig (dict) --

      Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

      • CapacityReservationPreference (string) --

        Options that you can choose for the capacity reservation. SageMaker AI supports the following options:

        capacity-reservations-only

        SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

      • MlReservationArn (string) --

        The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

    • AcceleratorPartitionLayouts (list) --

      The accelerator partition layout for the accelerators backing this production variant. A layout lists the partition profiles to carve on the accelerators it applies to; on NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles such as mig-3g.40gb. Specify a single layout: SageMaker applies it uniformly to every accelerator on the variant's instance type, and you place one inference component on each partition. Omit this field to use whole, unpartitioned accelerators.

      • (dict) --

        A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

        • AcceleratorPartitionTypes (list) --

          The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

          • (string) --

type DataCaptureConfig:

dict

param DataCaptureConfig:

Configuration to control how SageMaker AI captures inference data.

  • EnableCapture (boolean) --

    Whether data capture should be enabled or disabled (defaults to enabled).

  • InitialSamplingPercentage (integer) -- [REQUIRED]

    The percentage of requests SageMaker AI will capture. A lower value is recommended for Endpoints with high traffic.

  • DestinationS3Uri (string) -- [REQUIRED]

    The Amazon S3 location used to capture the data.

  • KmsKeyId (string) --

    The Amazon Resource Name (ARN) of an Key Management Service key that SageMaker AI uses to encrypt the captured data at rest using Amazon S3 server-side encryption.

    The KmsKeyId can be any of the following formats:

    • Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab

    • Key ARN: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

    • Alias name: alias/ExampleAlias

    • Alias name ARN: arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias

  • CaptureOptions (list) -- [REQUIRED]

    Specifies data Model Monitor will capture. You can configure whether to collect only input, only output, or both

    • (dict) --

      Specifies data Model Monitor will capture.

      • CaptureMode (string) -- [REQUIRED]

        Specify the boundary of data to capture.

  • CaptureContentTypeHeader (dict) --

    Configuration specifying how to treat different headers. If no headers are specified SageMaker AI will by default base64 encode when capturing the data.

    • CsvContentTypes (list) --

      The list of all content type headers that Amazon SageMaker AI will treat as CSV and capture accordingly.

      • (string) --

    • JsonContentTypes (list) --

      The list of all content type headers that SageMaker AI will treat as JSON and capture accordingly.

      • (string) --

type Tags:

list

param Tags:

An array of key-value pairs. You can use tags to categorize your Amazon Web Services resources in different ways, for example, by purpose, owner, or environment. For more information, see Tagging Amazon Web Services Resources.

  • (dict) --

    A tag object that consists of a key and an optional value, used to manage metadata for SageMaker Amazon Web Services resources.

    You can add tags to notebook instances, training jobs, hyperparameter tuning jobs, batch transform jobs, models, labeling jobs, work teams, endpoint configurations, and endpoints. For more information on adding tags to SageMaker resources, see AddTags.

    For more information on adding metadata to your Amazon Web Services resources with tagging, see Tagging Amazon Web Services resources. For advice on best practices for managing Amazon Web Services resources with tagging, see Tagging Best Practices: Implement an Effective Amazon Web Services Resource Tagging Strategy.

    • Key (string) -- [REQUIRED]

      The tag key. Tag keys must be unique per resource.

    • Value (string) -- [REQUIRED]

      The tag value.

type KmsKeyId:

string

param KmsKeyId:

The Amazon Resource Name (ARN) of a Amazon Web Services Key Management Service key that SageMaker uses to encrypt data on the storage volume attached to the ML compute instance that hosts the endpoint.

The KmsKeyId can be any of the following formats:

  • Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab

  • Key ARN: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

  • Alias name: alias/ExampleAlias

  • Alias name ARN: arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias

The KMS key policy must grant permission to the IAM role that you specify in your CreateEndpoint, UpdateEndpoint requests. For more information, refer to the Amazon Web Services Key Management Service section Using Key Policies in Amazon Web Services KMS

type AsyncInferenceConfig:

dict

param AsyncInferenceConfig:

Specifies configuration for how an endpoint performs asynchronous inference. This is a required field in order for your Endpoint to be invoked using InvokeEndpointAsync.

  • ClientConfig (dict) --

    Configures the behavior of the client used by SageMaker to interact with the model container during asynchronous inference.

    • MaxConcurrentInvocationsPerInstance (integer) --

      The maximum number of concurrent requests sent by the SageMaker client to the model container. If no value is provided, SageMaker chooses an optimal value.

  • OutputConfig (dict) -- [REQUIRED]

    Specifies the configuration for asynchronous inference invocation outputs.

    • KmsKeyId (string) --

      The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the asynchronous inference output in Amazon S3.

    • S3OutputPath (string) --

      The Amazon S3 location to upload inference responses to.

    • NotificationConfig (dict) --

      Specifies the configuration for notifications of inference results for asynchronous inference.

      • SuccessTopic (string) --

        Amazon SNS topic to post a notification to when inference completes successfully. If no topic is provided, no notification is sent on success.

      • ErrorTopic (string) --

        Amazon SNS topic to post a notification to when inference fails. If no topic is provided, no notification is sent on failure.

      • IncludeInferenceResponseIn (list) --

        The Amazon SNS topics where you want the inference response to be included.

        • (string) --

    • S3FailurePath (string) --

      The Amazon S3 location to upload failure inference responses to.

type ExplainerConfig:

dict

param ExplainerConfig:

A member of CreateEndpointConfig that enables explainers.

  • ClarifyExplainerConfig (dict) --

    A member of ExplainerConfig that contains configuration parameters for the SageMaker Clarify explainer.

    • EnableExplanations (string) --

      A JMESPath boolean expression used to filter which records to explain. Explanations are activated by default. See `EnableExplanations <https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-online-explainability-create-endpoint.html#clarify-online-explainability-create-endpoint-enable>`__for additional information.

    • InferenceConfig (dict) --

      The inference configuration parameter for the model container.

      • FeaturesAttribute (string) --

        Provides the JMESPath expression to extract the features from a model container input in JSON Lines format. For example, if FeaturesAttribute is the JMESPath expression 'myfeatures', it extracts a list of features [1,2,3] from request data '{"myfeatures":[1,2,3]}'.

      • ContentTemplate (string) --

        A template string used to format a JSON record into an acceptable model container input. For example, a ContentTemplate string '{"myfeatures":$features}' will format a list of features [1,2,3] into the record string '{"myfeatures":[1,2,3]}'. Required only when the model container input is in JSON Lines format.

      • MaxRecordCount (integer) --

        The maximum number of records in a request that the model container can process when querying the model container for the predictions of a synthetic dataset. A record is a unit of input data that inference can be made on, for example, a single line in CSV data. If MaxRecordCount is 1, the model container expects one record per request. A value of 2 or greater means that the model expects batch requests, which can reduce overhead and speed up the inferencing process. If this parameter is not provided, the explainer will tune the record count per request according to the model container's capacity at runtime.

      • MaxPayloadInMB (integer) --

        The maximum payload size (MB) allowed of a request from the explainer to the model container. Defaults to 6 MB.

      • ProbabilityIndex (integer) --

        A zero-based index used to extract a probability value (score) or list from model container output in CSV format. If this value is not provided, the entire model container output will be treated as a probability value (score) or list.

        Example for a single class model: If the model container output consists of a string-formatted prediction label followed by its probability: '1,0.6', set ProbabilityIndex to 1 to select the probability value 0.6.

        Example for a multiclass model: If the model container output consists of a string-formatted prediction label followed by its probability: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set ProbabilityIndex to 1 to select the probability values [0.1,0.6,0.3].

      • LabelIndex (integer) --

        A zero-based index used to extract a label header or list of label headers from model container output in CSV format.

        Example for a multiclass model: If the model container output consists of label headers followed by probabilities: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set LabelIndex to 0 to select the label headers ['cat','dog','fish'].

      • ProbabilityAttribute (string) --

        A JMESPath expression used to extract the probability (or score) from the model container output if the model container is in JSON Lines format.

        Example: If the model container output of a single request is '{"predicted_label":1,"probability":0.6}', then set ProbabilityAttribute to 'probability'.

      • LabelAttribute (string) --

        A JMESPath expression used to locate the list of label headers in the model container output.

        Example: If the model container output of a batch request is '{"labels":["cat","dog","fish"],"probability":[0.6,0.3,0.1]}', then set LabelAttribute to 'labels' to extract the list of label headers ["cat","dog","fish"]

      • LabelHeaders (list) --

        For multiclass classification problems, the label headers are the names of the classes. Otherwise, the label header is the name of the predicted label. These are used to help readability for the output of the InvokeEndpoint API. See the response section under Invoke the endpoint in the Developer Guide for more information. If there are no label headers in the model container output, provide them manually using this parameter.

        • (string) --

      • FeatureHeaders (list) --

        The names of the features. If provided, these are included in the endpoint response payload to help readability of the InvokeEndpoint output. See the Response section under Invoke the endpoint in the Developer Guide for more information.

        • (string) --

      • FeatureTypes (list) --

        A list of data types of the features (optional). Applicable only to NLP explainability. If provided, FeatureTypes must have at least one 'text' string (for example, ['text']). If FeatureTypes is not provided, the explainer infers the feature types based on the baseline data. The feature types are included in the endpoint response payload. For additional information see the response section under Invoke the endpoint in the Developer Guide for more information.

        • (string) --

    • ShapConfig (dict) -- [REQUIRED]

      The configuration for SHAP analysis.

      • ShapBaselineConfig (dict) -- [REQUIRED]

        The configuration for the SHAP baseline of the Kernal SHAP algorithm.

        • MimeType (string) --

          The MIME type of the baseline data. Choose from 'text/csv' or 'application/jsonlines'. Defaults to 'text/csv'.

        • ShapBaseline (string) --

          The inline SHAP baseline data in string format. ShapBaseline can have one or multiple records to be used as the baseline dataset. The format of the SHAP baseline file should be the same format as the training dataset. For example, if the training dataset is in CSV format and each record contains four features, and all features are numerical, then the format of the baseline data should also share these characteristics. For natural language processing (NLP) of text columns, the baseline value should be the value used to replace the unit of text specified by the Granularity of the TextConfig parameter. The size limit for ShapBasline is 4 KB. Use the ShapBaselineUri parameter if you want to provide more than 4 KB of baseline data.

        • ShapBaselineUri (string) --

          The uniform resource identifier (URI) of the S3 bucket where the SHAP baseline file is stored. The format of the SHAP baseline file should be the same format as the format of the training dataset. For example, if the training dataset is in CSV format, and each record in the training dataset has four features, and all features are numerical, then the baseline file should also have this same format. Each record should contain only the features. If you are using a virtual private cloud (VPC), the ShapBaselineUri should be accessible to the VPC. For more information about setting up endpoints with Amazon Virtual Private Cloud, see Give SageMaker access to Resources in your Amazon Virtual Private Cloud.

      • NumberOfSamples (integer) --

        The number of samples to be used for analysis by the Kernal SHAP algorithm.

      • UseLogit (boolean) --

        A Boolean toggle to indicate if you want to use the logit function (true) or log-odds units (false) for model predictions. Defaults to false.

      • Seed (integer) --

        The starting value used to initialize the random number generator in the explainer. Provide a value for this parameter to obtain a deterministic SHAP result.

      • TextConfig (dict) --

        A parameter that indicates if text features are treated as text and explanations are provided for individual units of text. Required for natural language processing (NLP) explainability only.

        • Language (string) -- [REQUIRED]

          Specifies the language of the text features in ISO 639-1 or ISO 639-3 code of a supported language.

        • Granularity (string) -- [REQUIRED]

          The unit of granularity for the analysis of text features. For example, if the unit is 'token', then each token (like a word in English) of the text is treated as a feature. SHAP values are computed for each unit/feature.

type ShadowProductionVariants:

list

param ShadowProductionVariants:

An array of ProductionVariant objects, one for each model that you want to host at this endpoint in shadow mode with production traffic replicated from the model specified on ProductionVariants. If you use this field, you can only specify one variant for ProductionVariants and one variant for ShadowProductionVariants.

  • (dict) --

    Identifies a model that you want to host and the resources chosen to deploy for hosting it. If you are deploying multiple models, tell SageMaker how to distribute traffic among the models by specifying variant weights. For more information on production variants, check Production variants.

    • VariantName (string) -- [REQUIRED]

      The name of the production variant.

    • ModelName (string) --

      The name of the model that you want to host. This is the name that you specified when creating the model.

    • InitialInstanceCount (integer) --

      Number of instances to launch initially.

    • InstanceType (string) --

      The ML compute instance type.

    • InstancePools (list) --

      A list of instance pools for the production variant. Each instance pool specifies an instance type and its priority for provisioning. Use instance pools to configure heterogeneous endpoints that deploy models across multiple instance types.

      • (dict) --

        Specifies an instance type and its priority for a heterogeneous endpoint. Use instance pools to configure a production variant with multiple instance types, enabling the endpoint to provision instances across different types based on priority.

        • InstanceType (string) -- [REQUIRED]

          The ML compute instance type for the instance pool.

        • ModelNameOverride (string) --

          The name of a SageMaker model to use for this instance pool instead of the model specified for the production variant. Use this to deploy a different model optimized for the instance type in this pool.

        • Priority (integer) -- [REQUIRED]

          The priority for the instance pool. SageMaker attempts to provision instances in order of priority, starting with the lowest value. If instances for a higher-priority pool are unavailable, SageMaker attempts to provision from the next pool.

          Valid values: 1 to 5, where 1 is the highest priority.

    • VariantInstanceProvisionTimeoutInSeconds (integer) --

      The timeout value, in seconds, for provisioning instances for the production variant. When SageMaker encounters an insufficient capacity error while provisioning instances, it retries with the next instance pool (if configured) or waits until the timeout expires. This timeout applies only to capacity provisioning and does not include the time for model download or container startup.

      Valid values: 300 to 3600.

    • InitialVariantWeight (float) --

      Determines initial traffic distribution among all of the models that you specify in the endpoint configuration. The traffic to a production variant is determined by the ratio of the VariantWeight to the sum of all VariantWeight values across all ProductionVariants. If unspecified, it defaults to 1.0.

    • AcceleratorType (string) --

      This parameter is no longer supported. Elastic Inference (EI) is no longer available.

      This parameter was used to specify the size of the EI instance to use for the production variant.

    • CoreDumpConfig (dict) --

      Specifies configuration for a core dump from the model container when the process crashes.

      • DestinationS3Uri (string) -- [REQUIRED]

        The Amazon S3 bucket to send the core dump to.

      • KmsKeyId (string) --

        The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the core dump data at rest using Amazon S3 server-side encryption. The KmsKeyId can be any of the following formats:

        • // KMS Key ID "1234abcd-12ab-34cd-56ef-1234567890ab"

        • // Amazon Resource Name (ARN) of a KMS Key "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"

        • // KMS Key Alias "alias/ExampleAlias"

        • // Amazon Resource Name (ARN) of a KMS Key Alias "arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias"

        If you use a KMS key ID or an alias of your KMS key, the SageMaker execution role must include permissions to call kms:Encrypt. If you don't provide a KMS key ID, SageMaker uses the default KMS key for Amazon S3 for your role's account. SageMaker uses server-side encryption with KMS-managed keys for OutputDataConfig. If you use a bucket policy with an s3:PutObject permission that only allows objects with server-side encryption, set the condition key of s3:x-amz-server-side-encryption to "aws:kms". For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.

        The KMS key policy must grant permission to the IAM role that you specify in your CreateEndpoint and UpdateEndpoint requests. For more information, see Using Key Policies in Amazon Web Services KMS in the Amazon Web Services Key Management Service Developer Guide.

    • ServerlessConfig (dict) --

      The serverless configuration for an endpoint. Specifies a serverless endpoint configuration instead of an instance-based endpoint configuration.

      • MemorySizeInMB (integer) -- [REQUIRED]

        The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

      • MaxConcurrency (integer) -- [REQUIRED]

        The maximum number of concurrent invocations your serverless endpoint can process.

      • ProvisionedConcurrency (integer) --

        The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

    • VolumeSizeInGB (integer) --

      The size, in GB, of the ML storage volume attached to individual inference instance associated with the production variant. Currently only Amazon EBS gp2 storage volumes are supported.

    • ModelDataDownloadTimeoutInSeconds (integer) --

      The timeout value, in seconds, to download and extract the model that you want to host from Amazon S3 to the individual inference instance associated with this production variant.

    • ContainerStartupHealthCheckTimeoutInSeconds (integer) --

      The timeout value, in seconds, for your inference container to pass health check by SageMaker Hosting. For more information about health check, see How Your Container Should Respond to Health Check (Ping) Requests.

    • EnableSSMAccess (boolean) --

      You can use this parameter to turn on native Amazon Web Services Systems Manager (SSM) access for a production variant behind an endpoint. By default, SSM access is disabled for all production variants behind an endpoint. You can turn on or turn off SSM access for a production variant behind an existing endpoint by creating a new endpoint configuration and calling UpdateEndpoint.

    • ManagedInstanceScaling (dict) --

      Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

      • Status (string) --

        Indicates whether managed instance scaling is enabled.

      • MinInstanceCount (integer) --

        The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

      • MaxInstanceCount (integer) --

        The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

      • ScaleInPolicy (dict) --

        Configures the scale-in behavior for managed instance scaling.

        • Strategy (string) -- [REQUIRED]

          The strategy for scaling in instances.

          IDLE_RELEASE

          Releases instances that have no hosted inference component copies.

          CONSOLIDATION

          Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

        • MaximumStepSize (integer) --

          The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

          Default value: 1.

        • CooldownInMinutes (integer) --

          The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

          Default value: 20.

    • RoutingConfig (dict) --

      Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

      • RoutingStrategy (string) -- [REQUIRED]

        Sets how the endpoint routes incoming traffic:

        • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

        • RANDOM: The endpoint routes each request to a randomly chosen instance.

        • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

        • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

      • PrefixAwareRoutingConfig (dict) --

        The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

        • PrefixLength (integer) --

          The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

          • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

          • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

          The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

        • ConcurrencyThreshold (integer) --

          The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

          On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

          On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

          The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

    • InferenceAmiVersion (string) --

      Specifies an option from a collection of preconfigured Amazon Machine Image (AMI) images. Each image is configured by Amazon Web Services with a set of software and driver versions. Amazon Web Services optimizes these configurations for different machine learning workloads.

      By selecting an AMI version, you can ensure that your inference environment is compatible with specific software requirements, such as CUDA driver versions, Linux kernel versions, or Amazon Web Services Neuron driver versions.

      The AMI version names, and their configurations, are the following:

      al2-ami-sagemaker-inference-gpu-2

      • Accelerator: GPU

      • NVIDIA driver version: 535

      • CUDA version: 12.2

        al2-ami-sagemaker-inference-gpu-2-1

      • Accelerator: GPU

      • NVIDIA driver version: 535

      • CUDA version: 12.2

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2-ami-sagemaker-inference-gpu-3-1

      • Accelerator: GPU

      • NVIDIA driver version: 550

      • CUDA version: 12.4

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2023-ami-sagemaker-inference-gpu-4-1

      • Accelerator: GPU

      • NVIDIA driver version: 580

      • CUDA version: 13.0

      • NVIDIA Container Toolkit with disabled CUDA-compat mounting

        al2-ami-sagemaker-inference-neuron-2

      • Accelerator: Inferentia2 and Trainium

      • Neuron driver version: 2.19

    • CapacityReservationConfig (dict) --

      Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

      • CapacityReservationPreference (string) --

        Options that you can choose for the capacity reservation. SageMaker AI supports the following options:

        capacity-reservations-only

        SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

      • MlReservationArn (string) --

        The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

    • AcceleratorPartitionLayouts (list) --

      The accelerator partition layout for the accelerators backing this production variant. A layout lists the partition profiles to carve on the accelerators it applies to; on NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles such as mig-3g.40gb. Specify a single layout: SageMaker applies it uniformly to every accelerator on the variant's instance type, and you place one inference component on each partition. Omit this field to use whole, unpartitioned accelerators.

      • (dict) --

        A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

        • AcceleratorPartitionTypes (list) --

          The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

          • (string) --

type ExecutionRoleArn:

string

param ExecutionRoleArn:

The Amazon Resource Name (ARN) of an IAM role that Amazon SageMaker AI can assume to perform actions on your behalf. For more information, see SageMaker AI Roles.

type VpcConfig:

dict

param VpcConfig:

Specifies an Amazon Virtual Private Cloud (VPC) that your SageMaker jobs, hosted models, and compute resources have access to. You can control access to and from your resources by configuring a VPC. For more information, see Give SageMaker Access to Resources in your Amazon VPC.

  • SecurityGroupIds (list) -- [REQUIRED]

    The VPC security group IDs, in the form sg-xxxxxxxx. Specify the security groups for the VPC that is specified in the Subnets field.

    • (string) --

  • Subnets (list) -- [REQUIRED]

    The ID of the subnets in the VPC to which you want to connect your training job or model. For information about the availability of specific instance types, see Supported Instance Types and Availability Zones.

    • (string) --

type EnableNetworkIsolation:

boolean

param EnableNetworkIsolation:

Sets whether all model containers deployed to the endpoint are isolated. If they are, no inbound or outbound network calls can be made to or from the model containers.

type MetricsConfig:

dict

param MetricsConfig:

The configuration parameters for utilization metrics.

  • EnableEnhancedMetrics (boolean) --

    Specifies whether to enable enhanced metrics for the endpoint. Enhanced metrics provide utilization and invocation data at instance and container granularity. Container granularity is supported for Inference Components. The default is False.

  • EnableDetailedObservability (boolean) --

    Indicates whether detailed observability is enabled for the endpoint. When set to True, the following metrics are published at the configured frequency:

    • Container-level inference metrics scraped from the container's Prometheus endpoint (such as request latency, error counts, and throughput). Available metrics vary by framework.

    • Per-GPU metrics (utilization, memory, and temperature) attributed to individual inference components.

    • Per-instance host metrics (CPU, memory, and disk utilization).

    • Inference component placement metrics (copy count per Availability Zone).

    For first-party and Deep Learning Containers (DLC), the Prometheus endpoint path is determined automatically. For Bring-Your-Own-Container (BYOC) cases, you can optionally set ContainerMetricsConfig to specify a custom endpoint path. If not specified, the default path /metrics on port 8080 is used.

    When set to False, these additional metrics are not published. Standard invocation and utilization metrics controlled by EnableEnhancedMetrics are unaffected.

    The default value for new endpoint configurations is True. For existing endpoint configurations created before this feature, the value is False unless explicitly set.

  • MetricPublishFrequencyInSeconds (integer) --

    The interval, in seconds, at which metrics are published to Amazon CloudWatch. Defaults to 60. Valid values: 10, 30, 60, 120, 180, 240, 300.

    When EnableEnhancedMetrics is set to False, this interval applies to utilization metrics only. Invocation metrics continue to be published at the default 60-second interval. When EnableEnhancedMetrics is set to True, this interval applies to both utilization and invocation metrics.

    When EnableDetailedObservability is set to True, this interval applies to per-GPU metrics, per-instance host metrics, container metrics, and fleet-level inference component lifecycle and placement metrics.

type CrossRegionTrafficConfig:

dict

param CrossRegionTrafficConfig:

The cross-Region traffic cap for the endpoint. The cap limits the requests per second that the endpoint accepts from inference routers in other Amazon Web Services Regions. If you don't specify it, these requests aren't limited. You can't specify it with serverless variants or with AsyncInferenceConfig. For information about the cross-Region traffic cap, see Limit requests from other Regions.

  • MaxInvocationsPerSecondPerInstance (integer) -- [REQUIRED]

    The number of invocations per second that you expect one InService instance to serve. One value applies to every production variant, whatever its instance type.

  • MaxCrossRegionPercentPerEndpoint (integer) -- [REQUIRED]

    The percentage of the endpoint's capacity that requests from inference routers in other Amazon Web Services Regions can use.

rtype:

dict

returns:

Response Syntax

{
    'EndpointConfigArn': 'string'
}

Response Structure

  • (dict) --

    • EndpointConfigArn (string) --

      The Amazon Resource Name (ARN) of the endpoint configuration.

CreateJob (updated) Link ¶
Changes (request)
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}

Creates a model customization job in Amazon SageMaker. A job runs a workload based on the job category and configuration you provide. You specify the job category, a schema-versioned configuration document, and an IAM role that grants Amazon SageMaker permission to access resources on your behalf.

Specify one of the following job categories, which determines the type of workload that the job runs:

AgentRFT

Fine-tune a model using multi-turn reinforcement learning with reward signals.

AgentRFTEvaluation

Evaluate a fine-tuned or base model by running multi-turn rollouts against a held-out prompt dataset. This category computes metrics such as pass@k and mean reward.

DataQualityEvaluation

Evaluate the quality of a generated or bring-your-own dataset using quality metrics or your own specification.

DataTransformation

Transform datasets into the format you need, for example for model customization.

SyntheticDataGeneration

Generate synthetic training and testing datasets from your context data and specifications.

Before creating a job, call ListJobSchemaVersions and DescribeJobSchemaVersion to retrieve the configuration schema for your job category. The JobConfigDocument must conform to the schema specified by JobConfigSchemaVersion.

The following operations are related to CreateJob:

  • DescribeJob

  • ListJobs

  • StopJob

  • DeleteJob

  • ListJobSchemaVersions

  • DescribeJobSchemaVersion

See also: AWS API Documentation

Request Syntax

client.create_job(
    JobName='string',
    RoleArn='string',
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    JobConfigSchemaVersion='string',
    JobConfigDocument='string',
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
type JobName:

string

param JobName:

[REQUIRED]

The name of the job. The name must be unique within your account and Amazon Web Services Region.

type RoleArn:

string

param RoleArn:

[REQUIRED]

The Amazon Resource Name (ARN) of the IAM role that Amazon SageMaker assumes to perform the job. The role must have the necessary permissions to access the resources required by the job configuration.

type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of the job. The category determines the type of workload that the job runs.

type JobConfigSchemaVersion:

string

param JobConfigSchemaVersion:

[REQUIRED]

The version of the configuration schema to use for the job configuration document. Use ListJobSchemaVersions to get available schema versions for a job category.

type JobConfigDocument:

string

param JobConfigDocument:

[REQUIRED]

The JSON configuration document for the job. The document must conform to the schema specified by JobConfigSchemaVersion. Use DescribeJobSchemaVersion to retrieve the schema for validation.

type Tags:

list

param Tags:

An array of key-value pairs to apply to the job as tags. For more information, see Tagging Amazon Web Services Resources.

  • (dict) --

    A tag object that consists of a key and an optional value, used to manage metadata for SageMaker Amazon Web Services resources.

    You can add tags to notebook instances, training jobs, hyperparameter tuning jobs, batch transform jobs, models, labeling jobs, work teams, endpoint configurations, and endpoints. For more information on adding tags to SageMaker resources, see AddTags.

    For more information on adding metadata to your Amazon Web Services resources with tagging, see Tagging Amazon Web Services resources. For advice on best practices for managing Amazon Web Services resources with tagging, see Tagging Best Practices: Implement an Effective Amazon Web Services Resource Tagging Strategy.

    • Key (string) -- [REQUIRED]

      The tag key. Tag keys must be unique per resource.

    • Value (string) -- [REQUIRED]

      The tag value.

rtype:

dict

returns:

Response Syntax

{
    'JobArn': 'string'
}

Response Structure

  • (dict) --

    • JobArn (string) --

      The Amazon Resource Name (ARN) of the job.

DeleteJob (updated) Link ¶
Changes (request)
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}

Deletes a job. This operation is idempotent. If the job is currently running, you must stop it before deleting it by calling StopJob.

The following operations are related to DeleteJob:

  • CreateJob

  • StopJob

  • DescribeJob

See also: AWS API Documentation

Request Syntax

client.delete_job(
    JobName='string',
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation'
)
type JobName:

string

param JobName:

[REQUIRED]

The name of the job to delete.

type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of the job to delete.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

  • (dict) --

DescribeEndpoint (updated) Link ¶
Changes (response)
{'CrossRegionTrafficConfig': {'MaxCrossRegionPercentPerEndpoint': 'integer',
                              'MaxInvocationsPerSecondPerInstance': 'integer'},
 'PendingDeploymentSummary': {'ProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}},
                              'ShadowProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}}},
 'ProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}},
 'ShadowProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}}}

Returns the description of an endpoint.

See also: AWS API Documentation

Request Syntax

client.describe_endpoint(
    EndpointName='string'
)
type EndpointName:

string

param EndpointName:

[REQUIRED]

The name of the endpoint.

rtype:

dict

returns:

Response Syntax

{
    'EndpointName': 'string',
    'EndpointArn': 'string',
    'EndpointConfigName': 'string',
    'ProductionVariants': [
        {
            'VariantName': 'string',
            'DeployedImages': [
                {
                    'SpecifiedImage': 'string',
                    'ResolvedImage': 'string',
                    'ResolutionTime': datetime(2015, 1, 1)
                },
            ],
            'CurrentWeight': ...,
            'DesiredWeight': ...,
            'CurrentInstanceCount': 123,
            'DesiredInstanceCount': 123,
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'CurrentInstanceCount': 123
                },
            ],
            'VariantStatus': [
                {
                    'Status': 'Creating'|'Updating'|'Deleting'|'ActivatingTraffic'|'Baking',
                    'StatusMessage': 'string',
                    'StartTime': datetime(2015, 1, 1)
                },
            ],
            'CurrentServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'DesiredServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'CapacityReservationConfig': {
                'MlReservationArn': 'string',
                'CapacityReservationPreference': 'capacity-reservations-only',
                'TotalInstanceCount': 123,
                'AvailableInstanceCount': 123,
                'UsedByCurrentEndpoint': 123,
                'Ec2CapacityReservations': [
                    {
                        'Ec2CapacityReservationId': 'string',
                        'TotalInstanceCount': 123,
                        'AvailableInstanceCount': 123,
                        'UsedByCurrentEndpoint': 123
                    },
                ]
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    'DataCaptureConfig': {
        'EnableCapture': True|False,
        'CaptureStatus': 'Started'|'Stopped',
        'CurrentSamplingPercentage': 123,
        'DestinationS3Uri': 'string',
        'KmsKeyId': 'string'
    },
    'EndpointStatus': 'OutOfService'|'Creating'|'Updating'|'SystemUpdating'|'RollingBack'|'InService'|'Deleting'|'Failed'|'UpdateRollbackFailed',
    'FailureReason': 'string',
    'CreationTime': datetime(2015, 1, 1),
    'LastModifiedTime': datetime(2015, 1, 1),
    'LastDeploymentConfig': {
        'BlueGreenUpdatePolicy': {
            'TrafficRoutingConfiguration': {
                'Type': 'ALL_AT_ONCE'|'CANARY'|'LINEAR',
                'WaitIntervalInSeconds': 123,
                'CanarySize': {
                    'Type': 'INSTANCE_COUNT'|'CAPACITY_PERCENT',
                    'Value': 123
                },
                'LinearStepSize': {
                    'Type': 'INSTANCE_COUNT'|'CAPACITY_PERCENT',
                    'Value': 123
                }
            },
            'TerminationWaitInSeconds': 123,
            'MaximumExecutionTimeoutInSeconds': 123
        },
        'RollingUpdatePolicy': {
            'MaximumBatchSize': {
                'Type': 'INSTANCE_COUNT'|'CAPACITY_PERCENT',
                'Value': 123
            },
            'WaitIntervalInSeconds': 123,
            'MaximumExecutionTimeoutInSeconds': 123,
            'RollbackMaximumBatchSize': {
                'Type': 'INSTANCE_COUNT'|'CAPACITY_PERCENT',
                'Value': 123
            }
        },
        'AutoRollbackConfiguration': {
            'Alarms': [
                {
                    'AlarmName': 'string'
                },
            ]
        }
    },
    'AsyncInferenceConfig': {
        'ClientConfig': {
            'MaxConcurrentInvocationsPerInstance': 123
        },
        'OutputConfig': {
            'KmsKeyId': 'string',
            'S3OutputPath': 'string',
            'NotificationConfig': {
                'SuccessTopic': 'string',
                'ErrorTopic': 'string',
                'IncludeInferenceResponseIn': [
                    'SUCCESS_NOTIFICATION_TOPIC'|'ERROR_NOTIFICATION_TOPIC',
                ]
            },
            'S3FailurePath': 'string'
        }
    },
    'PendingDeploymentSummary': {
        'EndpointConfigName': 'string',
        'ProductionVariants': [
            {
                'VariantName': 'string',
                'DeployedImages': [
                    {
                        'SpecifiedImage': 'string',
                        'ResolvedImage': 'string',
                        'ResolutionTime': datetime(2015, 1, 1)
                    },
                ],
                'CurrentWeight': ...,
                'DesiredWeight': ...,
                'CurrentInstanceCount': 123,
                'DesiredInstanceCount': 123,
                'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                'InstancePools': [
                    {
                        'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                        'CurrentInstanceCount': 123
                    },
                ],
                'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
                'VariantStatus': [
                    {
                        'Status': 'Creating'|'Updating'|'Deleting'|'ActivatingTraffic'|'Baking',
                        'StatusMessage': 'string',
                        'StartTime': datetime(2015, 1, 1)
                    },
                ],
                'CurrentServerlessConfig': {
                    'MemorySizeInMB': 123,
                    'MaxConcurrency': 123,
                    'ProvisionedConcurrency': 123
                },
                'DesiredServerlessConfig': {
                    'MemorySizeInMB': 123,
                    'MaxConcurrency': 123,
                    'ProvisionedConcurrency': 123
                },
                'ManagedInstanceScaling': {
                    'Status': 'ENABLED'|'DISABLED',
                    'MinInstanceCount': 123,
                    'MaxInstanceCount': 123,
                    'ScaleInPolicy': {
                        'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                        'MaximumStepSize': 123,
                        'CooldownInMinutes': 123
                    }
                },
                'RoutingConfig': {
                    'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                    'PrefixAwareRoutingConfig': {
                        'PrefixLength': 123,
                        'ConcurrencyThreshold': 123
                    }
                },
                'AcceleratorPartitionLayouts': [
                    {
                        'AcceleratorPartitionTypes': [
                            'string',
                        ]
                    },
                ]
            },
        ],
        'StartTime': datetime(2015, 1, 1),
        'ShadowProductionVariants': [
            {
                'VariantName': 'string',
                'DeployedImages': [
                    {
                        'SpecifiedImage': 'string',
                        'ResolvedImage': 'string',
                        'ResolutionTime': datetime(2015, 1, 1)
                    },
                ],
                'CurrentWeight': ...,
                'DesiredWeight': ...,
                'CurrentInstanceCount': 123,
                'DesiredInstanceCount': 123,
                'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                'InstancePools': [
                    {
                        'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                        'CurrentInstanceCount': 123
                    },
                ],
                'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
                'VariantStatus': [
                    {
                        'Status': 'Creating'|'Updating'|'Deleting'|'ActivatingTraffic'|'Baking',
                        'StatusMessage': 'string',
                        'StartTime': datetime(2015, 1, 1)
                    },
                ],
                'CurrentServerlessConfig': {
                    'MemorySizeInMB': 123,
                    'MaxConcurrency': 123,
                    'ProvisionedConcurrency': 123
                },
                'DesiredServerlessConfig': {
                    'MemorySizeInMB': 123,
                    'MaxConcurrency': 123,
                    'ProvisionedConcurrency': 123
                },
                'ManagedInstanceScaling': {
                    'Status': 'ENABLED'|'DISABLED',
                    'MinInstanceCount': 123,
                    'MaxInstanceCount': 123,
                    'ScaleInPolicy': {
                        'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                        'MaximumStepSize': 123,
                        'CooldownInMinutes': 123
                    }
                },
                'RoutingConfig': {
                    'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                    'PrefixAwareRoutingConfig': {
                        'PrefixLength': 123,
                        'ConcurrencyThreshold': 123
                    }
                },
                'AcceleratorPartitionLayouts': [
                    {
                        'AcceleratorPartitionTypes': [
                            'string',
                        ]
                    },
                ]
            },
        ]
    },
    'ExplainerConfig': {
        'ClarifyExplainerConfig': {
            'EnableExplanations': 'string',
            'InferenceConfig': {
                'FeaturesAttribute': 'string',
                'ContentTemplate': 'string',
                'MaxRecordCount': 123,
                'MaxPayloadInMB': 123,
                'ProbabilityIndex': 123,
                'LabelIndex': 123,
                'ProbabilityAttribute': 'string',
                'LabelAttribute': 'string',
                'LabelHeaders': [
                    'string',
                ],
                'FeatureHeaders': [
                    'string',
                ],
                'FeatureTypes': [
                    'numerical'|'categorical'|'text',
                ]
            },
            'ShapConfig': {
                'ShapBaselineConfig': {
                    'MimeType': 'string',
                    'ShapBaseline': 'string',
                    'ShapBaselineUri': 'string'
                },
                'NumberOfSamples': 123,
                'UseLogit': True|False,
                'Seed': 123,
                'TextConfig': {
                    'Language': 'af'|'sq'|'ar'|'hy'|'eu'|'bn'|'bg'|'ca'|'zh'|'hr'|'cs'|'da'|'nl'|'en'|'et'|'fi'|'fr'|'de'|'el'|'gu'|'he'|'hi'|'hu'|'is'|'id'|'ga'|'it'|'kn'|'ky'|'lv'|'lt'|'lb'|'mk'|'ml'|'mr'|'ne'|'nb'|'fa'|'pl'|'pt'|'ro'|'ru'|'sa'|'sr'|'tn'|'si'|'sk'|'sl'|'es'|'sv'|'tl'|'ta'|'tt'|'te'|'tr'|'uk'|'ur'|'yo'|'lij'|'xx',
                    'Granularity': 'token'|'sentence'|'paragraph'
                }
            }
        }
    },
    'ShadowProductionVariants': [
        {
            'VariantName': 'string',
            'DeployedImages': [
                {
                    'SpecifiedImage': 'string',
                    'ResolvedImage': 'string',
                    'ResolutionTime': datetime(2015, 1, 1)
                },
            ],
            'CurrentWeight': ...,
            'DesiredWeight': ...,
            'CurrentInstanceCount': 123,
            'DesiredInstanceCount': 123,
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'CurrentInstanceCount': 123
                },
            ],
            'VariantStatus': [
                {
                    'Status': 'Creating'|'Updating'|'Deleting'|'ActivatingTraffic'|'Baking',
                    'StatusMessage': 'string',
                    'StartTime': datetime(2015, 1, 1)
                },
            ],
            'CurrentServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'DesiredServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'CapacityReservationConfig': {
                'MlReservationArn': 'string',
                'CapacityReservationPreference': 'capacity-reservations-only',
                'TotalInstanceCount': 123,
                'AvailableInstanceCount': 123,
                'UsedByCurrentEndpoint': 123,
                'Ec2CapacityReservations': [
                    {
                        'Ec2CapacityReservationId': 'string',
                        'TotalInstanceCount': 123,
                        'AvailableInstanceCount': 123,
                        'UsedByCurrentEndpoint': 123
                    },
                ]
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    'MetricsConfig': {
        'EnableEnhancedMetrics': True|False,
        'EnableDetailedObservability': True|False,
        'MetricPublishFrequencyInSeconds': 123
    },
    'CrossRegionTrafficConfig': {
        'MaxInvocationsPerSecondPerInstance': 123,
        'MaxCrossRegionPercentPerEndpoint': 123
    }
}

Response Structure

  • (dict) --

    • EndpointName (string) --

      Name of the endpoint.

    • EndpointArn (string) --

      The Amazon Resource Name (ARN) of the endpoint.

    • EndpointConfigName (string) --

      The name of the endpoint configuration associated with this endpoint.

    • ProductionVariants (list) --

      An array of ProductionVariantSummary objects, one for each model hosted behind this endpoint.

      • (dict) --

        Describes weight and capacities for a production variant associated with an endpoint. If you sent a request to the UpdateEndpointWeightsAndCapacities API and the endpoint status is Updating, you get different desired and current values.

        • VariantName (string) --

          The name of the variant.

        • DeployedImages (list) --

          An array of DeployedImage objects that specify the Amazon EC2 Container Registry paths of the inference images deployed on instances of this ProductionVariant.

          • (dict) --

            Gets the Amazon EC2 Container Registry path of the docker image of the model that is hosted in this ProductionVariant.

            If you used the registry/repository[:tag] form to specify the image path of the primary container when you created the model hosted in this ProductionVariant, the path resolves to a path of the form registry/repository[@digest]. A digest is a hash value that identifies a specific version of an image. For information about Amazon ECR paths, see Pulling an Image in the Amazon ECR User Guide.

            • SpecifiedImage (string) --

              The image path you specified when you created the model.

            • ResolvedImage (string) --

              The specific digest path of the image hosted in this ProductionVariant.

            • ResolutionTime (datetime) --

              The date and time when the image path for the model resolved to the ResolvedImage

        • CurrentWeight (float) --

          The weight associated with the variant.

        • DesiredWeight (float) --

          The requested weight, as specified in the UpdateEndpointWeightsAndCapacities request.

        • CurrentInstanceCount (integer) --

          The number of instances associated with the variant.

        • DesiredInstanceCount (integer) --

          The number of instances requested in the UpdateEndpointWeightsAndCapacities request.

        • InstancePools (list) --

          A list of instance pools for the production variant. Each pool indicates the instance type and the current number of instances of that type.

          • (dict) --

            A summary of an instance pool for a production variant, including the instance type and the current number of instances.

            • InstanceType (string) --

              The ML compute instance type for the instance pool.

            • CurrentInstanceCount (integer) --

              The current number of instances of this type in the instance pool.

        • VariantStatus (list) --

          The endpoint variant status which describes the current deployment stage status or operational status.

          • (dict) --

            Describes the status of the production variant.

            • Status (string) --

              The endpoint variant status which describes the current deployment stage status or operational status.

              • Creating: Creating inference resources for the production variant.

              • Deleting: Terminating inference resources for the production variant.

              • Updating: Updating capacity for the production variant.

              • ActivatingTraffic: Turning on traffic for the production variant.

              • Baking: Waiting period to monitor the CloudWatch alarms in the automatic rollback configuration.

            • StatusMessage (string) --

              A message that describes the status of the production variant.

            • StartTime (datetime) --

              The start time of the current status change.

        • CurrentServerlessConfig (dict) --

          The serverless configuration for the endpoint.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • DesiredServerlessConfig (dict) --

          The serverless configuration requested for the endpoint update.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • ManagedInstanceScaling (dict) --

          Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

          • Status (string) --

            Indicates whether managed instance scaling is enabled.

          • MinInstanceCount (integer) --

            The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

          • MaxInstanceCount (integer) --

            The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

          • ScaleInPolicy (dict) --

            Configures the scale-in behavior for managed instance scaling.

            • Strategy (string) --

              The strategy for scaling in instances.

              IDLE_RELEASE

              Releases instances that have no hosted inference component copies.

              CONSOLIDATION

              Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

            • MaximumStepSize (integer) --

              The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

              Default value: 1.

            • CooldownInMinutes (integer) --

              The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

              Default value: 20.

        • RoutingConfig (dict) --

          Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

          • RoutingStrategy (string) --

            Sets how the endpoint routes incoming traffic:

            • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

            • RANDOM: The endpoint routes each request to a randomly chosen instance.

            • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

            • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

          • PrefixAwareRoutingConfig (dict) --

            The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

            • PrefixLength (integer) --

              The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

              • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

              • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

              The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

            • ConcurrencyThreshold (integer) --

              The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

              On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

              On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

              The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

        • CapacityReservationConfig (dict) --

          Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

          • MlReservationArn (string) --

            The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

          • CapacityReservationPreference (string) --

            The option that you chose for the capacity reservation. SageMaker AI supports the following options:

            capacity-reservations-only

            SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

          • TotalInstanceCount (integer) --

            The number of instances that you allocated to the ML capacity reservation.

          • AvailableInstanceCount (integer) --

            The number of instances that are currently available in the ML capacity reservation.

          • UsedByCurrentEndpoint (integer) --

            The number of instances from the ML capacity reservation that are being used by the endpoint.

          • Ec2CapacityReservations (list) --

            The EC2 capacity reservations that are shared to this ML capacity reservation, if any.

            • (dict) --

              The EC2 capacity reservations that are shared to an ML capacity reservation.

              • Ec2CapacityReservationId (string) --

                The unique identifier for an EC2 capacity reservation that's part of the ML capacity reservation.

              • TotalInstanceCount (integer) --

                The number of instances that you allocated to the EC2 capacity reservation.

              • AvailableInstanceCount (integer) --

                The number of instances that are currently available in the EC2 capacity reservation.

              • UsedByCurrentEndpoint (integer) --

                The number of instances from the EC2 capacity reservation that are being used by the endpoint.

        • AcceleratorPartitionLayouts (list) --

          The accelerator partition layout that SageMaker applies to this production variant. On NVIDIA GPUs, the profiles are Multi-Instance GPU (MIG) profiles. SageMaker returns the accelerator partition layout as you provided it when you created the variant, without normalization.

          • (dict) --

            A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

            • AcceleratorPartitionTypes (list) --

              The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

              • (string) --

    • DataCaptureConfig (dict) --

      The currently active data capture configuration used by your Endpoint.

      • EnableCapture (boolean) --

        Whether data capture is enabled or disabled.

      • CaptureStatus (string) --

        Whether data capture is currently functional.

      • CurrentSamplingPercentage (integer) --

        The percentage of requests being captured by your Endpoint.

      • DestinationS3Uri (string) --

        The Amazon S3 location being used to capture the data.

      • KmsKeyId (string) --

        The KMS key being used to encrypt the data in Amazon S3.

    • EndpointStatus (string) --

      The status of the endpoint.

      • OutOfService: Endpoint is not available to take incoming requests.

      • Creating: CreateEndpoint is executing.

      • Updating: UpdateEndpoint or UpdateEndpointWeightsAndCapacities is executing.

      • SystemUpdating: Endpoint is undergoing maintenance and cannot be updated or deleted or re-scaled until it has completed. This maintenance operation does not change any customer-specified values such as VPC config, KMS encryption, model, instance type, or instance count.

      • RollingBack: Endpoint fails to scale up or down or change its variant weight and is in the process of rolling back to its previous configuration. Once the rollback completes, endpoint returns to an InService status. This transitional status only applies to an endpoint that has autoscaling enabled and is undergoing variant weight or capacity changes as part of an UpdateEndpointWeightsAndCapacities call or when the UpdateEndpointWeightsAndCapacities operation is called explicitly.

      • InService: Endpoint is available to process incoming requests.

      • Deleting: DeleteEndpoint is executing.

      • Failed: Endpoint could not be created, updated, or re-scaled. Use the FailureReason value returned by DescribeEndpoint for information about the failure. DeleteEndpoint is the only operation that can be performed on a failed endpoint.

      • UpdateRollbackFailed: Both the rolling deployment and auto-rollback failed. Your endpoint is in service with a mix of the old and new endpoint configurations. For information about how to remedy this issue and restore the endpoint's status to InService, see Rolling Deployments.

    • FailureReason (string) --

      If the status of the endpoint is Failed, the reason why it failed.

    • CreationTime (datetime) --

      A timestamp that shows when the endpoint was created.

    • LastModifiedTime (datetime) --

      A timestamp that shows when the endpoint was last modified.

    • LastDeploymentConfig (dict) --

      The most recent deployment configuration for the endpoint.

      • BlueGreenUpdatePolicy (dict) --

        Update policy for a blue/green deployment. If this update policy is specified, SageMaker creates a new fleet during the deployment while maintaining the old fleet. SageMaker flips traffic to the new fleet according to the specified traffic routing configuration. Only one update policy should be used in the deployment configuration. If no update policy is specified, SageMaker uses a blue/green deployment strategy with all at once traffic shifting by default.

        • TrafficRoutingConfiguration (dict) --

          Defines the traffic routing strategy to shift traffic from the old fleet to the new fleet during an endpoint deployment.

          • Type (string) --

            Traffic routing strategy type.

            • ALL_AT_ONCE: Endpoint traffic shifts to the new fleet in a single step.

            • CANARY: Endpoint traffic shifts to the new fleet in two steps. The first step is the canary, which is a small portion of the traffic. The second step is the remainder of the traffic.

            • LINEAR: Endpoint traffic shifts to the new fleet in n steps of a configurable size.

          • WaitIntervalInSeconds (integer) --

            The waiting time (in seconds) between incremental steps to turn on traffic on the new endpoint fleet.

          • CanarySize (dict) --

            Batch size for the first step to turn on traffic on the new endpoint fleet. Value must be less than or equal to 50% of the variant's total instance count.

            • Type (string) --

              Specifies the endpoint capacity type.

              • INSTANCE_COUNT: The endpoint activates based on the number of instances.

              • CAPACITY_PERCENT: The endpoint activates based on the specified percentage of capacity.

            • Value (integer) --

              Defines the capacity size, either as a number of instances or a capacity percentage.

          • LinearStepSize (dict) --

            Batch size for each step to turn on traffic on the new endpoint fleet. Value must be 10-50% of the variant's total instance count.

            • Type (string) --

              Specifies the endpoint capacity type.

              • INSTANCE_COUNT: The endpoint activates based on the number of instances.

              • CAPACITY_PERCENT: The endpoint activates based on the specified percentage of capacity.

            • Value (integer) --

              Defines the capacity size, either as a number of instances or a capacity percentage.

        • TerminationWaitInSeconds (integer) --

          Additional waiting time in seconds after the completion of an endpoint deployment before terminating the old endpoint fleet. Default is 0.

        • MaximumExecutionTimeoutInSeconds (integer) --

          Maximum execution timeout for the deployment. Note that the timeout value should be larger than the total waiting time specified in TerminationWaitInSeconds and WaitIntervalInSeconds.

      • RollingUpdatePolicy (dict) --

        Specifies a rolling deployment strategy for updating a SageMaker endpoint.

        • MaximumBatchSize (dict) --

          Batch size for each rolling step to provision capacity and turn on traffic on the new endpoint fleet, and terminate capacity on the old endpoint fleet. Value must be between 5% to 50% of the variant's total instance count.

          • Type (string) --

            Specifies the endpoint capacity type.

            • INSTANCE_COUNT: The endpoint activates based on the number of instances.

            • CAPACITY_PERCENT: The endpoint activates based on the specified percentage of capacity.

          • Value (integer) --

            Defines the capacity size, either as a number of instances or a capacity percentage.

        • WaitIntervalInSeconds (integer) --

          The length of the baking period, during which SageMaker monitors alarms for each batch on the new fleet.

        • MaximumExecutionTimeoutInSeconds (integer) --

          The time limit for the total deployment. Exceeding this limit causes a timeout.

        • RollbackMaximumBatchSize (dict) --

          Batch size for rollback to the old endpoint fleet. Each rolling step to provision capacity and turn on traffic on the old endpoint fleet, and terminate capacity on the new endpoint fleet. If this field is absent, the default value will be set to 100% of total capacity which means to bring up the whole capacity of the old fleet at once during rollback.

          • Type (string) --

            Specifies the endpoint capacity type.

            • INSTANCE_COUNT: The endpoint activates based on the number of instances.

            • CAPACITY_PERCENT: The endpoint activates based on the specified percentage of capacity.

          • Value (integer) --

            Defines the capacity size, either as a number of instances or a capacity percentage.

      • AutoRollbackConfiguration (dict) --

        Automatic rollback configuration for handling endpoint deployment failures and recovery.

        • Alarms (list) --

          List of CloudWatch alarms in your account that are configured to monitor metrics on an endpoint. If any alarms are tripped during a deployment, SageMaker rolls back the deployment.

          • (dict) --

            An Amazon CloudWatch alarm configured to monitor metrics on an endpoint.

            • AlarmName (string) --

              The name of a CloudWatch alarm in your account.

    • AsyncInferenceConfig (dict) --

      Returns the description of an endpoint configuration created using the CreateEndpointConfig API.

      • ClientConfig (dict) --

        Configures the behavior of the client used by SageMaker to interact with the model container during asynchronous inference.

        • MaxConcurrentInvocationsPerInstance (integer) --

          The maximum number of concurrent requests sent by the SageMaker client to the model container. If no value is provided, SageMaker chooses an optimal value.

      • OutputConfig (dict) --

        Specifies the configuration for asynchronous inference invocation outputs.

        • KmsKeyId (string) --

          The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the asynchronous inference output in Amazon S3.

        • S3OutputPath (string) --

          The Amazon S3 location to upload inference responses to.

        • NotificationConfig (dict) --

          Specifies the configuration for notifications of inference results for asynchronous inference.

          • SuccessTopic (string) --

            Amazon SNS topic to post a notification to when inference completes successfully. If no topic is provided, no notification is sent on success.

          • ErrorTopic (string) --

            Amazon SNS topic to post a notification to when inference fails. If no topic is provided, no notification is sent on failure.

          • IncludeInferenceResponseIn (list) --

            The Amazon SNS topics where you want the inference response to be included.

            • (string) --

        • S3FailurePath (string) --

          The Amazon S3 location to upload failure inference responses to.

    • PendingDeploymentSummary (dict) --

      Returns the summary of an in-progress deployment. This field is only returned when the endpoint is creating or updating with a new endpoint configuration.

      • EndpointConfigName (string) --

        The name of the endpoint configuration used in the deployment.

      • ProductionVariants (list) --

        An array of PendingProductionVariantSummary objects, one for each model hosted behind this endpoint for the in-progress deployment.

        • (dict) --

          The production variant summary for a deployment when an endpoint is creating or updating with the CreateEndpoint or UpdateEndpoint operations. Describes the ``VariantStatus ``, weight and capacity for a production variant associated with an endpoint.

          • VariantName (string) --

            The name of the variant.

          • DeployedImages (list) --

            An array of DeployedImage objects that specify the Amazon EC2 Container Registry paths of the inference images deployed on instances of this ProductionVariant.

            • (dict) --

              Gets the Amazon EC2 Container Registry path of the docker image of the model that is hosted in this ProductionVariant.

              If you used the registry/repository[:tag] form to specify the image path of the primary container when you created the model hosted in this ProductionVariant, the path resolves to a path of the form registry/repository[@digest]. A digest is a hash value that identifies a specific version of an image. For information about Amazon ECR paths, see Pulling an Image in the Amazon ECR User Guide.

              • SpecifiedImage (string) --

                The image path you specified when you created the model.

              • ResolvedImage (string) --

                The specific digest path of the image hosted in this ProductionVariant.

              • ResolutionTime (datetime) --

                The date and time when the image path for the model resolved to the ResolvedImage

          • CurrentWeight (float) --

            The weight associated with the variant.

          • DesiredWeight (float) --

            The requested weight for the variant in this deployment, as specified in the endpoint configuration for the endpoint. The value is taken from the request to the CreateEndpointConfig operation.

          • CurrentInstanceCount (integer) --

            The number of instances associated with the variant.

          • DesiredInstanceCount (integer) --

            The number of instances requested in this deployment, as specified in the endpoint configuration for the endpoint. The value is taken from the request to the CreateEndpointConfig operation.

          • InstanceType (string) --

            The type of instances associated with the variant.

          • InstancePools (list) --

            A list of instance pools for the production variant. Each pool indicates the instance type and the current number of instances of that type.

            • (dict) --

              A summary of an instance pool for a production variant, including the instance type and the current number of instances.

              • InstanceType (string) --

                The ML compute instance type for the instance pool.

              • CurrentInstanceCount (integer) --

                The current number of instances of this type in the instance pool.

          • AcceleratorType (string) --

            This parameter is no longer supported. Elastic Inference (EI) is no longer available.

            This parameter was used to specify the size of the EI instance to use for the production variant.

          • VariantStatus (list) --

            The endpoint variant status which describes the current deployment stage status or operational status.

            • (dict) --

              Describes the status of the production variant.

              • Status (string) --

                The endpoint variant status which describes the current deployment stage status or operational status.

                • Creating: Creating inference resources for the production variant.

                • Deleting: Terminating inference resources for the production variant.

                • Updating: Updating capacity for the production variant.

                • ActivatingTraffic: Turning on traffic for the production variant.

                • Baking: Waiting period to monitor the CloudWatch alarms in the automatic rollback configuration.

              • StatusMessage (string) --

                A message that describes the status of the production variant.

              • StartTime (datetime) --

                The start time of the current status change.

          • CurrentServerlessConfig (dict) --

            The serverless configuration for the endpoint.

            • MemorySizeInMB (integer) --

              The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

            • MaxConcurrency (integer) --

              The maximum number of concurrent invocations your serverless endpoint can process.

            • ProvisionedConcurrency (integer) --

              The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

          • DesiredServerlessConfig (dict) --

            The serverless configuration requested for this deployment, as specified in the endpoint configuration for the endpoint.

            • MemorySizeInMB (integer) --

              The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

            • MaxConcurrency (integer) --

              The maximum number of concurrent invocations your serverless endpoint can process.

            • ProvisionedConcurrency (integer) --

              The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

          • ManagedInstanceScaling (dict) --

            Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

            • Status (string) --

              Indicates whether managed instance scaling is enabled.

            • MinInstanceCount (integer) --

              The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

            • MaxInstanceCount (integer) --

              The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

            • ScaleInPolicy (dict) --

              Configures the scale-in behavior for managed instance scaling.

              • Strategy (string) --

                The strategy for scaling in instances.

                IDLE_RELEASE

                Releases instances that have no hosted inference component copies.

                CONSOLIDATION

                Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

              • MaximumStepSize (integer) --

                The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

                Default value: 1.

              • CooldownInMinutes (integer) --

                The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

                Default value: 20.

          • RoutingConfig (dict) --

            Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

            • RoutingStrategy (string) --

              Sets how the endpoint routes incoming traffic:

              • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

              • RANDOM: The endpoint routes each request to a randomly chosen instance.

              • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

              • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

            • PrefixAwareRoutingConfig (dict) --

              The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

              • PrefixLength (integer) --

                The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

                • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

                • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

                The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

              • ConcurrencyThreshold (integer) --

                The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

                On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

                On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

                The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

          • AcceleratorPartitionLayouts (list) --

            The accelerator partition layout that SageMaker is applying to this production variant. On NVIDIA GPUs, the profiles are Multi-Instance GPU (MIG) profiles. SageMaker returns the accelerator partition layout as you provided it in the pending deployment configuration, without normalization.

            • (dict) --

              A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

              • AcceleratorPartitionTypes (list) --

                The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

                • (string) --

      • StartTime (datetime) --

        The start time of the deployment.

      • ShadowProductionVariants (list) --

        An array of PendingProductionVariantSummary objects, one for each model hosted behind this endpoint in shadow mode with production traffic replicated from the model specified on ProductionVariants for the in-progress deployment.

        • (dict) --

          The production variant summary for a deployment when an endpoint is creating or updating with the CreateEndpoint or UpdateEndpoint operations. Describes the ``VariantStatus ``, weight and capacity for a production variant associated with an endpoint.

          • VariantName (string) --

            The name of the variant.

          • DeployedImages (list) --

            An array of DeployedImage objects that specify the Amazon EC2 Container Registry paths of the inference images deployed on instances of this ProductionVariant.

            • (dict) --

              Gets the Amazon EC2 Container Registry path of the docker image of the model that is hosted in this ProductionVariant.

              If you used the registry/repository[:tag] form to specify the image path of the primary container when you created the model hosted in this ProductionVariant, the path resolves to a path of the form registry/repository[@digest]. A digest is a hash value that identifies a specific version of an image. For information about Amazon ECR paths, see Pulling an Image in the Amazon ECR User Guide.

              • SpecifiedImage (string) --

                The image path you specified when you created the model.

              • ResolvedImage (string) --

                The specific digest path of the image hosted in this ProductionVariant.

              • ResolutionTime (datetime) --

                The date and time when the image path for the model resolved to the ResolvedImage

          • CurrentWeight (float) --

            The weight associated with the variant.

          • DesiredWeight (float) --

            The requested weight for the variant in this deployment, as specified in the endpoint configuration for the endpoint. The value is taken from the request to the CreateEndpointConfig operation.

          • CurrentInstanceCount (integer) --

            The number of instances associated with the variant.

          • DesiredInstanceCount (integer) --

            The number of instances requested in this deployment, as specified in the endpoint configuration for the endpoint. The value is taken from the request to the CreateEndpointConfig operation.

          • InstanceType (string) --

            The type of instances associated with the variant.

          • InstancePools (list) --

            A list of instance pools for the production variant. Each pool indicates the instance type and the current number of instances of that type.

            • (dict) --

              A summary of an instance pool for a production variant, including the instance type and the current number of instances.

              • InstanceType (string) --

                The ML compute instance type for the instance pool.

              • CurrentInstanceCount (integer) --

                The current number of instances of this type in the instance pool.

          • AcceleratorType (string) --

            This parameter is no longer supported. Elastic Inference (EI) is no longer available.

            This parameter was used to specify the size of the EI instance to use for the production variant.

          • VariantStatus (list) --

            The endpoint variant status which describes the current deployment stage status or operational status.

            • (dict) --

              Describes the status of the production variant.

              • Status (string) --

                The endpoint variant status which describes the current deployment stage status or operational status.

                • Creating: Creating inference resources for the production variant.

                • Deleting: Terminating inference resources for the production variant.

                • Updating: Updating capacity for the production variant.

                • ActivatingTraffic: Turning on traffic for the production variant.

                • Baking: Waiting period to monitor the CloudWatch alarms in the automatic rollback configuration.

              • StatusMessage (string) --

                A message that describes the status of the production variant.

              • StartTime (datetime) --

                The start time of the current status change.

          • CurrentServerlessConfig (dict) --

            The serverless configuration for the endpoint.

            • MemorySizeInMB (integer) --

              The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

            • MaxConcurrency (integer) --

              The maximum number of concurrent invocations your serverless endpoint can process.

            • ProvisionedConcurrency (integer) --

              The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

          • DesiredServerlessConfig (dict) --

            The serverless configuration requested for this deployment, as specified in the endpoint configuration for the endpoint.

            • MemorySizeInMB (integer) --

              The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

            • MaxConcurrency (integer) --

              The maximum number of concurrent invocations your serverless endpoint can process.

            • ProvisionedConcurrency (integer) --

              The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

          • ManagedInstanceScaling (dict) --

            Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

            • Status (string) --

              Indicates whether managed instance scaling is enabled.

            • MinInstanceCount (integer) --

              The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

            • MaxInstanceCount (integer) --

              The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

            • ScaleInPolicy (dict) --

              Configures the scale-in behavior for managed instance scaling.

              • Strategy (string) --

                The strategy for scaling in instances.

                IDLE_RELEASE

                Releases instances that have no hosted inference component copies.

                CONSOLIDATION

                Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

              • MaximumStepSize (integer) --

                The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

                Default value: 1.

              • CooldownInMinutes (integer) --

                The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

                Default value: 20.

          • RoutingConfig (dict) --

            Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

            • RoutingStrategy (string) --

              Sets how the endpoint routes incoming traffic:

              • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

              • RANDOM: The endpoint routes each request to a randomly chosen instance.

              • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

              • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

            • PrefixAwareRoutingConfig (dict) --

              The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

              • PrefixLength (integer) --

                The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

                • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

                • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

                The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

              • ConcurrencyThreshold (integer) --

                The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

                On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

                On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

                The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

          • AcceleratorPartitionLayouts (list) --

            The accelerator partition layout that SageMaker is applying to this production variant. On NVIDIA GPUs, the profiles are Multi-Instance GPU (MIG) profiles. SageMaker returns the accelerator partition layout as you provided it in the pending deployment configuration, without normalization.

            • (dict) --

              A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

              • AcceleratorPartitionTypes (list) --

                The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

                • (string) --

    • ExplainerConfig (dict) --

      The configuration parameters for an explainer.

      • ClarifyExplainerConfig (dict) --

        A member of ExplainerConfig that contains configuration parameters for the SageMaker Clarify explainer.

        • EnableExplanations (string) --

          A JMESPath boolean expression used to filter which records to explain. Explanations are activated by default. See `EnableExplanations <https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-online-explainability-create-endpoint.html#clarify-online-explainability-create-endpoint-enable>`__for additional information.

        • InferenceConfig (dict) --

          The inference configuration parameter for the model container.

          • FeaturesAttribute (string) --

            Provides the JMESPath expression to extract the features from a model container input in JSON Lines format. For example, if FeaturesAttribute is the JMESPath expression 'myfeatures', it extracts a list of features [1,2,3] from request data '{"myfeatures":[1,2,3]}'.

          • ContentTemplate (string) --

            A template string used to format a JSON record into an acceptable model container input. For example, a ContentTemplate string '{"myfeatures":$features}' will format a list of features [1,2,3] into the record string '{"myfeatures":[1,2,3]}'. Required only when the model container input is in JSON Lines format.

          • MaxRecordCount (integer) --

            The maximum number of records in a request that the model container can process when querying the model container for the predictions of a synthetic dataset. A record is a unit of input data that inference can be made on, for example, a single line in CSV data. If MaxRecordCount is 1, the model container expects one record per request. A value of 2 or greater means that the model expects batch requests, which can reduce overhead and speed up the inferencing process. If this parameter is not provided, the explainer will tune the record count per request according to the model container's capacity at runtime.

          • MaxPayloadInMB (integer) --

            The maximum payload size (MB) allowed of a request from the explainer to the model container. Defaults to 6 MB.

          • ProbabilityIndex (integer) --

            A zero-based index used to extract a probability value (score) or list from model container output in CSV format. If this value is not provided, the entire model container output will be treated as a probability value (score) or list.

            Example for a single class model: If the model container output consists of a string-formatted prediction label followed by its probability: '1,0.6', set ProbabilityIndex to 1 to select the probability value 0.6.

            Example for a multiclass model: If the model container output consists of a string-formatted prediction label followed by its probability: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set ProbabilityIndex to 1 to select the probability values [0.1,0.6,0.3].

          • LabelIndex (integer) --

            A zero-based index used to extract a label header or list of label headers from model container output in CSV format.

            Example for a multiclass model: If the model container output consists of label headers followed by probabilities: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set LabelIndex to 0 to select the label headers ['cat','dog','fish'].

          • ProbabilityAttribute (string) --

            A JMESPath expression used to extract the probability (or score) from the model container output if the model container is in JSON Lines format.

            Example: If the model container output of a single request is '{"predicted_label":1,"probability":0.6}', then set ProbabilityAttribute to 'probability'.

          • LabelAttribute (string) --

            A JMESPath expression used to locate the list of label headers in the model container output.

            Example: If the model container output of a batch request is '{"labels":["cat","dog","fish"],"probability":[0.6,0.3,0.1]}', then set LabelAttribute to 'labels' to extract the list of label headers ["cat","dog","fish"]

          • LabelHeaders (list) --

            For multiclass classification problems, the label headers are the names of the classes. Otherwise, the label header is the name of the predicted label. These are used to help readability for the output of the InvokeEndpoint API. See the response section under Invoke the endpoint in the Developer Guide for more information. If there are no label headers in the model container output, provide them manually using this parameter.

            • (string) --

          • FeatureHeaders (list) --

            The names of the features. If provided, these are included in the endpoint response payload to help readability of the InvokeEndpoint output. See the Response section under Invoke the endpoint in the Developer Guide for more information.

            • (string) --

          • FeatureTypes (list) --

            A list of data types of the features (optional). Applicable only to NLP explainability. If provided, FeatureTypes must have at least one 'text' string (for example, ['text']). If FeatureTypes is not provided, the explainer infers the feature types based on the baseline data. The feature types are included in the endpoint response payload. For additional information see the response section under Invoke the endpoint in the Developer Guide for more information.

            • (string) --

        • ShapConfig (dict) --

          The configuration for SHAP analysis.

          • ShapBaselineConfig (dict) --

            The configuration for the SHAP baseline of the Kernal SHAP algorithm.

            • MimeType (string) --

              The MIME type of the baseline data. Choose from 'text/csv' or 'application/jsonlines'. Defaults to 'text/csv'.

            • ShapBaseline (string) --

              The inline SHAP baseline data in string format. ShapBaseline can have one or multiple records to be used as the baseline dataset. The format of the SHAP baseline file should be the same format as the training dataset. For example, if the training dataset is in CSV format and each record contains four features, and all features are numerical, then the format of the baseline data should also share these characteristics. For natural language processing (NLP) of text columns, the baseline value should be the value used to replace the unit of text specified by the Granularity of the TextConfig parameter. The size limit for ShapBasline is 4 KB. Use the ShapBaselineUri parameter if you want to provide more than 4 KB of baseline data.

            • ShapBaselineUri (string) --

              The uniform resource identifier (URI) of the S3 bucket where the SHAP baseline file is stored. The format of the SHAP baseline file should be the same format as the format of the training dataset. For example, if the training dataset is in CSV format, and each record in the training dataset has four features, and all features are numerical, then the baseline file should also have this same format. Each record should contain only the features. If you are using a virtual private cloud (VPC), the ShapBaselineUri should be accessible to the VPC. For more information about setting up endpoints with Amazon Virtual Private Cloud, see Give SageMaker access to Resources in your Amazon Virtual Private Cloud.

          • NumberOfSamples (integer) --

            The number of samples to be used for analysis by the Kernal SHAP algorithm.

          • UseLogit (boolean) --

            A Boolean toggle to indicate if you want to use the logit function (true) or log-odds units (false) for model predictions. Defaults to false.

          • Seed (integer) --

            The starting value used to initialize the random number generator in the explainer. Provide a value for this parameter to obtain a deterministic SHAP result.

          • TextConfig (dict) --

            A parameter that indicates if text features are treated as text and explanations are provided for individual units of text. Required for natural language processing (NLP) explainability only.

            • Language (string) --

              Specifies the language of the text features in ISO 639-1 or ISO 639-3 code of a supported language.

            • Granularity (string) --

              The unit of granularity for the analysis of text features. For example, if the unit is 'token', then each token (like a word in English) of the text is treated as a feature. SHAP values are computed for each unit/feature.

    • ShadowProductionVariants (list) --

      An array of ProductionVariantSummary objects, one for each model that you want to host at this endpoint in shadow mode with production traffic replicated from the model specified on ProductionVariants.

      • (dict) --

        Describes weight and capacities for a production variant associated with an endpoint. If you sent a request to the UpdateEndpointWeightsAndCapacities API and the endpoint status is Updating, you get different desired and current values.

        • VariantName (string) --

          The name of the variant.

        • DeployedImages (list) --

          An array of DeployedImage objects that specify the Amazon EC2 Container Registry paths of the inference images deployed on instances of this ProductionVariant.

          • (dict) --

            Gets the Amazon EC2 Container Registry path of the docker image of the model that is hosted in this ProductionVariant.

            If you used the registry/repository[:tag] form to specify the image path of the primary container when you created the model hosted in this ProductionVariant, the path resolves to a path of the form registry/repository[@digest]. A digest is a hash value that identifies a specific version of an image. For information about Amazon ECR paths, see Pulling an Image in the Amazon ECR User Guide.

            • SpecifiedImage (string) --

              The image path you specified when you created the model.

            • ResolvedImage (string) --

              The specific digest path of the image hosted in this ProductionVariant.

            • ResolutionTime (datetime) --

              The date and time when the image path for the model resolved to the ResolvedImage

        • CurrentWeight (float) --

          The weight associated with the variant.

        • DesiredWeight (float) --

          The requested weight, as specified in the UpdateEndpointWeightsAndCapacities request.

        • CurrentInstanceCount (integer) --

          The number of instances associated with the variant.

        • DesiredInstanceCount (integer) --

          The number of instances requested in the UpdateEndpointWeightsAndCapacities request.

        • InstancePools (list) --

          A list of instance pools for the production variant. Each pool indicates the instance type and the current number of instances of that type.

          • (dict) --

            A summary of an instance pool for a production variant, including the instance type and the current number of instances.

            • InstanceType (string) --

              The ML compute instance type for the instance pool.

            • CurrentInstanceCount (integer) --

              The current number of instances of this type in the instance pool.

        • VariantStatus (list) --

          The endpoint variant status which describes the current deployment stage status or operational status.

          • (dict) --

            Describes the status of the production variant.

            • Status (string) --

              The endpoint variant status which describes the current deployment stage status or operational status.

              • Creating: Creating inference resources for the production variant.

              • Deleting: Terminating inference resources for the production variant.

              • Updating: Updating capacity for the production variant.

              • ActivatingTraffic: Turning on traffic for the production variant.

              • Baking: Waiting period to monitor the CloudWatch alarms in the automatic rollback configuration.

            • StatusMessage (string) --

              A message that describes the status of the production variant.

            • StartTime (datetime) --

              The start time of the current status change.

        • CurrentServerlessConfig (dict) --

          The serverless configuration for the endpoint.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • DesiredServerlessConfig (dict) --

          The serverless configuration requested for the endpoint update.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • ManagedInstanceScaling (dict) --

          Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

          • Status (string) --

            Indicates whether managed instance scaling is enabled.

          • MinInstanceCount (integer) --

            The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

          • MaxInstanceCount (integer) --

            The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

          • ScaleInPolicy (dict) --

            Configures the scale-in behavior for managed instance scaling.

            • Strategy (string) --

              The strategy for scaling in instances.

              IDLE_RELEASE

              Releases instances that have no hosted inference component copies.

              CONSOLIDATION

              Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

            • MaximumStepSize (integer) --

              The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

              Default value: 1.

            • CooldownInMinutes (integer) --

              The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

              Default value: 20.

        • RoutingConfig (dict) --

          Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

          • RoutingStrategy (string) --

            Sets how the endpoint routes incoming traffic:

            • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

            • RANDOM: The endpoint routes each request to a randomly chosen instance.

            • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

            • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

          • PrefixAwareRoutingConfig (dict) --

            The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

            • PrefixLength (integer) --

              The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

              • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

              • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

              The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

            • ConcurrencyThreshold (integer) --

              The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

              On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

              On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

              The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

        • CapacityReservationConfig (dict) --

          Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

          • MlReservationArn (string) --

            The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

          • CapacityReservationPreference (string) --

            The option that you chose for the capacity reservation. SageMaker AI supports the following options:

            capacity-reservations-only

            SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

          • TotalInstanceCount (integer) --

            The number of instances that you allocated to the ML capacity reservation.

          • AvailableInstanceCount (integer) --

            The number of instances that are currently available in the ML capacity reservation.

          • UsedByCurrentEndpoint (integer) --

            The number of instances from the ML capacity reservation that are being used by the endpoint.

          • Ec2CapacityReservations (list) --

            The EC2 capacity reservations that are shared to this ML capacity reservation, if any.

            • (dict) --

              The EC2 capacity reservations that are shared to an ML capacity reservation.

              • Ec2CapacityReservationId (string) --

                The unique identifier for an EC2 capacity reservation that's part of the ML capacity reservation.

              • TotalInstanceCount (integer) --

                The number of instances that you allocated to the EC2 capacity reservation.

              • AvailableInstanceCount (integer) --

                The number of instances that are currently available in the EC2 capacity reservation.

              • UsedByCurrentEndpoint (integer) --

                The number of instances from the EC2 capacity reservation that are being used by the endpoint.

        • AcceleratorPartitionLayouts (list) --

          The accelerator partition layout that SageMaker applies to this production variant. On NVIDIA GPUs, the profiles are Multi-Instance GPU (MIG) profiles. SageMaker returns the accelerator partition layout as you provided it when you created the variant, without normalization.

          • (dict) --

            A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

            • AcceleratorPartitionTypes (list) --

              The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

              • (string) --

    • MetricsConfig (dict) --

      The configuration parameters for utilization metrics.

      • EnableEnhancedMetrics (boolean) --

        Specifies whether to enable enhanced metrics for the endpoint. Enhanced metrics provide utilization and invocation data at instance and container granularity. Container granularity is supported for Inference Components. The default is False.

      • EnableDetailedObservability (boolean) --

        Indicates whether detailed observability is enabled for the endpoint. When set to True, the following metrics are published at the configured frequency:

        • Container-level inference metrics scraped from the container's Prometheus endpoint (such as request latency, error counts, and throughput). Available metrics vary by framework.

        • Per-GPU metrics (utilization, memory, and temperature) attributed to individual inference components.

        • Per-instance host metrics (CPU, memory, and disk utilization).

        • Inference component placement metrics (copy count per Availability Zone).

        For first-party and Deep Learning Containers (DLC), the Prometheus endpoint path is determined automatically. For Bring-Your-Own-Container (BYOC) cases, you can optionally set ContainerMetricsConfig to specify a custom endpoint path. If not specified, the default path /metrics on port 8080 is used.

        When set to False, these additional metrics are not published. Standard invocation and utilization metrics controlled by EnableEnhancedMetrics are unaffected.

        The default value for new endpoint configurations is True. For existing endpoint configurations created before this feature, the value is False unless explicitly set.

      • MetricPublishFrequencyInSeconds (integer) --

        The interval, in seconds, at which metrics are published to Amazon CloudWatch. Defaults to 60. Valid values: 10, 30, 60, 120, 180, 240, 300.

        When EnableEnhancedMetrics is set to False, this interval applies to utilization metrics only. Invocation metrics continue to be published at the default 60-second interval. When EnableEnhancedMetrics is set to True, this interval applies to both utilization and invocation metrics.

        When EnableDetailedObservability is set to True, this interval applies to per-GPU metrics, per-instance host metrics, container metrics, and fleet-level inference component lifecycle and placement metrics.

    • CrossRegionTrafficConfig (dict) --

      The cross-Region traffic cap, if the endpoint configuration sets one.

      • MaxInvocationsPerSecondPerInstance (integer) --

        The number of invocations per second that you expect one InService instance to serve. One value applies to every production variant, whatever its instance type.

      • MaxCrossRegionPercentPerEndpoint (integer) --

        The percentage of the endpoint's capacity that requests from inference routers in other Amazon Web Services Regions can use.

DescribeEndpointConfig (updated) Link ¶
Changes (response)
{'CrossRegionTrafficConfig': {'MaxCrossRegionPercentPerEndpoint': 'integer',
                              'MaxInvocationsPerSecondPerInstance': 'integer'},
 'ProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}},
 'ShadowProductionVariants': {'RoutingConfig': {'RoutingStrategy': {'KV_AWARE'}}}}

Returns the description of an endpoint configuration created using the CreateEndpointConfig API.

See also: AWS API Documentation

Request Syntax

client.describe_endpoint_config(
    EndpointConfigName='string'
)
type EndpointConfigName:

string

param EndpointConfigName:

[REQUIRED]

The name of the endpoint configuration.

rtype:

dict

returns:

Response Syntax

{
    'EndpointConfigName': 'string',
    'EndpointConfigArn': 'string',
    'ProductionVariants': [
        {
            'VariantName': 'string',
            'ModelName': 'string',
            'InitialInstanceCount': 123,
            'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'ModelNameOverride': 'string',
                    'Priority': 123
                },
            ],
            'VariantInstanceProvisionTimeoutInSeconds': 123,
            'InitialVariantWeight': ...,
            'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
            'CoreDumpConfig': {
                'DestinationS3Uri': 'string',
                'KmsKeyId': 'string'
            },
            'ServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'VolumeSizeInGB': 123,
            'ModelDataDownloadTimeoutInSeconds': 123,
            'ContainerStartupHealthCheckTimeoutInSeconds': 123,
            'EnableSSMAccess': True|False,
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'InferenceAmiVersion': 'al2-ami-sagemaker-inference-gpu-2'|'al2-ami-sagemaker-inference-gpu-2-1'|'al2-ami-sagemaker-inference-gpu-3-1'|'al2-ami-sagemaker-inference-neuron-2'|'al2023-ami-sagemaker-inference-gpu-4-1',
            'CapacityReservationConfig': {
                'CapacityReservationPreference': 'capacity-reservations-only',
                'MlReservationArn': 'string'
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    'DataCaptureConfig': {
        'EnableCapture': True|False,
        'InitialSamplingPercentage': 123,
        'DestinationS3Uri': 'string',
        'KmsKeyId': 'string',
        'CaptureOptions': [
            {
                'CaptureMode': 'Input'|'Output'|'InputAndOutput'
            },
        ],
        'CaptureContentTypeHeader': {
            'CsvContentTypes': [
                'string',
            ],
            'JsonContentTypes': [
                'string',
            ]
        }
    },
    'KmsKeyId': 'string',
    'CreationTime': datetime(2015, 1, 1),
    'AsyncInferenceConfig': {
        'ClientConfig': {
            'MaxConcurrentInvocationsPerInstance': 123
        },
        'OutputConfig': {
            'KmsKeyId': 'string',
            'S3OutputPath': 'string',
            'NotificationConfig': {
                'SuccessTopic': 'string',
                'ErrorTopic': 'string',
                'IncludeInferenceResponseIn': [
                    'SUCCESS_NOTIFICATION_TOPIC'|'ERROR_NOTIFICATION_TOPIC',
                ]
            },
            'S3FailurePath': 'string'
        }
    },
    'ExplainerConfig': {
        'ClarifyExplainerConfig': {
            'EnableExplanations': 'string',
            'InferenceConfig': {
                'FeaturesAttribute': 'string',
                'ContentTemplate': 'string',
                'MaxRecordCount': 123,
                'MaxPayloadInMB': 123,
                'ProbabilityIndex': 123,
                'LabelIndex': 123,
                'ProbabilityAttribute': 'string',
                'LabelAttribute': 'string',
                'LabelHeaders': [
                    'string',
                ],
                'FeatureHeaders': [
                    'string',
                ],
                'FeatureTypes': [
                    'numerical'|'categorical'|'text',
                ]
            },
            'ShapConfig': {
                'ShapBaselineConfig': {
                    'MimeType': 'string',
                    'ShapBaseline': 'string',
                    'ShapBaselineUri': 'string'
                },
                'NumberOfSamples': 123,
                'UseLogit': True|False,
                'Seed': 123,
                'TextConfig': {
                    'Language': 'af'|'sq'|'ar'|'hy'|'eu'|'bn'|'bg'|'ca'|'zh'|'hr'|'cs'|'da'|'nl'|'en'|'et'|'fi'|'fr'|'de'|'el'|'gu'|'he'|'hi'|'hu'|'is'|'id'|'ga'|'it'|'kn'|'ky'|'lv'|'lt'|'lb'|'mk'|'ml'|'mr'|'ne'|'nb'|'fa'|'pl'|'pt'|'ro'|'ru'|'sa'|'sr'|'tn'|'si'|'sk'|'sl'|'es'|'sv'|'tl'|'ta'|'tt'|'te'|'tr'|'uk'|'ur'|'yo'|'lij'|'xx',
                    'Granularity': 'token'|'sentence'|'paragraph'
                }
            }
        }
    },
    'ShadowProductionVariants': [
        {
            'VariantName': 'string',
            'ModelName': 'string',
            'InitialInstanceCount': 123,
            'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
            'InstancePools': [
                {
                    'InstanceType': 'ml.t2.medium'|'ml.t2.large'|'ml.t2.xlarge'|'ml.t2.2xlarge'|'ml.m4.xlarge'|'ml.m4.2xlarge'|'ml.m4.4xlarge'|'ml.m4.10xlarge'|'ml.m4.16xlarge'|'ml.m5.large'|'ml.m5.xlarge'|'ml.m5.2xlarge'|'ml.m5.4xlarge'|'ml.m5.12xlarge'|'ml.m5.24xlarge'|'ml.m5d.large'|'ml.m5d.xlarge'|'ml.m5d.2xlarge'|'ml.m5d.4xlarge'|'ml.m5d.12xlarge'|'ml.m5d.24xlarge'|'ml.c4.large'|'ml.c4.xlarge'|'ml.c4.2xlarge'|'ml.c4.4xlarge'|'ml.c4.8xlarge'|'ml.p2.xlarge'|'ml.p2.8xlarge'|'ml.p2.16xlarge'|'ml.p3.2xlarge'|'ml.p3.8xlarge'|'ml.p3.16xlarge'|'ml.c5.large'|'ml.c5.xlarge'|'ml.c5.2xlarge'|'ml.c5.4xlarge'|'ml.c5.9xlarge'|'ml.c5.18xlarge'|'ml.c5d.large'|'ml.c5d.xlarge'|'ml.c5d.2xlarge'|'ml.c5d.4xlarge'|'ml.c5d.9xlarge'|'ml.c5d.18xlarge'|'ml.g4dn.xlarge'|'ml.g4dn.2xlarge'|'ml.g4dn.4xlarge'|'ml.g4dn.8xlarge'|'ml.g4dn.12xlarge'|'ml.g4dn.16xlarge'|'ml.r5.large'|'ml.r5.xlarge'|'ml.r5.2xlarge'|'ml.r5.4xlarge'|'ml.r5.12xlarge'|'ml.r5.24xlarge'|'ml.r5d.large'|'ml.r5d.xlarge'|'ml.r5d.2xlarge'|'ml.r5d.4xlarge'|'ml.r5d.12xlarge'|'ml.r5d.24xlarge'|'ml.inf1.xlarge'|'ml.inf1.2xlarge'|'ml.inf1.6xlarge'|'ml.inf1.24xlarge'|'ml.dl1.24xlarge'|'ml.c6i.large'|'ml.c6i.xlarge'|'ml.c6i.2xlarge'|'ml.c6i.4xlarge'|'ml.c6i.8xlarge'|'ml.c6i.12xlarge'|'ml.c6i.16xlarge'|'ml.c6i.24xlarge'|'ml.c6i.32xlarge'|'ml.m6i.large'|'ml.m6i.xlarge'|'ml.m6i.2xlarge'|'ml.m6i.4xlarge'|'ml.m6i.8xlarge'|'ml.m6i.12xlarge'|'ml.m6i.16xlarge'|'ml.m6i.24xlarge'|'ml.m6i.32xlarge'|'ml.r6i.large'|'ml.r6i.xlarge'|'ml.r6i.2xlarge'|'ml.r6i.4xlarge'|'ml.r6i.8xlarge'|'ml.r6i.12xlarge'|'ml.r6i.16xlarge'|'ml.r6i.24xlarge'|'ml.r6i.32xlarge'|'ml.g5.xlarge'|'ml.g5.2xlarge'|'ml.g5.4xlarge'|'ml.g5.8xlarge'|'ml.g5.12xlarge'|'ml.g5.16xlarge'|'ml.g5.24xlarge'|'ml.g5.48xlarge'|'ml.g6.xlarge'|'ml.g6.2xlarge'|'ml.g6.4xlarge'|'ml.g6.8xlarge'|'ml.g6.12xlarge'|'ml.g6.16xlarge'|'ml.g6.24xlarge'|'ml.g6.48xlarge'|'ml.r8g.medium'|'ml.r8g.large'|'ml.r8g.xlarge'|'ml.r8g.2xlarge'|'ml.r8g.4xlarge'|'ml.r8g.8xlarge'|'ml.r8g.12xlarge'|'ml.r8g.16xlarge'|'ml.r8g.24xlarge'|'ml.r8g.48xlarge'|'ml.g6e.xlarge'|'ml.g6e.2xlarge'|'ml.g6e.4xlarge'|'ml.g6e.8xlarge'|'ml.g6e.12xlarge'|'ml.g6e.16xlarge'|'ml.g6e.24xlarge'|'ml.g6e.48xlarge'|'ml.g7e.2xlarge'|'ml.g7e.4xlarge'|'ml.g7e.8xlarge'|'ml.g7e.12xlarge'|'ml.g7e.24xlarge'|'ml.g7e.48xlarge'|'ml.g7.2xlarge'|'ml.g7.4xlarge'|'ml.g7.8xlarge'|'ml.g7.12xlarge'|'ml.g7.24xlarge'|'ml.g7.48xlarge'|'ml.p4d.24xlarge'|'ml.c7g.large'|'ml.c7g.xlarge'|'ml.c7g.2xlarge'|'ml.c7g.4xlarge'|'ml.c7g.8xlarge'|'ml.c7g.12xlarge'|'ml.c7g.16xlarge'|'ml.m6g.large'|'ml.m6g.xlarge'|'ml.m6g.2xlarge'|'ml.m6g.4xlarge'|'ml.m6g.8xlarge'|'ml.m6g.12xlarge'|'ml.m6g.16xlarge'|'ml.m6gd.large'|'ml.m6gd.xlarge'|'ml.m6gd.2xlarge'|'ml.m6gd.4xlarge'|'ml.m6gd.8xlarge'|'ml.m6gd.12xlarge'|'ml.m6gd.16xlarge'|'ml.c6g.large'|'ml.c6g.xlarge'|'ml.c6g.2xlarge'|'ml.c6g.4xlarge'|'ml.c6g.8xlarge'|'ml.c6g.12xlarge'|'ml.c6g.16xlarge'|'ml.c6gd.large'|'ml.c6gd.xlarge'|'ml.c6gd.2xlarge'|'ml.c6gd.4xlarge'|'ml.c6gd.8xlarge'|'ml.c6gd.12xlarge'|'ml.c6gd.16xlarge'|'ml.c6gn.large'|'ml.c6gn.xlarge'|'ml.c6gn.2xlarge'|'ml.c6gn.4xlarge'|'ml.c6gn.8xlarge'|'ml.c6gn.12xlarge'|'ml.c6gn.16xlarge'|'ml.r6g.large'|'ml.r6g.xlarge'|'ml.r6g.2xlarge'|'ml.r6g.4xlarge'|'ml.r6g.8xlarge'|'ml.r6g.12xlarge'|'ml.r6g.16xlarge'|'ml.r6gd.large'|'ml.r6gd.xlarge'|'ml.r6gd.2xlarge'|'ml.r6gd.4xlarge'|'ml.r6gd.8xlarge'|'ml.r6gd.12xlarge'|'ml.r6gd.16xlarge'|'ml.p4de.24xlarge'|'ml.trn1.2xlarge'|'ml.trn1.32xlarge'|'ml.trn1n.32xlarge'|'ml.trn2.48xlarge'|'ml.inf2.xlarge'|'ml.inf2.8xlarge'|'ml.inf2.24xlarge'|'ml.inf2.48xlarge'|'ml.p5.48xlarge'|'ml.p5e.48xlarge'|'ml.p5en.48xlarge'|'ml.m7i.large'|'ml.m7i.xlarge'|'ml.m7i.2xlarge'|'ml.m7i.4xlarge'|'ml.m7i.8xlarge'|'ml.m7i.12xlarge'|'ml.m7i.16xlarge'|'ml.m7i.24xlarge'|'ml.m7i.48xlarge'|'ml.c7i.large'|'ml.c7i.xlarge'|'ml.c7i.2xlarge'|'ml.c7i.4xlarge'|'ml.c7i.8xlarge'|'ml.c7i.12xlarge'|'ml.c7i.16xlarge'|'ml.c7i.24xlarge'|'ml.c7i.48xlarge'|'ml.r7i.large'|'ml.r7i.xlarge'|'ml.r7i.2xlarge'|'ml.r7i.4xlarge'|'ml.r7i.8xlarge'|'ml.r7i.12xlarge'|'ml.r7i.16xlarge'|'ml.r7i.24xlarge'|'ml.r7i.48xlarge'|'ml.c8g.medium'|'ml.c8g.large'|'ml.c8g.xlarge'|'ml.c8g.2xlarge'|'ml.c8g.4xlarge'|'ml.c8g.8xlarge'|'ml.c8g.12xlarge'|'ml.c8g.16xlarge'|'ml.c8g.24xlarge'|'ml.c8g.48xlarge'|'ml.r7gd.medium'|'ml.r7gd.large'|'ml.r7gd.xlarge'|'ml.r7gd.2xlarge'|'ml.r7gd.4xlarge'|'ml.r7gd.8xlarge'|'ml.r7gd.12xlarge'|'ml.r7gd.16xlarge'|'ml.m8g.medium'|'ml.m8g.large'|'ml.m8g.xlarge'|'ml.m8g.2xlarge'|'ml.m8g.4xlarge'|'ml.m8g.8xlarge'|'ml.m8g.12xlarge'|'ml.m8g.16xlarge'|'ml.m8g.24xlarge'|'ml.m8g.48xlarge'|'ml.c6in.large'|'ml.c6in.xlarge'|'ml.c6in.2xlarge'|'ml.c6in.4xlarge'|'ml.c6in.8xlarge'|'ml.c6in.12xlarge'|'ml.c6in.16xlarge'|'ml.c6in.24xlarge'|'ml.c6in.32xlarge'|'ml.p6-b200.48xlarge'|'ml.p6-b300.48xlarge'|'ml.p6e-gb200.36xlarge'|'ml.p5.4xlarge',
                    'ModelNameOverride': 'string',
                    'Priority': 123
                },
            ],
            'VariantInstanceProvisionTimeoutInSeconds': 123,
            'InitialVariantWeight': ...,
            'AcceleratorType': 'ml.eia1.medium'|'ml.eia1.large'|'ml.eia1.xlarge'|'ml.eia2.medium'|'ml.eia2.large'|'ml.eia2.xlarge',
            'CoreDumpConfig': {
                'DestinationS3Uri': 'string',
                'KmsKeyId': 'string'
            },
            'ServerlessConfig': {
                'MemorySizeInMB': 123,
                'MaxConcurrency': 123,
                'ProvisionedConcurrency': 123
            },
            'VolumeSizeInGB': 123,
            'ModelDataDownloadTimeoutInSeconds': 123,
            'ContainerStartupHealthCheckTimeoutInSeconds': 123,
            'EnableSSMAccess': True|False,
            'ManagedInstanceScaling': {
                'Status': 'ENABLED'|'DISABLED',
                'MinInstanceCount': 123,
                'MaxInstanceCount': 123,
                'ScaleInPolicy': {
                    'Strategy': 'IDLE_RELEASE'|'CONSOLIDATION',
                    'MaximumStepSize': 123,
                    'CooldownInMinutes': 123
                }
            },
            'RoutingConfig': {
                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'|'RANDOM'|'PREFIX_AWARE'|'KV_AWARE',
                'PrefixAwareRoutingConfig': {
                    'PrefixLength': 123,
                    'ConcurrencyThreshold': 123
                }
            },
            'InferenceAmiVersion': 'al2-ami-sagemaker-inference-gpu-2'|'al2-ami-sagemaker-inference-gpu-2-1'|'al2-ami-sagemaker-inference-gpu-3-1'|'al2-ami-sagemaker-inference-neuron-2'|'al2023-ami-sagemaker-inference-gpu-4-1',
            'CapacityReservationConfig': {
                'CapacityReservationPreference': 'capacity-reservations-only',
                'MlReservationArn': 'string'
            },
            'AcceleratorPartitionLayouts': [
                {
                    'AcceleratorPartitionTypes': [
                        'string',
                    ]
                },
            ]
        },
    ],
    'ExecutionRoleArn': 'string',
    'VpcConfig': {
        'SecurityGroupIds': [
            'string',
        ],
        'Subnets': [
            'string',
        ]
    },
    'EnableNetworkIsolation': True|False,
    'MetricsConfig': {
        'EnableEnhancedMetrics': True|False,
        'EnableDetailedObservability': True|False,
        'MetricPublishFrequencyInSeconds': 123
    },
    'CrossRegionTrafficConfig': {
        'MaxInvocationsPerSecondPerInstance': 123,
        'MaxCrossRegionPercentPerEndpoint': 123
    }
}

Response Structure

  • (dict) --

    • EndpointConfigName (string) --

      Name of the SageMaker endpoint configuration.

    • EndpointConfigArn (string) --

      The Amazon Resource Name (ARN) of the endpoint configuration.

    • ProductionVariants (list) --

      An array of ProductionVariant objects, one for each model that you want to host at this endpoint.

      • (dict) --

        Identifies a model that you want to host and the resources chosen to deploy for hosting it. If you are deploying multiple models, tell SageMaker how to distribute traffic among the models by specifying variant weights. For more information on production variants, check Production variants.

        • VariantName (string) --

          The name of the production variant.

        • ModelName (string) --

          The name of the model that you want to host. This is the name that you specified when creating the model.

        • InitialInstanceCount (integer) --

          Number of instances to launch initially.

        • InstanceType (string) --

          The ML compute instance type.

        • InstancePools (list) --

          A list of instance pools for the production variant. Each instance pool specifies an instance type and its priority for provisioning. Use instance pools to configure heterogeneous endpoints that deploy models across multiple instance types.

          • (dict) --

            Specifies an instance type and its priority for a heterogeneous endpoint. Use instance pools to configure a production variant with multiple instance types, enabling the endpoint to provision instances across different types based on priority.

            • InstanceType (string) --

              The ML compute instance type for the instance pool.

            • ModelNameOverride (string) --

              The name of a SageMaker model to use for this instance pool instead of the model specified for the production variant. Use this to deploy a different model optimized for the instance type in this pool.

            • Priority (integer) --

              The priority for the instance pool. SageMaker attempts to provision instances in order of priority, starting with the lowest value. If instances for a higher-priority pool are unavailable, SageMaker attempts to provision from the next pool.

              Valid values: 1 to 5, where 1 is the highest priority.

        • VariantInstanceProvisionTimeoutInSeconds (integer) --

          The timeout value, in seconds, for provisioning instances for the production variant. When SageMaker encounters an insufficient capacity error while provisioning instances, it retries with the next instance pool (if configured) or waits until the timeout expires. This timeout applies only to capacity provisioning and does not include the time for model download or container startup.

          Valid values: 300 to 3600.

        • InitialVariantWeight (float) --

          Determines initial traffic distribution among all of the models that you specify in the endpoint configuration. The traffic to a production variant is determined by the ratio of the VariantWeight to the sum of all VariantWeight values across all ProductionVariants. If unspecified, it defaults to 1.0.

        • AcceleratorType (string) --

          This parameter is no longer supported. Elastic Inference (EI) is no longer available.

          This parameter was used to specify the size of the EI instance to use for the production variant.

        • CoreDumpConfig (dict) --

          Specifies configuration for a core dump from the model container when the process crashes.

          • DestinationS3Uri (string) --

            The Amazon S3 bucket to send the core dump to.

          • KmsKeyId (string) --

            The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the core dump data at rest using Amazon S3 server-side encryption. The KmsKeyId can be any of the following formats:

            • // KMS Key ID "1234abcd-12ab-34cd-56ef-1234567890ab"

            • // Amazon Resource Name (ARN) of a KMS Key "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"

            • // KMS Key Alias "alias/ExampleAlias"

            • // Amazon Resource Name (ARN) of a KMS Key Alias "arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias"

            If you use a KMS key ID or an alias of your KMS key, the SageMaker execution role must include permissions to call kms:Encrypt. If you don't provide a KMS key ID, SageMaker uses the default KMS key for Amazon S3 for your role's account. SageMaker uses server-side encryption with KMS-managed keys for OutputDataConfig. If you use a bucket policy with an s3:PutObject permission that only allows objects with server-side encryption, set the condition key of s3:x-amz-server-side-encryption to "aws:kms". For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.

            The KMS key policy must grant permission to the IAM role that you specify in your CreateEndpoint and UpdateEndpoint requests. For more information, see Using Key Policies in Amazon Web Services KMS in the Amazon Web Services Key Management Service Developer Guide.

        • ServerlessConfig (dict) --

          The serverless configuration for an endpoint. Specifies a serverless endpoint configuration instead of an instance-based endpoint configuration.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • VolumeSizeInGB (integer) --

          The size, in GB, of the ML storage volume attached to individual inference instance associated with the production variant. Currently only Amazon EBS gp2 storage volumes are supported.

        • ModelDataDownloadTimeoutInSeconds (integer) --

          The timeout value, in seconds, to download and extract the model that you want to host from Amazon S3 to the individual inference instance associated with this production variant.

        • ContainerStartupHealthCheckTimeoutInSeconds (integer) --

          The timeout value, in seconds, for your inference container to pass health check by SageMaker Hosting. For more information about health check, see How Your Container Should Respond to Health Check (Ping) Requests.

        • EnableSSMAccess (boolean) --

          You can use this parameter to turn on native Amazon Web Services Systems Manager (SSM) access for a production variant behind an endpoint. By default, SSM access is disabled for all production variants behind an endpoint. You can turn on or turn off SSM access for a production variant behind an existing endpoint by creating a new endpoint configuration and calling UpdateEndpoint.

        • ManagedInstanceScaling (dict) --

          Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

          • Status (string) --

            Indicates whether managed instance scaling is enabled.

          • MinInstanceCount (integer) --

            The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

          • MaxInstanceCount (integer) --

            The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

          • ScaleInPolicy (dict) --

            Configures the scale-in behavior for managed instance scaling.

            • Strategy (string) --

              The strategy for scaling in instances.

              IDLE_RELEASE

              Releases instances that have no hosted inference component copies.

              CONSOLIDATION

              Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

            • MaximumStepSize (integer) --

              The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

              Default value: 1.

            • CooldownInMinutes (integer) --

              The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

              Default value: 20.

        • RoutingConfig (dict) --

          Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

          • RoutingStrategy (string) --

            Sets how the endpoint routes incoming traffic:

            • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

            • RANDOM: The endpoint routes each request to a randomly chosen instance.

            • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

            • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

          • PrefixAwareRoutingConfig (dict) --

            The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

            • PrefixLength (integer) --

              The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

              • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

              • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

              The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

            • ConcurrencyThreshold (integer) --

              The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

              On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

              On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

              The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

        • InferenceAmiVersion (string) --

          Specifies an option from a collection of preconfigured Amazon Machine Image (AMI) images. Each image is configured by Amazon Web Services with a set of software and driver versions. Amazon Web Services optimizes these configurations for different machine learning workloads.

          By selecting an AMI version, you can ensure that your inference environment is compatible with specific software requirements, such as CUDA driver versions, Linux kernel versions, or Amazon Web Services Neuron driver versions.

          The AMI version names, and their configurations, are the following:

          al2-ami-sagemaker-inference-gpu-2

          • Accelerator: GPU

          • NVIDIA driver version: 535

          • CUDA version: 12.2

            al2-ami-sagemaker-inference-gpu-2-1

          • Accelerator: GPU

          • NVIDIA driver version: 535

          • CUDA version: 12.2

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2-ami-sagemaker-inference-gpu-3-1

          • Accelerator: GPU

          • NVIDIA driver version: 550

          • CUDA version: 12.4

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2023-ami-sagemaker-inference-gpu-4-1

          • Accelerator: GPU

          • NVIDIA driver version: 580

          • CUDA version: 13.0

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2-ami-sagemaker-inference-neuron-2

          • Accelerator: Inferentia2 and Trainium

          • Neuron driver version: 2.19

        • CapacityReservationConfig (dict) --

          Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

          • CapacityReservationPreference (string) --

            Options that you can choose for the capacity reservation. SageMaker AI supports the following options:

            capacity-reservations-only

            SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

          • MlReservationArn (string) --

            The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

        • AcceleratorPartitionLayouts (list) --

          The accelerator partition layout for the accelerators backing this production variant. A layout lists the partition profiles to carve on the accelerators it applies to; on NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles such as mig-3g.40gb. Specify a single layout: SageMaker applies it uniformly to every accelerator on the variant's instance type, and you place one inference component on each partition. Omit this field to use whole, unpartitioned accelerators.

          • (dict) --

            A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

            • AcceleratorPartitionTypes (list) --

              The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

              • (string) --

    • DataCaptureConfig (dict) --

      Configuration to control how SageMaker AI captures inference data.

      • EnableCapture (boolean) --

        Whether data capture should be enabled or disabled (defaults to enabled).

      • InitialSamplingPercentage (integer) --

        The percentage of requests SageMaker AI will capture. A lower value is recommended for Endpoints with high traffic.

      • DestinationS3Uri (string) --

        The Amazon S3 location used to capture the data.

      • KmsKeyId (string) --

        The Amazon Resource Name (ARN) of an Key Management Service key that SageMaker AI uses to encrypt the captured data at rest using Amazon S3 server-side encryption.

        The KmsKeyId can be any of the following formats:

        • Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab

        • Key ARN: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab

        • Alias name: alias/ExampleAlias

        • Alias name ARN: arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias

      • CaptureOptions (list) --

        Specifies data Model Monitor will capture. You can configure whether to collect only input, only output, or both

        • (dict) --

          Specifies data Model Monitor will capture.

          • CaptureMode (string) --

            Specify the boundary of data to capture.

      • CaptureContentTypeHeader (dict) --

        Configuration specifying how to treat different headers. If no headers are specified SageMaker AI will by default base64 encode when capturing the data.

        • CsvContentTypes (list) --

          The list of all content type headers that Amazon SageMaker AI will treat as CSV and capture accordingly.

          • (string) --

        • JsonContentTypes (list) --

          The list of all content type headers that SageMaker AI will treat as JSON and capture accordingly.

          • (string) --

    • KmsKeyId (string) --

      Amazon Web Services KMS key ID Amazon SageMaker uses to encrypt data when storing it on the ML storage volume attached to the instance.

    • CreationTime (datetime) --

      A timestamp that shows when the endpoint configuration was created.

    • AsyncInferenceConfig (dict) --

      Returns the description of an endpoint configuration created using the CreateEndpointConfig API.

      • ClientConfig (dict) --

        Configures the behavior of the client used by SageMaker to interact with the model container during asynchronous inference.

        • MaxConcurrentInvocationsPerInstance (integer) --

          The maximum number of concurrent requests sent by the SageMaker client to the model container. If no value is provided, SageMaker chooses an optimal value.

      • OutputConfig (dict) --

        Specifies the configuration for asynchronous inference invocation outputs.

        • KmsKeyId (string) --

          The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the asynchronous inference output in Amazon S3.

        • S3OutputPath (string) --

          The Amazon S3 location to upload inference responses to.

        • NotificationConfig (dict) --

          Specifies the configuration for notifications of inference results for asynchronous inference.

          • SuccessTopic (string) --

            Amazon SNS topic to post a notification to when inference completes successfully. If no topic is provided, no notification is sent on success.

          • ErrorTopic (string) --

            Amazon SNS topic to post a notification to when inference fails. If no topic is provided, no notification is sent on failure.

          • IncludeInferenceResponseIn (list) --

            The Amazon SNS topics where you want the inference response to be included.

            • (string) --

        • S3FailurePath (string) --

          The Amazon S3 location to upload failure inference responses to.

    • ExplainerConfig (dict) --

      The configuration parameters for an explainer.

      • ClarifyExplainerConfig (dict) --

        A member of ExplainerConfig that contains configuration parameters for the SageMaker Clarify explainer.

        • EnableExplanations (string) --

          A JMESPath boolean expression used to filter which records to explain. Explanations are activated by default. See `EnableExplanations <https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-online-explainability-create-endpoint.html#clarify-online-explainability-create-endpoint-enable>`__for additional information.

        • InferenceConfig (dict) --

          The inference configuration parameter for the model container.

          • FeaturesAttribute (string) --

            Provides the JMESPath expression to extract the features from a model container input in JSON Lines format. For example, if FeaturesAttribute is the JMESPath expression 'myfeatures', it extracts a list of features [1,2,3] from request data '{"myfeatures":[1,2,3]}'.

          • ContentTemplate (string) --

            A template string used to format a JSON record into an acceptable model container input. For example, a ContentTemplate string '{"myfeatures":$features}' will format a list of features [1,2,3] into the record string '{"myfeatures":[1,2,3]}'. Required only when the model container input is in JSON Lines format.

          • MaxRecordCount (integer) --

            The maximum number of records in a request that the model container can process when querying the model container for the predictions of a synthetic dataset. A record is a unit of input data that inference can be made on, for example, a single line in CSV data. If MaxRecordCount is 1, the model container expects one record per request. A value of 2 or greater means that the model expects batch requests, which can reduce overhead and speed up the inferencing process. If this parameter is not provided, the explainer will tune the record count per request according to the model container's capacity at runtime.

          • MaxPayloadInMB (integer) --

            The maximum payload size (MB) allowed of a request from the explainer to the model container. Defaults to 6 MB.

          • ProbabilityIndex (integer) --

            A zero-based index used to extract a probability value (score) or list from model container output in CSV format. If this value is not provided, the entire model container output will be treated as a probability value (score) or list.

            Example for a single class model: If the model container output consists of a string-formatted prediction label followed by its probability: '1,0.6', set ProbabilityIndex to 1 to select the probability value 0.6.

            Example for a multiclass model: If the model container output consists of a string-formatted prediction label followed by its probability: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set ProbabilityIndex to 1 to select the probability values [0.1,0.6,0.3].

          • LabelIndex (integer) --

            A zero-based index used to extract a label header or list of label headers from model container output in CSV format.

            Example for a multiclass model: If the model container output consists of label headers followed by probabilities: '"[\'cat\',\'dog\',\'fish\']","[0.1,0.6,0.3]"', set LabelIndex to 0 to select the label headers ['cat','dog','fish'].

          • ProbabilityAttribute (string) --

            A JMESPath expression used to extract the probability (or score) from the model container output if the model container is in JSON Lines format.

            Example: If the model container output of a single request is '{"predicted_label":1,"probability":0.6}', then set ProbabilityAttribute to 'probability'.

          • LabelAttribute (string) --

            A JMESPath expression used to locate the list of label headers in the model container output.

            Example: If the model container output of a batch request is '{"labels":["cat","dog","fish"],"probability":[0.6,0.3,0.1]}', then set LabelAttribute to 'labels' to extract the list of label headers ["cat","dog","fish"]

          • LabelHeaders (list) --

            For multiclass classification problems, the label headers are the names of the classes. Otherwise, the label header is the name of the predicted label. These are used to help readability for the output of the InvokeEndpoint API. See the response section under Invoke the endpoint in the Developer Guide for more information. If there are no label headers in the model container output, provide them manually using this parameter.

            • (string) --

          • FeatureHeaders (list) --

            The names of the features. If provided, these are included in the endpoint response payload to help readability of the InvokeEndpoint output. See the Response section under Invoke the endpoint in the Developer Guide for more information.

            • (string) --

          • FeatureTypes (list) --

            A list of data types of the features (optional). Applicable only to NLP explainability. If provided, FeatureTypes must have at least one 'text' string (for example, ['text']). If FeatureTypes is not provided, the explainer infers the feature types based on the baseline data. The feature types are included in the endpoint response payload. For additional information see the response section under Invoke the endpoint in the Developer Guide for more information.

            • (string) --

        • ShapConfig (dict) --

          The configuration for SHAP analysis.

          • ShapBaselineConfig (dict) --

            The configuration for the SHAP baseline of the Kernal SHAP algorithm.

            • MimeType (string) --

              The MIME type of the baseline data. Choose from 'text/csv' or 'application/jsonlines'. Defaults to 'text/csv'.

            • ShapBaseline (string) --

              The inline SHAP baseline data in string format. ShapBaseline can have one or multiple records to be used as the baseline dataset. The format of the SHAP baseline file should be the same format as the training dataset. For example, if the training dataset is in CSV format and each record contains four features, and all features are numerical, then the format of the baseline data should also share these characteristics. For natural language processing (NLP) of text columns, the baseline value should be the value used to replace the unit of text specified by the Granularity of the TextConfig parameter. The size limit for ShapBasline is 4 KB. Use the ShapBaselineUri parameter if you want to provide more than 4 KB of baseline data.

            • ShapBaselineUri (string) --

              The uniform resource identifier (URI) of the S3 bucket where the SHAP baseline file is stored. The format of the SHAP baseline file should be the same format as the format of the training dataset. For example, if the training dataset is in CSV format, and each record in the training dataset has four features, and all features are numerical, then the baseline file should also have this same format. Each record should contain only the features. If you are using a virtual private cloud (VPC), the ShapBaselineUri should be accessible to the VPC. For more information about setting up endpoints with Amazon Virtual Private Cloud, see Give SageMaker access to Resources in your Amazon Virtual Private Cloud.

          • NumberOfSamples (integer) --

            The number of samples to be used for analysis by the Kernal SHAP algorithm.

          • UseLogit (boolean) --

            A Boolean toggle to indicate if you want to use the logit function (true) or log-odds units (false) for model predictions. Defaults to false.

          • Seed (integer) --

            The starting value used to initialize the random number generator in the explainer. Provide a value for this parameter to obtain a deterministic SHAP result.

          • TextConfig (dict) --

            A parameter that indicates if text features are treated as text and explanations are provided for individual units of text. Required for natural language processing (NLP) explainability only.

            • Language (string) --

              Specifies the language of the text features in ISO 639-1 or ISO 639-3 code of a supported language.

            • Granularity (string) --

              The unit of granularity for the analysis of text features. For example, if the unit is 'token', then each token (like a word in English) of the text is treated as a feature. SHAP values are computed for each unit/feature.

    • ShadowProductionVariants (list) --

      An array of ProductionVariant objects, one for each model that you want to host at this endpoint in shadow mode with production traffic replicated from the model specified on ProductionVariants.

      • (dict) --

        Identifies a model that you want to host and the resources chosen to deploy for hosting it. If you are deploying multiple models, tell SageMaker how to distribute traffic among the models by specifying variant weights. For more information on production variants, check Production variants.

        • VariantName (string) --

          The name of the production variant.

        • ModelName (string) --

          The name of the model that you want to host. This is the name that you specified when creating the model.

        • InitialInstanceCount (integer) --

          Number of instances to launch initially.

        • InstanceType (string) --

          The ML compute instance type.

        • InstancePools (list) --

          A list of instance pools for the production variant. Each instance pool specifies an instance type and its priority for provisioning. Use instance pools to configure heterogeneous endpoints that deploy models across multiple instance types.

          • (dict) --

            Specifies an instance type and its priority for a heterogeneous endpoint. Use instance pools to configure a production variant with multiple instance types, enabling the endpoint to provision instances across different types based on priority.

            • InstanceType (string) --

              The ML compute instance type for the instance pool.

            • ModelNameOverride (string) --

              The name of a SageMaker model to use for this instance pool instead of the model specified for the production variant. Use this to deploy a different model optimized for the instance type in this pool.

            • Priority (integer) --

              The priority for the instance pool. SageMaker attempts to provision instances in order of priority, starting with the lowest value. If instances for a higher-priority pool are unavailable, SageMaker attempts to provision from the next pool.

              Valid values: 1 to 5, where 1 is the highest priority.

        • VariantInstanceProvisionTimeoutInSeconds (integer) --

          The timeout value, in seconds, for provisioning instances for the production variant. When SageMaker encounters an insufficient capacity error while provisioning instances, it retries with the next instance pool (if configured) or waits until the timeout expires. This timeout applies only to capacity provisioning and does not include the time for model download or container startup.

          Valid values: 300 to 3600.

        • InitialVariantWeight (float) --

          Determines initial traffic distribution among all of the models that you specify in the endpoint configuration. The traffic to a production variant is determined by the ratio of the VariantWeight to the sum of all VariantWeight values across all ProductionVariants. If unspecified, it defaults to 1.0.

        • AcceleratorType (string) --

          This parameter is no longer supported. Elastic Inference (EI) is no longer available.

          This parameter was used to specify the size of the EI instance to use for the production variant.

        • CoreDumpConfig (dict) --

          Specifies configuration for a core dump from the model container when the process crashes.

          • DestinationS3Uri (string) --

            The Amazon S3 bucket to send the core dump to.

          • KmsKeyId (string) --

            The Amazon Web Services Key Management Service (Amazon Web Services KMS) key that SageMaker uses to encrypt the core dump data at rest using Amazon S3 server-side encryption. The KmsKeyId can be any of the following formats:

            • // KMS Key ID "1234abcd-12ab-34cd-56ef-1234567890ab"

            • // Amazon Resource Name (ARN) of a KMS Key "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"

            • // KMS Key Alias "alias/ExampleAlias"

            • // Amazon Resource Name (ARN) of a KMS Key Alias "arn:aws:kms:us-west-2:111122223333:alias/ExampleAlias"

            If you use a KMS key ID or an alias of your KMS key, the SageMaker execution role must include permissions to call kms:Encrypt. If you don't provide a KMS key ID, SageMaker uses the default KMS key for Amazon S3 for your role's account. SageMaker uses server-side encryption with KMS-managed keys for OutputDataConfig. If you use a bucket policy with an s3:PutObject permission that only allows objects with server-side encryption, set the condition key of s3:x-amz-server-side-encryption to "aws:kms". For more information, see KMS-Managed Encryption Keys in the Amazon Simple Storage Service Developer Guide.

            The KMS key policy must grant permission to the IAM role that you specify in your CreateEndpoint and UpdateEndpoint requests. For more information, see Using Key Policies in Amazon Web Services KMS in the Amazon Web Services Key Management Service Developer Guide.

        • ServerlessConfig (dict) --

          The serverless configuration for an endpoint. Specifies a serverless endpoint configuration instead of an instance-based endpoint configuration.

          • MemorySizeInMB (integer) --

            The memory size of your serverless endpoint. Valid values are in 1 GB increments: 1024 MB, 2048 MB, 3072 MB, 4096 MB, 5120 MB, or 6144 MB.

          • MaxConcurrency (integer) --

            The maximum number of concurrent invocations your serverless endpoint can process.

          • ProvisionedConcurrency (integer) --

            The amount of provisioned concurrency to allocate for the serverless endpoint. Should be less than or equal to MaxConcurrency.

        • VolumeSizeInGB (integer) --

          The size, in GB, of the ML storage volume attached to individual inference instance associated with the production variant. Currently only Amazon EBS gp2 storage volumes are supported.

        • ModelDataDownloadTimeoutInSeconds (integer) --

          The timeout value, in seconds, to download and extract the model that you want to host from Amazon S3 to the individual inference instance associated with this production variant.

        • ContainerStartupHealthCheckTimeoutInSeconds (integer) --

          The timeout value, in seconds, for your inference container to pass health check by SageMaker Hosting. For more information about health check, see How Your Container Should Respond to Health Check (Ping) Requests.

        • EnableSSMAccess (boolean) --

          You can use this parameter to turn on native Amazon Web Services Systems Manager (SSM) access for a production variant behind an endpoint. By default, SSM access is disabled for all production variants behind an endpoint. You can turn on or turn off SSM access for a production variant behind an existing endpoint by creating a new endpoint configuration and calling UpdateEndpoint.

        • ManagedInstanceScaling (dict) --

          Settings that control the range in the number of instances that the endpoint provisions as it scales up or down to accommodate traffic.

          • Status (string) --

            Indicates whether managed instance scaling is enabled.

          • MinInstanceCount (integer) --

            The minimum number of instances that the endpoint must retain when it scales down to accommodate a decrease in traffic.

          • MaxInstanceCount (integer) --

            The maximum number of instances that the endpoint can provision when it scales up to accommodate an increase in traffic.

          • ScaleInPolicy (dict) --

            Configures the scale-in behavior for managed instance scaling.

            • Strategy (string) --

              The strategy for scaling in instances.

              IDLE_RELEASE

              Releases instances that have no hosted inference component copies.

              CONSOLIDATION

              Consolidates inference component copies onto fewer instances to release more instances. Consolidation honors the scheduling configuration of each inference component. For example, if an inference component specifies Availability Zone balance, consolidation only proceeds when the resulting distribution does not increase the imbalance.

            • MaximumStepSize (integer) --

              The maximum number of instances that the endpoint can terminate at a time during a consolidation scale-in operation.

              Default value: 1.

            • CooldownInMinutes (integer) --

              The cooldown period, in minutes, after the last endpoint operation before the endpoint evaluates consolidation scale-in opportunities.

              Default value: 20.

        • RoutingConfig (dict) --

          Settings that control how the endpoint routes incoming traffic to the instances that the endpoint hosts.

          • RoutingStrategy (string) --

            Sets how the endpoint routes incoming traffic:

            • LEAST_OUTSTANDING_REQUESTS: The endpoint routes requests to the specific instances that have more capacity to process them.

            • RANDOM: The endpoint routes each request to a randomly chosen instance.

            • PREFIX_AWARE: The endpoint routes requests that share the same prompt prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. When the number of in-flight requests on the selected instance or copy reaches the ConcurrencyThreshold value, the endpoint routes the request to another copy on that instance or to another instance. The endpoint applies prefix-aware routing only when two or more instances can serve the request.

            • KV_AWARE: The endpoint routes each request to an instance that already holds the start of the request's prompt in its key-value (KV) cache, and weighs the length of that match against how busy each instance is. On an endpoint with inference components, the endpoint chooses among the copies of the invoked inference component in the same way. When the endpoint can't find an instance or copy that holds the start of the prompt and isn't too busy, it chooses an instance as it does with LEAST_OUTSTANDING_REQUESTS. KV_AWARE works only with supported containers, such as the SageMaker AI vLLM Deep Learning Container.

          • PrefixAwareRoutingConfig (dict) --

            The configuration for prefix-aware routing. Specify this parameter only when you set RoutingStrategy to PREFIX_AWARE.

            • PrefixLength (integer) --

              The maximum length of the prefix used for routing decisions. Required when RoutingStrategy is PREFIX_AWARE.

              • For the SageMaker Runtime InvokeEndpoint and InvokeEndpointWithResponseStream APIs, this value specifies the number of bytes from the beginning of the request body.

              • For OpenAI-compatible API, this value specifies the number of characters from the text content of the messages array.

              The endpoint routes requests that share the same prefix to the same instance and, when that instance hosts more than one copy of the invoked inference component, to the same copy. Set this value to cover shared content (such as system prompts) plus enough unique content to distribute workloads across instances and copies.

            • ConcurrencyThreshold (integer) --

              The number of in-flight requests on the selected instance or inference component copy at which the endpoint routes a request to another copy or instance. Required when RoutingStrategy is PREFIX_AWARE.

              On an endpoint without inference components, the endpoint compares this threshold with the number of in-flight requests on the selected instance. When that number reaches the threshold, the endpoint routes the request to another instance.

              On an endpoint with inference components, the endpoint compares this threshold with the number of in-flight requests on the selected copy of the invoked inference component. Requests to other inference components on the same instance don't count. When the instance hosts more than one copy of the inference component, the endpoint routes requests that share a prefix to the same copy and, while that copy has reached the threshold, to another copy on the instance that is under the threshold. When every copy on the instance has reached the threshold, the endpoint routes the request to another instance.

              The threshold applies to every instance and every copy of every inference component in the production variant, so choose a value that suits the instance or copy that can serve the fewest concurrent requests.

        • InferenceAmiVersion (string) --

          Specifies an option from a collection of preconfigured Amazon Machine Image (AMI) images. Each image is configured by Amazon Web Services with a set of software and driver versions. Amazon Web Services optimizes these configurations for different machine learning workloads.

          By selecting an AMI version, you can ensure that your inference environment is compatible with specific software requirements, such as CUDA driver versions, Linux kernel versions, or Amazon Web Services Neuron driver versions.

          The AMI version names, and their configurations, are the following:

          al2-ami-sagemaker-inference-gpu-2

          • Accelerator: GPU

          • NVIDIA driver version: 535

          • CUDA version: 12.2

            al2-ami-sagemaker-inference-gpu-2-1

          • Accelerator: GPU

          • NVIDIA driver version: 535

          • CUDA version: 12.2

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2-ami-sagemaker-inference-gpu-3-1

          • Accelerator: GPU

          • NVIDIA driver version: 550

          • CUDA version: 12.4

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2023-ami-sagemaker-inference-gpu-4-1

          • Accelerator: GPU

          • NVIDIA driver version: 580

          • CUDA version: 13.0

          • NVIDIA Container Toolkit with disabled CUDA-compat mounting

            al2-ami-sagemaker-inference-neuron-2

          • Accelerator: Inferentia2 and Trainium

          • Neuron driver version: 2.19

        • CapacityReservationConfig (dict) --

          Settings for the capacity reservation for the compute instances that SageMaker AI reserves for an endpoint.

          • CapacityReservationPreference (string) --

            Options that you can choose for the capacity reservation. SageMaker AI supports the following options:

            capacity-reservations-only

            SageMaker AI launches instances only into an ML capacity reservation. If no capacity is available, the instances fail to launch.

          • MlReservationArn (string) --

            The Amazon Resource Name (ARN) that uniquely identifies the ML capacity reservation that SageMaker AI applies when it deploys the endpoint.

        • AcceleratorPartitionLayouts (list) --

          The accelerator partition layout for the accelerators backing this production variant. A layout lists the partition profiles to carve on the accelerators it applies to; on NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles such as mig-3g.40gb. Specify a single layout: SageMaker applies it uniformly to every accelerator on the variant's instance type, and you place one inference component on each partition. Omit this field to use whole, unpartitioned accelerators.

          • (dict) --

            A layout of accelerator partitions for a production variant. On NVIDIA GPUs, the partitions are Multi-Instance GPU (MIG) instances. SageMaker applies the layout to every accelerator on the variant's instance type. You can't change the layout of a running endpoint. To use a different layout, create a new endpoint.

            • AcceleratorPartitionTypes (list) --

              The list of partition profiles to carve on each accelerator that this layout applies to. On NVIDIA GPUs these are Multi-Instance GPU (MIG) profiles, for example ["mig-3g.40gb", "mig-4g.40gb"]. The order of the profiles doesn't change how the accelerator is partitioned. However, when you update an endpoint, a layout with the same profiles in a different order counts as a different layout, and the update is rejected. Repeat a profile to request more than one partition of that size: for example, ["mig-3g.40gb", "mig-3g.40gb"] requests two mig-3g.40gb partitions.

              • (string) --

    • ExecutionRoleArn (string) --

      The Amazon Resource Name (ARN) of the IAM role that you assigned to the endpoint configuration.

    • VpcConfig (dict) --

      Specifies an Amazon Virtual Private Cloud (VPC) that your SageMaker jobs, hosted models, and compute resources have access to. You can control access to and from your resources by configuring a VPC. For more information, see Give SageMaker Access to Resources in your Amazon VPC.

      • SecurityGroupIds (list) --

        The VPC security group IDs, in the form sg-xxxxxxxx. Specify the security groups for the VPC that is specified in the Subnets field.

        • (string) --

      • Subnets (list) --

        The ID of the subnets in the VPC to which you want to connect your training job or model. For information about the availability of specific instance types, see Supported Instance Types and Availability Zones.

        • (string) --

    • EnableNetworkIsolation (boolean) --

      Indicates whether all model containers deployed to the endpoint are isolated. If they are, no inbound or outbound network calls can be made to or from the model containers.

    • MetricsConfig (dict) --

      The configuration parameters for utilization metrics.

      • EnableEnhancedMetrics (boolean) --

        Specifies whether to enable enhanced metrics for the endpoint. Enhanced metrics provide utilization and invocation data at instance and container granularity. Container granularity is supported for Inference Components. The default is False.

      • EnableDetailedObservability (boolean) --

        Indicates whether detailed observability is enabled for the endpoint. When set to True, the following metrics are published at the configured frequency:

        • Container-level inference metrics scraped from the container's Prometheus endpoint (such as request latency, error counts, and throughput). Available metrics vary by framework.

        • Per-GPU metrics (utilization, memory, and temperature) attributed to individual inference components.

        • Per-instance host metrics (CPU, memory, and disk utilization).

        • Inference component placement metrics (copy count per Availability Zone).

        For first-party and Deep Learning Containers (DLC), the Prometheus endpoint path is determined automatically. For Bring-Your-Own-Container (BYOC) cases, you can optionally set ContainerMetricsConfig to specify a custom endpoint path. If not specified, the default path /metrics on port 8080 is used.

        When set to False, these additional metrics are not published. Standard invocation and utilization metrics controlled by EnableEnhancedMetrics are unaffected.

        The default value for new endpoint configurations is True. For existing endpoint configurations created before this feature, the value is False unless explicitly set.

      • MetricPublishFrequencyInSeconds (integer) --

        The interval, in seconds, at which metrics are published to Amazon CloudWatch. Defaults to 60. Valid values: 10, 30, 60, 120, 180, 240, 300.

        When EnableEnhancedMetrics is set to False, this interval applies to utilization metrics only. Invocation metrics continue to be published at the default 60-second interval. When EnableEnhancedMetrics is set to True, this interval applies to both utilization and invocation metrics.

        When EnableDetailedObservability is set to True, this interval applies to per-GPU metrics, per-instance host metrics, container metrics, and fleet-level inference component lifecycle and placement metrics.

    • CrossRegionTrafficConfig (dict) --

      The cross-Region traffic cap, if the endpoint configuration sets one.

      • MaxInvocationsPerSecondPerInstance (integer) --

        The number of invocations per second that you expect one InService instance to serve. One value applies to every production variant, whatever its instance type.

      • MaxCrossRegionPercentPerEndpoint (integer) --

        The percentage of the endpoint's capacity that requests from inference routers in other Amazon Web Services Regions can use.

DescribeJob (updated) Link ¶
Changes (request, response)
Request
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}
Response
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'},
 'SecondaryStatus': {'Executing',
                     'Finalizing',
                     'Generating',
                     'PartialGeneration',
                     'Preparing',
                     'Publishing',
                     'Transforming',
                     'Validating'},
 'SecondaryStatusTransitions': {'Status': {'Executing',
                                           'Finalizing',
                                           'Generating',
                                           'PartialGeneration',
                                           'Preparing',
                                           'Publishing',
                                           'Transforming',
                                           'Validating'}}}

Returns detailed information about a job, including its current status, secondary status, configuration, and timestamps. Use SecondaryStatus for granular progress tracking and SecondaryStatusTransitions to see the full history of status changes with timestamps.

The following operations are related to DescribeJob:

  • CreateJob

  • ListJobs

  • StopJob

  • DeleteJob

See also: AWS API Documentation

Request Syntax

client.describe_job(
    JobName='string',
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation'
)
type JobName:

string

param JobName:

[REQUIRED]

The name of the job to describe.

type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of the job.

rtype:

dict

returns:

Response Syntax

{
    'JobName': 'string',
    'JobArn': 'string',
    'RoleArn': 'string',
    'JobCategory': 'AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    'JobConfigSchemaVersion': 'string',
    'JobConfigDocument': 'string',
    'CreationTime': datetime(2015, 1, 1),
    'LastModifiedTime': datetime(2015, 1, 1),
    'EndTime': datetime(2015, 1, 1),
    'JobStatus': 'InProgress'|'Completed'|'Failed'|'Stopping'|'Stopped'|'Deleting'|'DeleteFailed',
    'SecondaryStatus': 'Starting'|'Downloading'|'Training'|'Uploading'|'Stopping'|'Stopped'|'MaxRuntimeExceeded'|'Interrupted'|'Failed'|'Completed'|'Restarting'|'Pending'|'Generating'|'PartialGeneration'|'Evaluating'|'Validating'|'Preparing'|'Executing'|'Publishing'|'Finalizing'|'Transforming'|'Deleting'|'DeleteFailed',
    'SecondaryStatusTransitions': [
        {
            'Status': 'Starting'|'Downloading'|'Training'|'Uploading'|'Stopping'|'Stopped'|'MaxRuntimeExceeded'|'Interrupted'|'Failed'|'Completed'|'Restarting'|'Pending'|'Generating'|'PartialGeneration'|'Evaluating'|'Validating'|'Preparing'|'Executing'|'Publishing'|'Finalizing'|'Transforming'|'Deleting'|'DeleteFailed',
            'StartTime': datetime(2015, 1, 1),
            'EndTime': datetime(2015, 1, 1),
            'StatusMessage': 'string'
        },
    ],
    'FailureReason': 'string',
    'Tags': [
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • JobName (string) --

      The name of the job.

    • JobArn (string) --

      The Amazon Resource Name (ARN) of the job.

    • RoleArn (string) --

      The ARN of the IAM role associated with the job.

    • JobCategory (string) --

      The category of the job.

    • JobConfigSchemaVersion (string) --

      The schema version used for the job configuration document.

    • JobConfigDocument (string) --

      The JSON configuration document for the job.

    • CreationTime (datetime) --

      The date and time that the job was created.

    • LastModifiedTime (datetime) --

      The date and time that the job was last modified.

    • EndTime (datetime) --

      The date and time that the job ended.

    • JobStatus (string) --

      The current status of the job.

    • SecondaryStatus (string) --

      The detailed secondary status of the job, providing more granular information about the job's progress. Secondary statuses may change between releases.

    • SecondaryStatusTransitions (list) --

      A list of secondary status transitions for the job, with timestamps and optional status messages.

      • (dict) --

        Represents a secondary status transition for a job. Jobs progress through multiple secondary statuses during execution. Each transition records the status, start time, optional end time, and an optional message with additional details.

        • Status (string) --

          The secondary status of the job at this transition point.

        • StartTime (datetime) --

          The date and time that the status transition started.

        • EndTime (datetime) --

          The date and time that the status transition ended.

        • StatusMessage (string) --

          A detailed message about the status transition.

    • FailureReason (string) --

      If the job failed, the reason it failed.

    • Tags (list) --

      The tags associated with the job.

      • (dict) --

        A tag object that consists of a key and an optional value, used to manage metadata for SageMaker Amazon Web Services resources.

        You can add tags to notebook instances, training jobs, hyperparameter tuning jobs, batch transform jobs, models, labeling jobs, work teams, endpoint configurations, and endpoints. For more information on adding tags to SageMaker resources, see AddTags.

        For more information on adding metadata to your Amazon Web Services resources with tagging, see Tagging Amazon Web Services resources. For advice on best practices for managing Amazon Web Services resources with tagging, see Tagging Best Practices: Implement an Effective Amazon Web Services Resource Tagging Strategy.

        • Key (string) --

          The tag key. Tag keys must be unique per resource.

        • Value (string) --

          The tag value.

DescribeJobSchemaVersion (updated) Link ¶
Changes (both)
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}

Returns the JSON schema for a specified job category and schema version. Use this schema to validate your JobConfigDocument before calling CreateJob. If you don't specify a schema version, the latest version is returned. The schema defines required fields, allowed values, and constraints for the job configuration.

The following operations are related to DescribeJobSchemaVersion:

  • ListJobSchemaVersions

  • CreateJob

See also: AWS API Documentation

Request Syntax

client.describe_job_schema_version(
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    JobConfigSchemaVersion='string'
)
type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of the job schema to describe.

type JobConfigSchemaVersion:

string

param JobConfigSchemaVersion:

The version of the schema to retrieve. If not specified, the latest version is returned.

rtype:

dict

returns:

Response Syntax

{
    'JobCategory': 'AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    'JobConfigSchemaVersion': 'string',
    'JobConfigSchema': 'string'
}

Response Structure

  • (dict) --

    • JobCategory (string) --

      The category of the job schema.

    • JobConfigSchemaVersion (string) --

      The version of the schema.

    • JobConfigSchema (string) --

      The JSON schema document that defines the structure of the job configuration.

ListJobSchemaVersions (updated) Link ¶
Changes (request)
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}

Lists available configuration schema versions for a specified job category. Use the schema versions with DescribeJobSchemaVersion to retrieve the full schema document.

The following operations are related to ListJobSchemaVersions:

  • DescribeJobSchemaVersion

  • CreateJob

See also: AWS API Documentation

Request Syntax

client.list_job_schema_versions(
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    NextToken='string',
    MaxResults=123
)
type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of job schemas to list.

type NextToken:

string

param NextToken:

If the previous response was truncated, this token retrieves the next set of results.

type MaxResults:

integer

param MaxResults:

The maximum number of schema versions to return in the response. The default value is 5.

rtype:

dict

returns:

Response Syntax

{
    'NextToken': 'string',
    'JobConfigSchemas': [
        {
            'JobConfigSchemaVersion': 'string'
        },
    ]
}

Response Structure

  • (dict) --

    • NextToken (string) --

      If the response is truncated, this token retrieves the next set of results.

    • JobConfigSchemas (list) --

      An array of JobConfigSchemaVersionSummary objects listing the available schema versions.

      • (dict) --

        Provides summary information about a job configuration schema version.

        • JobConfigSchemaVersion (string) --

          The version of the job configuration schema.

ListJobs (updated) Link ¶
Changes (request, response)
Request
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}
Response
{'JobSummaries': {'JobCategory': {'DataQualityEvaluation',
                                  'DataTransformation',
                                  'SyntheticDataGeneration'},
                  'JobSecondaryStatus': {'Executing',
                                         'Finalizing',
                                         'Generating',
                                         'PartialGeneration',
                                         'Preparing',
                                         'Publishing',
                                         'Transforming',
                                         'Validating'}}}

Lists jobs in a specified category. You can filter results by creation time, last modified time, name, and status. Results are sorted by the field you specify in SortBy. Use pagination to retrieve large result sets efficiently.

The following operations are related to ListJobs:

  • CreateJob

  • DescribeJob

See also: AWS API Documentation

Request Syntax

client.list_jobs(
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
    NextToken='string',
    MaxResults=123,
    CreationTimeAfter=datetime(2015, 1, 1),
    CreationTimeBefore=datetime(2015, 1, 1),
    LastModifiedTimeAfter=datetime(2015, 1, 1),
    LastModifiedTimeBefore=datetime(2015, 1, 1),
    NameContains='string',
    SortBy='Name'|'CreationTime'|'Status',
    SortOrder='Ascending'|'Descending',
    StatusEquals='InProgress'|'Completed'|'Failed'|'Stopping'|'Stopped'|'Deleting'|'DeleteFailed'
)
type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of jobs to list.

type NextToken:

string

param NextToken:

If the previous response was truncated, this token retrieves the next set of results.

type MaxResults:

integer

param MaxResults:

The maximum number of jobs to return in the response. The default value is 50.

type CreationTimeAfter:

datetime

param CreationTimeAfter:

A filter that returns only jobs created after the specified time.

type CreationTimeBefore:

datetime

param CreationTimeBefore:

A filter that returns only jobs created before the specified time.

type LastModifiedTimeAfter:

datetime

param LastModifiedTimeAfter:

A filter that returns only jobs modified after the specified time.

type LastModifiedTimeBefore:

datetime

param LastModifiedTimeBefore:

A filter that returns only jobs modified before the specified time.

type NameContains:

string

param NameContains:

A string in the job name to filter results. Only jobs whose name contains the specified string are returned.

type SortBy:

string

param SortBy:

The field to sort results by.

type SortOrder:

string

param SortOrder:

The sort order for results. Valid values are Ascending and Descending.

type StatusEquals:

string

param StatusEquals:

A filter that returns only jobs with the specified status.

rtype:

dict

returns:

Response Syntax

{
    'NextToken': 'string',
    'JobSummaries': [
        {
            'JobArn': 'string',
            'JobName': 'string',
            'JobCategory': 'AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation',
            'JobStatus': 'InProgress'|'Completed'|'Failed'|'Stopping'|'Stopped'|'Deleting'|'DeleteFailed',
            'JobSecondaryStatus': 'Starting'|'Downloading'|'Training'|'Uploading'|'Stopping'|'Stopped'|'MaxRuntimeExceeded'|'Interrupted'|'Failed'|'Completed'|'Restarting'|'Pending'|'Generating'|'PartialGeneration'|'Evaluating'|'Validating'|'Preparing'|'Executing'|'Publishing'|'Finalizing'|'Transforming'|'Deleting'|'DeleteFailed',
            'CreationTime': datetime(2015, 1, 1),
            'LastModifiedTime': datetime(2015, 1, 1),
            'EndTime': datetime(2015, 1, 1)
        },
    ]
}

Response Structure

  • (dict) --

    • NextToken (string) --

      If the response is truncated, this token retrieves the next set of results.

    • JobSummaries (list) --

      An array of JobSummary objects that provide summary information about the jobs.

      • (dict) --

        Provides summary information about a job, returned by the ListJobs operation. Use DescribeJob to get full details for a specific job.

        • JobArn (string) --

          The Amazon Resource Name (ARN) of the job.

        • JobName (string) --

          The name of the job.

        • JobCategory (string) --

          The category of the job.

        • JobStatus (string) --

          The current status of the job.

        • JobSecondaryStatus (string) --

          The secondary status of the job, providing more granular information about the job's progress. Secondary statuses may change between releases.

        • CreationTime (datetime) --

          The date and time that the job was created.

        • LastModifiedTime (datetime) --

          The date and time that the job was last modified.

        • EndTime (datetime) --

          The date and time that the job ended.

StopJob (updated) Link ¶
Changes (request)
{'JobCategory': {'DataQualityEvaluation',
                 'DataTransformation',
                 'SyntheticDataGeneration'}}

Stops a running job. When you call StopJob, Amazon SageMaker sets the job status to Stopping. After the job stops, the status changes to Stopped. Partial results may be available in the output location if the job was in progress. To delete a stopped job, call DeleteJob.

The following operations are related to StopJob:

  • CreateJob

  • DescribeJob

  • DeleteJob

See also: AWS API Documentation

Request Syntax

client.stop_job(
    JobName='string',
    JobCategory='AgentRFT'|'SyntheticDataGeneration'|'DataQualityEvaluation'|'AgentRFTEvaluation'|'DataTransformation'
)
type JobName:

string

param JobName:

[REQUIRED]

The name of the job to stop.

type JobCategory:

string

param JobCategory:

[REQUIRED]

The category of the job to stop.

rtype:

dict

returns:

Response Syntax

{}

Response Structure

  • (dict) --