2026/08/17 - Amazon Bedrock AgentCore Control - 7 updated api methods
Changes Adds implementations of third-party evaluators, both managed-as-a-service and as templates within custom evaluators.
{'schemaType': {'GENERIC_EVALUATION_PREDEFINED_V1'}}
Creates a new dataset resource asynchronously. Returns immediately with status CREATING. Poll GetDataset until status transitions to ACTIVE or CREATE_FAILED.
See also: AWS API Documentation
Request Syntax
client.create_dataset(
clientToken='string',
datasetName='string',
description='string',
source={
'inlineExamples': {
'examples': [
{...}|[...]|123|123.4|'string'|True|None,
]
},
's3Source': {
's3Uri': 'string'
}
},
schemaType='AGENTCORE_EVALUATION_PREDEFINED_V1'|'AGENTCORE_EVALUATION_SIMULATED_V1'|'GENERIC_EVALUATION_PREDEFINED_V1',
kmsKeyArn='string',
tags={
'string': 'string'
}
)
string
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
This field is autopopulated if not provided.
string
[REQUIRED]
Human-readable name for the dataset. Must be unique within the account. Immutable after creation.
string
A description of the dataset.
dict
[REQUIRED]
Source of initial examples. Provide either inline examples or an S3 URI pointing to a JSONL file.
inlineExamples (dict) --
Inline examples provided directly in the request body.
examples (list) -- [REQUIRED]
Examples to add. Each example is assigned an auto-generated UUID.
(:ref:`document<document>`) --
s3Source (dict) --
Amazon S3 URI pointing to a JSONL file in the customer's bucket.
s3Uri (string) -- [REQUIRED]
Amazon S3 URI of the JSONL file (for example, s3://my-bucket/path/to/examples.jsonl).
string
[REQUIRED]
Versioned schema type governing the structure of examples. Immutable after creation.
string
Optional KMS key ARN for server-side encryption on service Amazon S3 writes.
dict
A map of tag keys and values to assign to the dataset.
(string) --
(string) --
dict
Response Syntax
{
'datasetArn': 'string',
'datasetId': 'string',
'status': 'CREATING'|'UPDATING'|'DELETING'|'ACTIVE'|'CREATE_FAILED'|'UPDATE_FAILED'|'DELETE_FAILED',
'createdAt': datetime(2015, 1, 1)
}
Response Structure
(dict) --
datasetArn (string) --
The Amazon Resource Name (ARN) of the created dataset.
datasetId (string) --
The unique identifier of the created dataset.
status (string) --
Always CREATING immediately after this call. Poll GetDataset until status transitions to ACTIVE or CREATE_FAILED.
createdAt (datetime) --
The timestamp when the dataset was created.
{'evaluatorConfig': {'derived': {'baseEvaluatorId': 'string',
'modelConfig': {'bedrockEvaluatorModelConfig': {'additionalModelRequestFields': {},
'inferenceConfig': {'maxTokens': 'integer',
'stopSequences': ['string'],
'temperature': 'float',
'topP': 'float'},
'modelId': 'string'},
'responsesEvaluatorModelConfig': {'maxOutputTokens': 'integer',
'modelId': 'string',
'reasoning': {'effort': 'string'},
'temperature': 'float',
'topP': 'float'}}}}}
Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.
See also: AWS API Documentation
Request Syntax
client.create_evaluator(
clientToken='string',
evaluatorName='string',
description='string',
evaluatorConfig={
'llmAsAJudge': {
'instructions': 'string',
'ratingScale': {
'numerical': [
{
'definition': 'string',
'value': 123.0,
'label': 'string'
},
],
'categorical': [
{
'definition': 'string',
'label': 'string'
},
]
},
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
},
'codeBased': {
'lambdaConfig': {
'lambdaArn': 'string',
'lambdaTimeoutInSeconds': 123
}
},
'derived': {
'baseEvaluatorId': 'string',
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
}
},
level='TOOL_CALL'|'TRACE'|'SESSION',
kmsKeyArn='string',
tags={
'string': 'string'
}
)
string
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
This field is autopopulated if not provided.
string
[REQUIRED]
The name of the evaluator. Must be unique within your account.
string
The description of the evaluator that explains its purpose and evaluation criteria.
dict
[REQUIRED]
The configuration for the evaluator. Specify either LLM-as-a-Judge settings with instructions, rating scale, and model configuration, or code-based settings with a customer-managed Lambda function.
llmAsAJudge (dict) --
The LLM-as-a-Judge configuration that uses a language model to evaluate agent performance based on custom instructions and rating scales.
instructions (string) -- [REQUIRED]
The evaluation instructions that guide the language model in assessing agent performance, including criteria and evaluation guidelines.
ratingScale (dict) -- [REQUIRED]
The rating scale that defines how the evaluator should score agent performance, either numerical or categorical.
numerical (list) --
The numerical rating scale with defined score values and descriptions for quantitative evaluation.
(dict) --
The definition of a numerical rating scale option that provides a numeric value with its description for evaluation scoring.
definition (string) -- [REQUIRED]
The description that explains what this numerical rating represents and when it should be used.
value (float) -- [REQUIRED]
The numerical value for this rating scale option.
label (string) -- [REQUIRED]
The label or name that describes this numerical rating option.
categorical (list) --
The categorical rating scale with named categories and definitions for qualitative evaluation.
(dict) --
The definition of a categorical rating scale option that provides a named category with its description for evaluation scoring.
definition (string) -- [REQUIRED]
The description that explains what this categorical rating represents and when it should be used.
label (string) -- [REQUIRED]
The label or name of this categorical rating option.
modelConfig (dict) -- [REQUIRED]
The model configuration that specifies which foundation model to use and how to configure it for evaluation.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
codeBased (dict) --
Configuration for a code-based evaluator that uses a customer-managed Lambda function to programmatically assess agent performance.
lambdaConfig (dict) --
The Lambda function configuration for code-based evaluation.
lambdaArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the Lambda function that implements the evaluation logic.
lambdaTimeoutInSeconds (integer) --
The timeout in seconds for the Lambda function invocation. Defaults to 60. Must be between 1 and 300.
derived (dict) --
The configuration for an evaluator derived from an existing base evaluator (a built-in or third-party evaluator), run on your own model. The base evaluator supplies the prompt and scoring.
baseEvaluatorId (string) -- [REQUIRED]
The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
modelConfig (dict) -- [REQUIRED]
The configuration of the evaluator model that you supply.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
string
[REQUIRED]
The evaluation level that determines the scope of evaluation. Valid values are TOOL_CALL for individual tool invocations, TRACE for single request-response interactions, or SESSION for entire conversation sessions.
string
The Amazon Resource Name (ARN) of a customer managed KMS key to use for encrypting sensitive evaluator data, including instructions and rating scale. If you don't specify a KMS key, the evaluator data is encrypted with an Amazon Web Services owned key. Only symmetric encryption KMS keys are supported. For more information, see Encryption at rest for AgentCore Evaluations.
dict
A map of tag keys and values to assign to an AgentCore Evaluator. Tags enable you to categorize your resources in different ways, for example, by purpose, owner, or environment.
(string) --
(string) --
dict
Response Syntax
{
'evaluatorArn': 'string',
'evaluatorId': 'string',
'createdAt': datetime(2015, 1, 1),
'status': 'ACTIVE'|'CREATING'|'CREATE_FAILED'|'UPDATING'|'UPDATE_FAILED'|'DELETING'
}
Response Structure
(dict) --
evaluatorArn (string) --
The Amazon Resource Name (ARN) of the created evaluator.
evaluatorId (string) --
The unique identifier of the created evaluator.
createdAt (datetime) --
The timestamp when the evaluator was created.
status (string) --
The status of the evaluator creation operation.
{'schemaType': {'GENERIC_EVALUATION_PREDEFINED_V1'}}
Retrieves dataset metadata. Use the datasetVersion query parameter to retrieve a specific version's metadata. If absent, defaults to DRAFT. For paginated example content, use ListDatasetExamples.
See also: AWS API Documentation
Request Syntax
client.get_dataset(
datasetId='string',
datasetVersion='string'
)
string
[REQUIRED]
The unique identifier of the dataset to retrieve.
string
Version to retrieve: "DRAFT" or a version number. Defaults to DRAFT if absent.
dict
Response Syntax
{
'datasetArn': 'string',
'datasetId': 'string',
'datasetVersion': 'string',
'datasetName': 'string',
'description': 'string',
'status': 'CREATING'|'UPDATING'|'DELETING'|'ACTIVE'|'CREATE_FAILED'|'UPDATE_FAILED'|'DELETE_FAILED',
'draftStatus': 'MODIFIED'|'UNMODIFIED',
'failureReason': 'string',
'schemaType': 'AGENTCORE_EVALUATION_PREDEFINED_V1'|'AGENTCORE_EVALUATION_SIMULATED_V1'|'GENERIC_EVALUATION_PREDEFINED_V1',
'kmsKeyArn': 'string',
'exampleCount': 123,
'downloadUrl': 'string',
'downloadUrlExpiresAt': datetime(2015, 1, 1),
'createdAt': datetime(2015, 1, 1),
'updatedAt': datetime(2015, 1, 1),
'tags': {
'string': 'string'
}
}
Response Structure
(dict) --
datasetArn (string) --
The Amazon Resource Name (ARN) of the dataset.
datasetId (string) --
The unique identifier of the dataset.
datasetVersion (string) --
The resolved version: "DRAFT" (default) or the requested version number.
datasetName (string) --
The name of the dataset.
description (string) --
The description of the dataset.
status (string) --
The current status of the dataset.
draftStatus (string) --
Publish synchronization state. Only authoritative when status is ACTIVE. MODIFIED indicates DRAFT has unpublished changes. UNMODIFIED indicates DRAFT matches the latest published version.
failureReason (string) --
Populated when status is CREATE_FAILED, UPDATE_FAILED, or DELETE_FAILED. Describes the reason for the failure.
schemaType (string) --
The schema type declared at create time. Immutable after creation.
kmsKeyArn (string) --
KMS key ARN used for server-side encryption on service Amazon S3 writes, if configured.
exampleCount (integer) --
The number of examples in the DRAFT.
downloadUrl (string) --
Presigned Amazon S3 URL to download the consolidated dataset file for the resolved version. Expires after 5 minutes. Omitted if the file does not yet exist.
downloadUrlExpiresAt (datetime) --
Expiry timestamp for the download URL.
createdAt (datetime) --
The timestamp when the dataset was created.
updatedAt (datetime) --
The timestamp when the dataset was last updated.
tags (dict) --
The tags associated with the dataset.
(string) --
(string) --
{'evaluatorConfig': {'derived': {'baseEvaluatorId': 'string',
'modelConfig': {'bedrockEvaluatorModelConfig': {'additionalModelRequestFields': {},
'inferenceConfig': {'maxTokens': 'integer',
'stopSequences': ['string'],
'temperature': 'float',
'topP': 'float'},
'modelId': 'string'},
'responsesEvaluatorModelConfig': {'maxOutputTokens': 'integer',
'modelId': 'string',
'reasoning': {'effort': 'string'},
'temperature': 'float',
'topP': 'float'}}}},
'evaluatorType': 'Builtin | ThirdParty | Custom | CustomCode | CustomDerived',
'provider': 'AWS | DeepEval | AutoEval | Custom'}
Retrieves detailed information about an evaluator, including its configuration, status, and metadata. Works with both built-in and custom evaluators.
See also: AWS API Documentation
Request Syntax
client.get_evaluator(
evaluatorId='string',
includedData='ALL_DATA'|'METADATA_ONLY'
)
string
[REQUIRED]
The unique identifier of the evaluator to retrieve. Can be a built-in evaluator ID (e.g., Builtin.Helpfulness) or a custom evaluator ID.
string
Controls which data is returned in the response. ALL_DATA (default) returns the full evaluator including decrypted instructions and rating scale. For evaluators encrypted with a customer managed KMS key, this requires kms:Decrypt permission on the key. METADATA_ONLY returns evaluator metadata and model configuration without instructions or rating scale, and does not require any KMS permissions.
dict
Response Syntax
{
'evaluatorArn': 'string',
'evaluatorId': 'string',
'evaluatorName': 'string',
'description': 'string',
'evaluatorConfig': {
'llmAsAJudge': {
'instructions': 'string',
'ratingScale': {
'numerical': [
{
'definition': 'string',
'value': 123.0,
'label': 'string'
},
],
'categorical': [
{
'definition': 'string',
'label': 'string'
},
]
},
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
},
'codeBased': {
'lambdaConfig': {
'lambdaArn': 'string',
'lambdaTimeoutInSeconds': 123
}
},
'derived': {
'baseEvaluatorId': 'string',
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
}
},
'evaluatorType': 'Builtin'|'ThirdParty'|'Custom'|'CustomCode'|'CustomDerived',
'provider': 'AWS'|'DeepEval'|'AutoEval'|'Custom',
'level': 'TOOL_CALL'|'TRACE'|'SESSION',
'status': 'ACTIVE'|'CREATING'|'CREATE_FAILED'|'UPDATING'|'UPDATE_FAILED'|'DELETING',
'createdAt': datetime(2015, 1, 1),
'updatedAt': datetime(2015, 1, 1),
'lockedForModification': True|False,
'kmsKeyArn': 'string'
}
Response Structure
(dict) --
evaluatorArn (string) --
The Amazon Resource Name (ARN) of the evaluator.
evaluatorId (string) --
The unique identifier of the evaluator.
evaluatorName (string) --
The name of the evaluator.
description (string) --
The description of the evaluator.
evaluatorConfig (dict) --
The configuration of the evaluator, including LLM-as-a-Judge or code-based settings.
llmAsAJudge (dict) --
The LLM-as-a-Judge configuration that uses a language model to evaluate agent performance based on custom instructions and rating scales.
instructions (string) --
The evaluation instructions that guide the language model in assessing agent performance, including criteria and evaluation guidelines.
ratingScale (dict) --
The rating scale that defines how the evaluator should score agent performance, either numerical or categorical.
numerical (list) --
The numerical rating scale with defined score values and descriptions for quantitative evaluation.
(dict) --
The definition of a numerical rating scale option that provides a numeric value with its description for evaluation scoring.
definition (string) --
The description that explains what this numerical rating represents and when it should be used.
value (float) --
The numerical value for this rating scale option.
label (string) --
The label or name that describes this numerical rating option.
categorical (list) --
The categorical rating scale with named categories and definitions for qualitative evaluation.
(dict) --
The definition of a categorical rating scale option that provides a named category with its description for evaluation scoring.
definition (string) --
The description that explains what this categorical rating represents and when it should be used.
label (string) --
The label or name of this categorical rating option.
modelConfig (dict) --
The model configuration that specifies which foundation model to use and how to configure it for evaluation.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) --
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) --
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
codeBased (dict) --
Configuration for a code-based evaluator that uses a customer-managed Lambda function to programmatically assess agent performance.
lambdaConfig (dict) --
The Lambda function configuration for code-based evaluation.
lambdaArn (string) --
The Amazon Resource Name (ARN) of the Lambda function that implements the evaluation logic.
lambdaTimeoutInSeconds (integer) --
The timeout in seconds for the Lambda function invocation. Defaults to 60. Must be between 1 and 300.
derived (dict) --
The configuration for an evaluator derived from an existing base evaluator (a built-in or third-party evaluator), run on your own model. The base evaluator supplies the prompt and scoring.
baseEvaluatorId (string) --
The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
modelConfig (dict) --
The configuration of the evaluator model that you supply.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) --
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) --
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
evaluatorType (string) --
The kind of evaluator resource. Valid values:
Builtin – An Amazon Web Services-managed global evaluator.
ThirdParty – An Amazon Web Services-managed global evaluator from a third-party provider.
Custom – A customer-created evaluator.
CustomCode – A customer-created code-based evaluator.
CustomDerived – A customer-created evaluator derived from an existing base evaluator.
provider (string) --
The source of the evaluator's logic: Amazon Web Services, a third-party library, or you.
level (string) --
The evaluation level ( TOOL_CALL, TRACE, or SESSION) that determines the scope of evaluation.
status (string) --
The current status of the evaluator.
createdAt (datetime) --
The timestamp when the evaluator was created.
updatedAt (datetime) --
The timestamp when the evaluator was last updated.
lockedForModification (boolean) --
Whether the evaluator is locked for modification due to being referenced by active online evaluation configurations.
kmsKeyArn (string) --
The Amazon Resource Name (ARN) of the customer managed KMS key used to encrypt the evaluator's sensitive data. This field is only present for evaluators encrypted with a customer managed key.
{'datasets': {'schemaType': {'GENERIC_EVALUATION_PREDEFINED_V1'}}}
Lists all datasets in the caller's account, paginated.
See also: AWS API Documentation
Request Syntax
client.list_datasets(
nextToken='string',
maxResults=123
)
string
The token for the next page of results.
integer
The maximum number of datasets to return per page.
dict
Response Syntax
{
'datasets': [
{
'datasetArn': 'string',
'datasetId': 'string',
'datasetName': 'string',
'description': 'string',
'status': 'CREATING'|'UPDATING'|'DELETING'|'ACTIVE'|'CREATE_FAILED'|'UPDATE_FAILED'|'DELETE_FAILED',
'draftStatus': 'MODIFIED'|'UNMODIFIED',
'schemaType': 'AGENTCORE_EVALUATION_PREDEFINED_V1'|'AGENTCORE_EVALUATION_SIMULATED_V1'|'GENERIC_EVALUATION_PREDEFINED_V1',
'exampleCount': 123,
'createdAt': datetime(2015, 1, 1),
'updatedAt': datetime(2015, 1, 1)
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
datasets (list) --
The list of datasets.
(dict) --
Summary information about a dataset.
datasetArn (string) --
The Amazon Resource Name (ARN) of the dataset.
datasetId (string) --
The unique identifier of the dataset.
datasetName (string) --
The name of the dataset.
description (string) --
The description of the dataset.
status (string) --
The current status of the dataset.
draftStatus (string) --
Publish synchronization state. Only authoritative when status is ACTIVE.
schemaType (string) --
The schema type of the dataset.
exampleCount (integer) --
The number of examples in the dataset.
createdAt (datetime) --
The timestamp when the dataset was created.
updatedAt (datetime) --
The timestamp when the dataset was last updated.
nextToken (string) --
The token for the next page of results, or null if there are no more results.
{'evaluators': {'evaluatorType': {'CustomDerived', 'ThirdParty'},
'provider': 'AWS | DeepEval | AutoEval | Custom'}}
Lists all available evaluators, including both builtin evaluators provided by the service and custom evaluators created by the user.
See also: AWS API Documentation
Request Syntax
client.list_evaluators(
nextToken='string',
maxResults=123
)
string
The pagination token from a previous request to retrieve the next page of results.
integer
The maximum number of evaluators to return in a single response.
dict
Response Syntax
{
'evaluators': [
{
'evaluatorArn': 'string',
'evaluatorId': 'string',
'evaluatorName': 'string',
'description': 'string',
'evaluatorType': 'Builtin'|'ThirdParty'|'Custom'|'CustomCode'|'CustomDerived',
'provider': 'AWS'|'DeepEval'|'AutoEval'|'Custom',
'level': 'TOOL_CALL'|'TRACE'|'SESSION',
'status': 'ACTIVE'|'CREATING'|'CREATE_FAILED'|'UPDATING'|'UPDATE_FAILED'|'DELETING',
'createdAt': datetime(2015, 1, 1),
'updatedAt': datetime(2015, 1, 1),
'lockedForModification': True|False,
'kmsKeyArn': 'string'
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
evaluators (list) --
The list of evaluator summaries containing basic information about each evaluator.
(dict) --
The summary information about an evaluator, including basic metadata and status information.
evaluatorArn (string) --
The Amazon Resource Name (ARN) of the evaluator.
evaluatorId (string) --
The unique identifier of the evaluator.
evaluatorName (string) --
The name of the evaluator.
description (string) --
The description of the evaluator.
evaluatorType (string) --
The type of evaluator, indicating whether it is a built-in evaluator provided by the service or a custom evaluator created by the user.
provider (string) --
The source of the evaluator's logic: Amazon Web Services, a third-party library, or you.
level (string) --
The evaluation level ( TOOL_CALL, TRACE, or SESSION) that determines the scope of evaluation.
status (string) --
The current status of the evaluator.
createdAt (datetime) --
The timestamp when the evaluator was created.
updatedAt (datetime) --
The timestamp when the evaluator was last updated.
lockedForModification (boolean) --
Whether the evaluator is locked for modification due to being referenced by active online evaluation configurations.
kmsKeyArn (string) --
The Amazon Resource Name (ARN) of the customer managed KMS key used to encrypt the evaluator's sensitive data. This field is only present for evaluators encrypted with a customer managed key.
nextToken (string) --
The pagination token to use in a subsequent request to retrieve the next page of results.
{'evaluatorConfig': {'derived': {'baseEvaluatorId': 'string',
'modelConfig': {'bedrockEvaluatorModelConfig': {'additionalModelRequestFields': {},
'inferenceConfig': {'maxTokens': 'integer',
'stopSequences': ['string'],
'temperature': 'float',
'topP': 'float'},
'modelId': 'string'},
'responsesEvaluatorModelConfig': {'maxOutputTokens': 'integer',
'modelId': 'string',
'reasoning': {'effort': 'string'},
'temperature': 'float',
'topP': 'float'}}}}}
Updates a custom evaluator's configuration, description, or evaluation level. Built-in evaluators cannot be updated. The evaluator must not be locked for modification.
See also: AWS API Documentation
Request Syntax
client.update_evaluator(
clientToken='string',
evaluatorId='string',
description='string',
evaluatorConfig={
'llmAsAJudge': {
'instructions': 'string',
'ratingScale': {
'numerical': [
{
'definition': 'string',
'value': 123.0,
'label': 'string'
},
],
'categorical': [
{
'definition': 'string',
'label': 'string'
},
]
},
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
},
'codeBased': {
'lambdaConfig': {
'lambdaArn': 'string',
'lambdaTimeoutInSeconds': 123
}
},
'derived': {
'baseEvaluatorId': 'string',
'modelConfig': {
'bedrockEvaluatorModelConfig': {
'modelId': 'string',
'inferenceConfig': {
'maxTokens': 123,
'temperature': ...,
'topP': ...,
'stopSequences': [
'string',
]
},
'additionalModelRequestFields': {...}|[...]|123|123.4|'string'|True|None
},
'responsesEvaluatorModelConfig': {
'modelId': 'string',
'maxOutputTokens': 123,
'temperature': ...,
'topP': ...,
'reasoning': {
'effort': 'string'
}
}
}
}
},
level='TOOL_CALL'|'TRACE'|'SESSION',
kmsKeyArn='string'
)
string
A unique, case-sensitive identifier to ensure that the API request completes no more than one time. If you don't specify this field, a value is randomly generated for you. If this token matches a previous request, the service ignores the request, but doesn't return an error. For more information, see Ensuring idempotency.
This field is autopopulated if not provided.
string
[REQUIRED]
The unique identifier of the evaluator to update.
string
The updated description of the evaluator.
dict
The updated configuration for the evaluator. Specify either LLM-as-a-Judge settings with instructions, rating scale, and model configuration, or code-based settings with a customer-managed Lambda function.
llmAsAJudge (dict) --
The LLM-as-a-Judge configuration that uses a language model to evaluate agent performance based on custom instructions and rating scales.
instructions (string) -- [REQUIRED]
The evaluation instructions that guide the language model in assessing agent performance, including criteria and evaluation guidelines.
ratingScale (dict) -- [REQUIRED]
The rating scale that defines how the evaluator should score agent performance, either numerical or categorical.
numerical (list) --
The numerical rating scale with defined score values and descriptions for quantitative evaluation.
(dict) --
The definition of a numerical rating scale option that provides a numeric value with its description for evaluation scoring.
definition (string) -- [REQUIRED]
The description that explains what this numerical rating represents and when it should be used.
value (float) -- [REQUIRED]
The numerical value for this rating scale option.
label (string) -- [REQUIRED]
The label or name that describes this numerical rating option.
categorical (list) --
The categorical rating scale with named categories and definitions for qualitative evaluation.
(dict) --
The definition of a categorical rating scale option that provides a named category with its description for evaluation scoring.
definition (string) -- [REQUIRED]
The description that explains what this categorical rating represents and when it should be used.
label (string) -- [REQUIRED]
The label or name of this categorical rating option.
modelConfig (dict) -- [REQUIRED]
The model configuration that specifies which foundation model to use and how to configure it for evaluation.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
codeBased (dict) --
Configuration for a code-based evaluator that uses a customer-managed Lambda function to programmatically assess agent performance.
lambdaConfig (dict) --
The Lambda function configuration for code-based evaluation.
lambdaArn (string) -- [REQUIRED]
The Amazon Resource Name (ARN) of the Lambda function that implements the evaluation logic.
lambdaTimeoutInSeconds (integer) --
The timeout in seconds for the Lambda function invocation. Defaults to 60. Must be between 1 and 300.
derived (dict) --
The configuration for an evaluator derived from an existing base evaluator (a built-in or third-party evaluator), run on your own model. The base evaluator supplies the prompt and scoring.
baseEvaluatorId (string) -- [REQUIRED]
The identifier of the base evaluator whose logic to run (a Builtin.* or ThirdParty.* evaluator).
modelConfig (dict) -- [REQUIRED]
The configuration of the evaluator model that you supply.
bedrockEvaluatorModelConfig (dict) --
The Amazon Bedrock model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the Amazon Bedrock model to use for evaluation. Must be a supported foundation model available in your region.
inferenceConfig (dict) --
The inference configuration parameters that control model behavior during evaluation, including temperature, token limits, and sampling settings.
maxTokens (integer) --
The maximum number of tokens to generate in the model response during evaluation.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
stopSequences (list) --
The list of sequences that will cause the model to stop generating tokens when encountered.
(string) --
additionalModelRequestFields (:ref:`document<document>`) --
Additional model-specific request fields to customize model behavior beyond the standard inference configuration.
responsesEvaluatorModelConfig (dict) --
The OpenResponses model configuration for evaluation.
modelId (string) -- [REQUIRED]
The identifier of the model to use for evaluation.
maxOutputTokens (integer) --
The maximum number of tokens to generate in the model response, including visible output and reasoning tokens.
temperature (float) --
The temperature value that controls randomness in the model's responses. Lower values produce more deterministic outputs.
topP (float) --
The top-p sampling parameter that controls the diversity of the model's responses by limiting the cumulative probability of token choices.
reasoning (dict) --
The reasoning configuration for reasoning models. Non-reasoning models ignore this configuration.
effort (string) --
The level of reasoning effort the model applies when generating a response. For supported values, see the model provider's documentation.
string
The updated evaluation level ( TOOL_CALL, TRACE, or SESSION) that determines the scope of evaluation.
string
The Amazon Resource Name (ARN) of a customer managed KMS key to use for encrypting sensitive evaluator data. Specify a new key ARN to rotate the encryption key, or specify a key ARN to add encryption to an evaluator that was previously created without one. When you rotate to a new key, the service decrypts the existing data with the old key and re-encrypts it with the new key. Only symmetric encryption KMS keys are supported. For more information, see Encryption at rest for AgentCore Evaluations.
dict
Response Syntax
{
'evaluatorArn': 'string',
'evaluatorId': 'string',
'updatedAt': datetime(2015, 1, 1),
'status': 'ACTIVE'|'CREATING'|'CREATE_FAILED'|'UPDATING'|'UPDATE_FAILED'|'DELETING'
}
Response Structure
(dict) --
evaluatorArn (string) --
The Amazon Resource Name (ARN) of the updated evaluator.
evaluatorId (string) --
The unique identifier of the updated evaluator.
updatedAt (datetime) --
The timestamp when the evaluator was last updated.
status (string) --
The status of the evaluator update operation.