2026/10/22 - Amazon Bedrock AgentCore - 3 updated api methods
Changes Adds the dimension name to evaluation results and batch evaluation summaries. For multi-dimensional LLM-as-a-Judge evaluators, each result and summary now identifies which dimension it corresponds to.
{'evaluationResults': {'evaluationDimensionName': 'string'}}
Performs on-demand evaluation of agent traces using a specified evaluator. This synchronous API accepts traces in OpenTelemetry format and returns immediate scoring results with detailed explanations.
See also: AWS API Documentation
Request Syntax
client.evaluate(
evaluatorId='string',
evaluationInput={
'sessionSpans': [
{...}|[...]|123|123.4|'string'|True|None,
]
},
evaluationTarget={
'spanIds': [
'string',
],
'traceIds': [
'string',
]
},
evaluationReferenceInputs=[
{
'context': {
'spanContext': {
'sessionId': 'string',
'traceId': 'string',
'spanId': 'string'
}
},
'expectedResponse': {
'text': 'string'
},
'assertions': [
{
'text': 'string'
},
],
'expectedTrajectory': {
'toolNames': [
'string',
]
}
},
]
)
string
[REQUIRED]
The unique identifier of the evaluator to use for scoring. Can be a built-in evaluator (e.g., Builtin.Helpfulness, Builtin.Correctness) or a custom evaluator Id created through the control plane API.
dict
[REQUIRED]
The input data containing agent session spans to be evaluated. Includes a list of spans in OpenTelemetry format from supported frameworks like Strands (AgentCore Runtime) or LangGraph with OpenInference instrumentation.
sessionSpans (list) --
The collection of spans representing agent execution traces within a session. Each span contains detailed information about tool calls, model interactions, and other agent activities that can be evaluated for quality and performance.
(:ref:`document<document>`) --
dict
The specific trace or span IDs to evaluate within the provided input. Allows targeting evaluation at different levels: individual tool calls, single request-response interactions (traces), or entire conversation sessions.
spanIds (list) --
The list of specific span IDs to evaluate within the provided traces. Used to target evaluation at individual tool calls or specific operations within the agent's execution flow.
(string) --
traceIds (list) --
The list of trace IDs to evaluate, representing complete request-response interactions. Used to evaluate entire conversation turns or specific agent interactions within a session.
(string) --
list
Ground truth data to compare against agent responses during evaluation. Allows to provide expected responses, assertions, and expected tool trajectories at different evaluation levels. Session-level reference inputs apply to the entire conversation, while trace-level reference inputs target specific request-response interactions identified by trace ID.
(dict) --
A reference input containing ground truth data for evaluation, scoped to a specific context level (session or trace) through its span context.
context (dict) -- [REQUIRED]
The span context that identifies which session or trace this reference input applies to, used for correlating ground truth with agent output.
spanContext (dict) --
The span context information that uniquely identifies the trace and span being evaluated, including session ID, trace ID, and span ID for precise targeting within the agent's execution flow.
sessionId (string) -- [REQUIRED]
The unique identifier of the session containing this span. Sessions represent complete conversation flows and are detected using configurable SessionTimeoutMinutes (default 15 minutes).
traceId (string) --
The unique identifier of the trace containing this span. Traces represent individual request-response interactions within a session and group related spans together.
spanId (string) --
The unique identifier of the specific span being referenced. Spans represent individual operations like tool calls, model invocations, or other discrete actions within the agent's execution.
expectedResponse (dict) --
The expected response for trace-level evaluation. Built-in evaluators that support this field compare the agent's actual response against this value for assessment. Custom evaluators can access it through the {expected_response} placeholder in their instructions.
text (string) --
The text content of the ground truth data. Used for expected response text and assertion statements.
assertions (list) --
A list of assertion statements for session-level evaluation. Each assertion describes an expected behavior or outcome the agent should demonstrate during the session.
(dict) --
A content block for ground truth data in evaluation reference inputs. Supports text content for expected responses and assertions.
text (string) --
The text content of the ground truth data. Used for expected response text and assertion statements.
expectedTrajectory (dict) --
The expected tool call sequence for session-level trajectory evaluation. Contains a list of tool names representing the tools the agent is expected to invoke.
toolNames (list) --
The list of tool names representing the expected tool call sequence.
(string) --
dict
Response Syntax
{
'evaluationResults': [
{
'evaluatorArn': 'string',
'evaluatorId': 'string',
'evaluatorName': 'string',
'evaluationDimensionName': 'string',
'explanation': 'string',
'context': {
'spanContext': {
'sessionId': 'string',
'traceId': 'string',
'spanId': 'string'
}
},
'value': 123.0,
'label': 'string',
'tokenUsage': {
'inputTokens': 123,
'outputTokens': 123,
'totalTokens': 123
},
'errorMessage': 'string',
'errorCode': 'string',
'ignoredReferenceInputFields': [
'string',
]
},
]
}
Response Structure
(dict) --
evaluationResults (list) --
The detailed evaluation results containing scores, explanations, and metadata. Includes the evaluator information, numerical or categorical ratings based on the evaluator's rating scale, and token usage statistics for the evaluation process.
(dict) --
The comprehensive result of an evaluation containing the score, explanation, evaluator metadata, and execution details. Provides both quantitative ratings and qualitative insights about agent performance.
evaluatorArn (string) --
The Amazon Resource Name (ARN) of the evaluator used to generate this result. For custom evaluators, this is the full ARN; for built-in evaluators, this follows the pattern Builtin.{EvaluatorName}.
evaluatorId (string) --
The unique identifier of the evaluator that produced this result. This matches the evaluatorId provided in the evaluation request and can be used to identify which evaluator generated specific results.
evaluatorName (string) --
The human-readable name of the evaluator used for this evaluation. For built-in evaluators, this is the descriptive name (e.g., "Helpfulness", "Correctness"); for custom evaluators, this is the user-defined name.
evaluationDimensionName (string) --
The name of the dimension that this result scores. This field is set only on results produced by an evaluator that uses a multi-dimensional rating scale, where each dimension is scored independently. For evaluators with a single rating scale, this field is absent.
explanation (string) --
The detailed explanation provided by the evaluator describing the reasoning behind the assigned score. This qualitative feedback helps understand why specific ratings were given and provides actionable insights for improvement.
context (dict) --
The contextual information associated with this evaluation result, including span context details that identify the specific traces and sessions that were evaluated.
spanContext (dict) --
The span context information that uniquely identifies the trace and span being evaluated, including session ID, trace ID, and span ID for precise targeting within the agent's execution flow.
sessionId (string) --
The unique identifier of the session containing this span. Sessions represent complete conversation flows and are detected using configurable SessionTimeoutMinutes (default 15 minutes).
traceId (string) --
The unique identifier of the trace containing this span. Traces represent individual request-response interactions within a session and group related spans together.
spanId (string) --
The unique identifier of the specific span being referenced. Spans represent individual operations like tool calls, model invocations, or other discrete actions within the agent's execution.
value (float) --
The numerical score assigned by the evaluator according to its configured rating scale. For numerical scales, this is a decimal value within the defined range. This field is not allowed for categorical scales.
label (string) --
The categorical label assigned by the evaluator when using a categorical rating scale. This provides a human-readable description of the evaluation result (e.g., "Excellent", "Good", "Poor") corresponding to the numerical value. For numerical scales, this field is optional and provides a natural language explanation of what the value means (e.g., value 0.5 = "Somewhat Helpful").
tokenUsage (dict) --
The token consumption statistics for this evaluation, including input tokens, output tokens, and total tokens used by the underlying language model during the evaluation process.
inputTokens (integer) --
The number of tokens consumed for input processing during the evaluation. Includes tokens from the evaluation prompt, agent traces, and any additional context provided to the evaluator model.
outputTokens (integer) --
The number of tokens generated by the evaluator model in its response. Includes tokens for the score, explanation, and any additional output produced during the evaluation process.
totalTokens (integer) --
The total number of tokens consumed during the evaluation, calculated as the sum of input and output tokens. Used for cost calculation and rate limiting within the service limits.
errorMessage (string) --
The error message describing what went wrong if the evaluation failed. Provides detailed information about evaluation failures to help diagnose and resolve issues with evaluator configuration or input data.
errorCode (string) --
The error code indicating the type of failure that occurred during evaluation. Used to programmatically identify and handle different categories of evaluation errors.
ignoredReferenceInputFields (list) --
The list of reference input field names that were provided but not used by the evaluator. Helps identify which ground truth data was not consumed during evaluation.
(string) --
{'evaluationResults': {'evaluatorSummaries': {'evaluationDimensionName': 'string'}}}
Retrieves detailed information about a batch evaluation, including its status, configuration, results, and any error details.
See also: AWS API Documentation
Request Syntax
client.get_batch_evaluation(
batchEvaluationId='string'
)
string
[REQUIRED]
The unique identifier of the batch evaluation to retrieve.
dict
Response Syntax
{
'batchEvaluationId': 'string',
'batchEvaluationArn': 'string',
'batchEvaluationName': 'string',
'status': 'PENDING'|'IN_PROGRESS'|'COMPLETED'|'COMPLETED_WITH_ERRORS'|'FAILED'|'STOPPING'|'STOPPED'|'DELETING',
'createdAt': datetime(2015, 1, 1),
'evaluators': [
{
'evaluatorId': 'string'
},
],
'insights': [
{
'insightId': 'string'
},
],
'dataSourceConfig': {
'cloudWatchLogs': {
'serviceNames': [
'string',
],
'logGroupNames': [
'string',
],
'logGroupNamePrefixes': [
'string',
],
'filterConfig': {
'sessionIds': [
'string',
],
'timeRange': {
'startTime': datetime(2015, 1, 1),
'endTime': datetime(2015, 1, 1)
},
'sessionTraceIds': [
{
'sessionId': 'string',
'traceIds': [
'string',
]
},
]
}
},
'onlineEvaluationConfigSource': {
'onlineEvaluationConfigArn': 'string',
'timeRange': {
'startTime': datetime(2015, 1, 1),
'endTime': datetime(2015, 1, 1)
}
}
},
'outputConfig': {
'cloudWatchConfig': {
'logGroupName': 'string',
'logStreamName': 'string',
'metricsNamespace': 'string',
'resultDestination': 'DEDICATED_LOG_GROUP'|'SOURCE_LOG_GROUP'
}
},
'evaluationResults': {
'numberOfSessionsCompleted': 123,
'numberOfSessionsInProgress': 123,
'numberOfSessionsFailed': 123,
'totalNumberOfSessions': 123,
'numberOfSessionsIgnored': 123,
'evaluatorSummaries': [
{
'evaluatorId': 'string',
'evaluationDimensionName': 'string',
'statistics': {
'averageScore': 123.0
},
'totalEvaluated': 123,
'totalFailed': 123
},
]
},
'insightResults': [
{
'type': 'FAILURE',
'name': 'string',
'description': 'string',
'createdAt': datetime(2015, 1, 1),
'affectedSessionCount': 123,
'sampleSessions': [
{
'sessionId': 'string',
'lastSeenAt': datetime(2015, 1, 1),
'findings': [
{
'targetId': 'string',
'targetType': 'SPAN_ID'|'TRACE_ID'|'SPAN_NAME',
'description': 'string',
'category': 'string',
'confidence': 123.0
},
]
},
],
'rootCauses': [
{
'name': 'string',
'explanation': 'string',
'affectedSessionCount': 123,
'suggestedFix': 'string',
'sampleSessions': [
{
'sessionId': 'string',
'lastSeenAt': datetime(2015, 1, 1),
'findings': [
{
'targetId': 'string',
'targetType': 'SPAN_ID'|'TRACE_ID'|'SPAN_NAME',
'description': 'string',
'category': 'string',
'confidence': 123.0
},
]
},
]
},
]
},
],
'failureAnalysisResult': {
'failures': [
{
'clusterId': 123,
'name': 'string',
'description': 'string',
'affectedSessionCount': 123,
'subCategories': [
{
'clusterId': 123,
'name': 'string',
'description': 'string',
'affectedSessionCount': 123,
'rootCauses': [
{
'clusterId': 123,
'name': 'string',
'rootCause': 'string',
'recommendation': 'string',
'affectedSessionCount': 123,
'affectedSessions': [
{
'sessionId': 'string',
'explanation': 'string',
'fixType': 'string',
'recommendation': 'string',
'failureSpans': [
{
'spanId': 'string',
'traceId': 'string',
'signals': [
{
'category': 'execution-error-category-authentication'|'execution-error-category-resource-not-found'|'execution-error-category-service-errors'|'execution-error-category-rate-limiting'|'execution-error-category-formatting'|'execution-error-category-timeout'|'execution-error-category-resource-exhaustion'|'execution-error-category-environment'|'execution-error-category-tool-schema'|'task-instruction-category-non-compliance'|'task-instruction-category-problem-id'|'incorrect-actions-category-tool-selection'|'incorrect-actions-category-poor-information-retrieval'|'incorrect-actions-category-clarification'|'incorrect-actions-category-inappropriate-info-request'|'context-handling-error-category-context-handling-failures'|'hallucination-category-hall-capabilities'|'hallucination-category-hall-misunderstand'|'hallucination-category-hall-usage'|'hallucination-category-hall-history'|'hallucination-category-hall-params'|'hallucination-category-fabricate-tool-outputs'|'repetitive-behavior-category-repetition-tool'|'repetitive-behavior-category-repetition-info'|'repetitive-behavior-category-step-repetition'|'orchestration-related-errors-category-reasoning-mismatch'|'orchestration-related-errors-category-goal-deviation'|'orchestration-related-errors-category-premature-termination'|'orchestration-related-errors-category-unaware-termination'|'llm-output-category-nonsensical'|'configuration-mismatch-category-tool-definition'|'coding-use-case-specific-failure-types-category-edge-case-oversights'|'coding-use-case-specific-failure-types-category-dependency-issues'|'other',
'evidence': 'string',
'confidence': 123.0
},
]
},
]
},
]
},
]
},
]
},
]
},
'userIntentResult': {
'userIntents': [
{
'clusterId': 123,
'name': 'string',
'description': 'string',
'affectedSessionCount': 123,
'affectedSessions': [
{
'sessionId': 'string',
'userMessages': [
'string',
]
},
]
},
]
},
'executionSummaryResult': {
'executionSummaries': [
{
'clusterId': 123,
'name': 'string',
'description': 'string',
'affectedSessionCount': 123,
'affectedSessions': [
{
'sessionId': 'string',
'approachTaken': 'string',
'finalOutcome': 'string'
},
]
},
]
},
'errorDetails': [
'string',
],
'description': 'string',
'updatedAt': datetime(2015, 1, 1),
'kmsKeyArn': 'string'
}
Response Structure
(dict) --
batchEvaluationId (string) --
The unique identifier of the batch evaluation.
batchEvaluationArn (string) --
The Amazon Resource Name (ARN) of the batch evaluation.
batchEvaluationName (string) --
The name of the batch evaluation.
status (string) --
The current status of the batch evaluation.
createdAt (datetime) --
The timestamp when the batch evaluation was created.
evaluators (list) --
The list of evaluators applied during the batch evaluation.
(dict) --
An evaluator to run against sessions during batch evaluation.
evaluatorId (string) --
The unique identifier of the evaluator. Can reference built-in evaluators (e.g., Builtin.Helpfulness) or custom evaluators.
insights (list) --
The list of insight analyses applied during the batch evaluation.
(dict) --
A reference to an insight analysis to run against sessions during batch evaluation. Insights provide deeper analysis beyond individual evaluator scores, including failure detection, user intent clustering, and execution summarization.
insightId (string) --
The unique identifier of the insight to run.
dataSourceConfig (dict) --
The data source configuration specifying where agent traces are pulled from.
cloudWatchLogs (dict) --
Configuration for pulling agent session traces from CloudWatch Logs.
serviceNames (list) --
The list of agent service names to filter traces within the specified log groups.
(string) --
logGroupNames (list) --
The list of CloudWatch log group names to read agent traces from. Maximum of 10 log groups.
(string) --
logGroupNamePrefixes (list) --
The list of CloudWatch log group name prefixes to read agent traces from. Specify this instead of logGroupNames to match log groups by prefix. Maximum of 5 prefixes. Specify either logGroupNames or logGroupNamePrefixes, not both. One of the two is required.
(string) --
Prefix of a CloudWatch Logs log group name.
filterConfig (dict) --
Optional filter configuration to narrow down which sessions to evaluate.
sessionIds (list) --
A list of specific session IDs to evaluate. If specified, only these sessions are included in the evaluation.
(string) --
timeRange (dict) --
The time range filter for selecting sessions to evaluate.
startTime (datetime) --
The start time of the time range. Only sessions with activity at or after this timestamp are included.
endTime (datetime) --
The end time of the time range. Only sessions with activity before this timestamp are included.
sessionTraceIds (list) --
A list of session and trace ID pairs that restrict evaluation to specific traces within a session. If specified, only the listed traces are evaluated instead of the entire session.
(dict) --
A pairing of a session with the specific trace IDs to evaluate within that session. Use this to evaluate individual traces rather than an entire session.
sessionId (string) --
The unique identifier of the session that contains the traces to evaluate.
traceIds (list) --
The list of trace IDs within the session to evaluate.
(string) --
onlineEvaluationConfigSource (dict) --
Reference an existing OnlineEvaluationConfig as session source
onlineEvaluationConfigArn (string) --
The Amazon Resource Name (ARN) of the online evaluation configuration to use as the session source.
timeRange (dict) --
Optional session filter configuration to narrow down which sessions from the online evaluation configuration to include.
startTime (datetime) --
The start time of the time range. Only sessions with activity at or after this timestamp are included.
endTime (datetime) --
The end time of the time range. Only sessions with activity before this timestamp are included.
outputConfig (dict) --
The output configuration specifying where evaluation results are written.
cloudWatchConfig (dict) --
The CloudWatch Logs configuration for writing evaluation results.
logGroupName (string) --
The name of the CloudWatch log group where evaluation results will be written. This value doesn't apply when resultDestination is SOURCE_LOG_GROUP, because results are written back to the trace source log group. The name can't be under the service-reserved /aws/bedrock-agentcore/evaluations/ namespace, apart from the service-managed default group.
logStreamName (string) --
The name of the CloudWatch log stream where evaluation results will be written.
metricsNamespace (string) --
The CloudWatch metrics namespace where evaluation result metrics are published. If you omit this value, the service publishes metrics to Bedrock-AgentCore/Evaluations. This value can't begin with AWS/.
resultDestination (string) --
The destination where evaluation results are written. Valid values:
DEDICATED_LOG_GROUP (default) – Writes results to a dedicated result log group.
SOURCE_LOG_GROUP – Writes results back to the log group that the agent traces were read from. If you use this value, don't specify logGroupName.
evaluationResults (dict) --
The aggregated evaluation results, including session completion counts and evaluator score summaries.
numberOfSessionsCompleted (integer) --
The number of sessions that have been successfully evaluated.
numberOfSessionsInProgress (integer) --
The number of sessions currently being evaluated.
numberOfSessionsFailed (integer) --
The number of sessions that failed evaluation.
totalNumberOfSessions (integer) --
The total number of sessions included in the batch evaluation.
numberOfSessionsIgnored (integer) --
The number of sessions that were ignored during evaluation.
evaluatorSummaries (list) --
A list of per-evaluator summary statistics.
(dict) --
Summary statistics for a single evaluator within a batch evaluation.
evaluatorId (string) --
The unique identifier of the evaluator.
evaluationDimensionName (string) --
The name of the dimension that this summary covers. This field is set only for evaluators that use a multi-dimensional rating scale, where each dimension is summarized separately. For single-scale evaluators, this field is not set.
statistics (dict) --
The aggregated statistics for this evaluator.
averageScore (float) --
The average score across all evaluated sessions for this evaluator.
totalEvaluated (integer) --
The total number of sessions evaluated by this evaluator.
totalFailed (integer) --
The total number of sessions that failed evaluation by this evaluator.
insightResults (list) --
The insights generated when the batch evaluation includes the Builtin.FailureAnalysis evaluator. Each insight groups related failures found across the evaluated sessions. An empty list means that no failures were found. If the analysis fails, this field is omitted and errorDetails describes the error.
(dict) --
An insight that groups related failures found across the sessions in a batch evaluation.
type (string) --
The type of insight. FAILURE indicates a group of related failures.
name (string) --
A generated label for the insight. The name is not an identifier and can differ between batch evaluations.
description (string) --
A description of the failure pattern that the insight represents.
createdAt (datetime) --
The timestamp when the insight was generated.
affectedSessionCount (integer) --
The number of distinct sessions affected by the insight.
sampleSessions (list) --
A sample of up to 10 sessions affected by the insight. Returned only when the insight has no root causes. Otherwise, sample sessions are returned for each root cause.
(dict) --
A session affected by an insight or root cause. Use the session ID to find the per-session evaluation results in CloudWatch Logs.
sessionId (string) --
The unique identifier of the session.
lastSeenAt (datetime) --
The latest timestamp observed for the session in the batch evaluation input.
findings (list) --
The failures found in the session that belong to the insight or root cause, up to 10.
(dict) --
A failure found in a session.
targetId (string) --
The identifier or name of the location in the session where the failure was found, interpreted according to targetType.
targetType (string) --
The type of location that targetId refers to.
description (string) --
A description of the failure.
category (string) --
The category of the failure.
confidence (float) --
The confidence that the finding is a failure, from 0 to 1.
rootCauses (list) --
The root causes identified for the insight.
(dict) --
A root cause that contributes to an insight.
name (string) --
A generated label for the root cause.
explanation (string) --
An explanation of how the root cause leads to the failures in the insight.
affectedSessionCount (integer) --
The number of distinct sessions affected by the root cause. A session can be affected by more than one root cause of the same insight.
suggestedFix (string) --
A suggested change to the agent that addresses the root cause.
sampleSessions (list) --
A sample of up to 10 sessions affected by the root cause.
(dict) --
A session affected by an insight or root cause. Use the session ID to find the per-session evaluation results in CloudWatch Logs.
sessionId (string) --
The unique identifier of the session.
lastSeenAt (datetime) --
The latest timestamp observed for the session in the batch evaluation input.
findings (list) --
The failures found in the session that belong to the insight or root cause, up to 10.
(dict) --
A failure found in a session.
targetId (string) --
The identifier or name of the location in the session where the failure was found, interpreted according to targetType.
targetType (string) --
The type of location that targetId refers to.
description (string) --
A description of the failure.
category (string) --
The category of the failure.
confidence (float) --
The confidence that the finding is a failure, from 0 to 1.
failureAnalysisResult (dict) --
The failure analysis results from insights, containing categorized failure clusters with root causes and recommendations.
failures (list) --
The list of failure category clusters identified across analyzed sessions.
(dict) --
A top-level failure category identified by clustering similar failure patterns across sessions.
clusterId (integer) --
The unique identifier of the failure category cluster.
name (string) --
The name of the failure category.
description (string) --
A description of the failure category pattern.
affectedSessionCount (integer) --
The number of sessions affected by this failure category.
subCategories (list) --
The list of failure subcategories within this category.
(dict) --
A subcategory of failures within a top-level failure category.
clusterId (integer) --
The unique identifier of the failure subcategory cluster.
name (string) --
The name of the failure subcategory.
description (string) --
A description of the failure subcategory pattern.
affectedSessionCount (integer) --
The number of sessions affected by this failure subcategory.
rootCauses (list) --
The list of root cause clusters identified within this subcategory.
(dict) --
A cluster of similar root causes identified within a failure subcategory.
clusterId (integer) --
The unique identifier of the root cause cluster.
name (string) --
The name of the root cause cluster.
rootCause (string) --
The root cause explanation for this cluster of failures.
recommendation (string) --
The recommended fix for this root cause.
affectedSessionCount (integer) --
The number of sessions affected by this root cause.
affectedSessions (list) --
The list of sessions affected by this root cause.
(dict) --
A session affected by a detected failure pattern, including root cause details.
sessionId (string) --
The unique identifier of the affected session.
explanation (string) --
An explanation of how the failure manifested in this session.
fixType (string) --
The type of fix recommended for this failure.
recommendation (string) --
The specific fix recommendation for this session.
failureSpans (list) --
The list of spans where failures were detected in this session.
(dict) --
Details about a specific span where a failure was detected.
spanId (string) --
The unique identifier of the span where the failure occurred.
traceId (string) --
The trace identifier associated with the failure span.
signals (list) --
The failure signals detected in this span.
(dict) --
A signal indicating a detected failure within a span.
category (string) --
The failure category classification for this signal.
evidence (string) --
The evidence supporting the failure detection.
confidence (float) --
The confidence score of the failure detection.
userIntentResult (dict) --
The user intent clustering results from insights, containing grouped user intents across evaluated sessions.
userIntents (list) --
The list of user intent clusters identified across analyzed sessions.
(dict) --
A cluster of similar user intents identified across sessions.
clusterId (integer) --
The unique identifier of the user intent cluster.
name (string) --
The name of the user intent cluster.
description (string) --
A description of the user intent pattern.
affectedSessionCount (integer) --
The number of sessions with this user intent.
affectedSessions (list) --
The list of sessions with this user intent.
(dict) --
A session associated with a user intent cluster.
sessionId (string) --
The unique identifier of the session.
userMessages (list) --
The user messages from this session that contributed to the intent cluster.
(string) --
executionSummaryResult (dict) --
The execution summary clustering results from insights, containing grouped execution patterns across evaluated sessions.
executionSummaries (list) --
The list of execution summary clusters identified across analyzed sessions.
(dict) --
A cluster of similar execution patterns identified across sessions.
clusterId (integer) --
The unique identifier of the execution summary cluster.
name (string) --
The name of the execution pattern cluster.
description (string) --
A description of the execution pattern.
affectedSessionCount (integer) --
The number of sessions with this execution pattern.
affectedSessions (list) --
The list of sessions with this execution pattern.
(dict) --
A session associated with an execution summary cluster.
sessionId (string) --
The unique identifier of the session.
approachTaken (string) --
The approach taken by the agent during this session.
finalOutcome (string) --
The final outcome of the session.
errorDetails (list) --
The error details if the batch evaluation encountered failures.
(string) --
description (string) --
The description of the batch evaluation.
updatedAt (datetime) --
The timestamp when the batch evaluation was last updated.
kmsKeyArn (string) --
The ARN of the KMS key used to encrypt evaluation data.
{'batchEvaluations': {'evaluationResults': {'evaluatorSummaries': {'evaluationDimensionName': 'string'}}}}
Lists all batch evaluations in the account, providing summary information about each evaluation's status and configuration.
See also: AWS API Documentation
Request Syntax
client.list_batch_evaluations(
maxResults=123,
nextToken='string'
)
integer
The maximum number of results to return in the response. If the total number of results is greater than this value, use the token returned in the response in the nextToken field when making another request to return the next batch of results.
string
If the total number of results is greater than the maxResults value provided in the request, enter the token returned in the nextToken field in the response in this field to return the next batch of results.
dict
Response Syntax
{
'batchEvaluations': [
{
'batchEvaluationId': 'string',
'batchEvaluationArn': 'string',
'batchEvaluationName': 'string',
'status': 'PENDING'|'IN_PROGRESS'|'COMPLETED'|'COMPLETED_WITH_ERRORS'|'FAILED'|'STOPPING'|'STOPPED'|'DELETING',
'createdAt': datetime(2015, 1, 1),
'description': 'string',
'evaluators': [
{
'evaluatorId': 'string'
},
],
'insights': [
{
'insightId': 'string'
},
],
'evaluationResults': {
'numberOfSessionsCompleted': 123,
'numberOfSessionsInProgress': 123,
'numberOfSessionsFailed': 123,
'totalNumberOfSessions': 123,
'numberOfSessionsIgnored': 123,
'evaluatorSummaries': [
{
'evaluatorId': 'string',
'evaluationDimensionName': 'string',
'statistics': {
'averageScore': 123.0
},
'totalEvaluated': 123,
'totalFailed': 123
},
]
},
'errorDetails': [
'string',
],
'kmsKeyArn': 'string',
'updatedAt': datetime(2015, 1, 1)
},
],
'nextToken': 'string'
}
Response Structure
(dict) --
batchEvaluations (list) --
The list of batch evaluation summaries.
(dict) --
Summary representation for list responses.
batchEvaluationId (string) --
The unique identifier of the batch evaluation.
batchEvaluationArn (string) --
The Amazon Resource Name (ARN) of the batch evaluation.
batchEvaluationName (string) --
The name of the batch evaluation.
status (string) --
The current status of the batch evaluation.
createdAt (datetime) --
The timestamp when the batch evaluation was created.
description (string) --
The description of the batch evaluation.
evaluators (list) --
The list of evaluators applied during the batch evaluation.
(dict) --
An evaluator to run against sessions during batch evaluation.
evaluatorId (string) --
The unique identifier of the evaluator. Can reference built-in evaluators (e.g., Builtin.Helpfulness) or custom evaluators.
insights (list) --
The list of insight analyses applied during the batch evaluation.
(dict) --
A reference to an insight analysis to run against sessions during batch evaluation. Insights provide deeper analysis beyond individual evaluator scores, including failure detection, user intent clustering, and execution summarization.
insightId (string) --
The unique identifier of the insight to run.
evaluationResults (dict) --
The aggregated evaluation results.
numberOfSessionsCompleted (integer) --
The number of sessions that have been successfully evaluated.
numberOfSessionsInProgress (integer) --
The number of sessions currently being evaluated.
numberOfSessionsFailed (integer) --
The number of sessions that failed evaluation.
totalNumberOfSessions (integer) --
The total number of sessions included in the batch evaluation.
numberOfSessionsIgnored (integer) --
The number of sessions that were ignored during evaluation.
evaluatorSummaries (list) --
A list of per-evaluator summary statistics.
(dict) --
Summary statistics for a single evaluator within a batch evaluation.
evaluatorId (string) --
The unique identifier of the evaluator.
evaluationDimensionName (string) --
The name of the dimension that this summary covers. This field is set only for evaluators that use a multi-dimensional rating scale, where each dimension is summarized separately. For single-scale evaluators, this field is not set.
statistics (dict) --
The aggregated statistics for this evaluator.
averageScore (float) --
The average score across all evaluated sessions for this evaluator.
totalEvaluated (integer) --
The total number of sessions evaluated by this evaluator.
totalFailed (integer) --
The total number of sessions that failed evaluation by this evaluator.
errorDetails (list) --
The error details if the batch evaluation encountered failures.
(string) --
kmsKeyArn (string) --
The ARN of the KMS key used to encrypt evaluation data.
updatedAt (datetime) --
The timestamp when the batch evaluation was last updated.
nextToken (string) --
If the total number of results is greater than the maxResults value provided in the request, use this token when making another request in the nextToken field to return the next batch of results.