Performs on-demand evaluation of agent traces using a specified evaluator¶

Description¶

Performs on-demand evaluation of agent traces using a specified evaluator. This synchronous API accepts traces in OpenTelemetry format and returns immediate scoring results with detailed explanations.

Usage¶

bedrockagentcore_evaluate(evaluatorId, evaluationInput,
  evaluationTarget, evaluationReferenceInputs)

Arguments¶

• evaluatorId

  [required] The unique identifier of the evaluator to use for scoring. Can be a built-in evaluator (e.g., Builtin.Helpfulness, Builtin.Correctness) or a custom evaluator Id created through the control plane API.

• evaluationInput

  [required] The input data containing agent session spans to be evaluated. Includes a list of spans in OpenTelemetry format from supported frameworks like Strands (AgentCore Runtime) or LangGraph with OpenInference instrumentation.

• evaluationTarget

  The specific trace or span IDs to evaluate within the provided input. Allows targeting evaluation at different levels: individual tool calls, single request-response interactions (traces), or entire conversation sessions.

• evaluationReferenceInputs

  Ground truth data to compare against agent responses during evaluation. Allows to provide expected responses, assertions, and expected tool trajectories at different evaluation levels. Session-level reference inputs apply to the entire conversation, while trace-level reference inputs target specific request-response interactions identified by trace ID.

Value¶

A list with the following syntax:

list(
  evaluationResults = list(
    list(
      evaluatorArn = "string",
      evaluatorId = "string",
      evaluatorName = "string",
      explanation = "string",
      context = list(
        spanContext = list(
          sessionId = "string",
          traceId = "string",
          spanId = "string"
        )
      ),
      value = 123.0,
      label = "string",
      tokenUsage = list(
        inputTokens = 123,
        outputTokens = 123,
        totalTokens = 123
      ),
      errorMessage = "string",
      errorCode = "string",
      ignoredReferenceInputFields = list(
        "string"
      )
    )
  )
)

Request syntax¶

svc$evaluate(
  evaluatorId = "string",
  evaluationInput = list(
    sessionSpans = list(
      list()
    )
  ),
  evaluationTarget = list(
    spanIds = list(
      "string"
    ),
    traceIds = list(
      "string"
    )
  ),
  evaluationReferenceInputs = list(
    list(
      context = list(
        spanContext = list(
          sessionId = "string",
          traceId = "string",
          spanId = "string"
        )
      ),
      expectedResponse = list(
        text = "string"
      ),
      assertions = list(
        list(
          text = "string"
        )
      ),
      expectedTrajectory = list(
        toolNames = list(
          "string"
        )
      )
    )
  )
)