Evaluate Trace

Use this skill when the user wants to evaluate traces with an existing live evaluation endpoint.

Interactive Q&A protocol (mandatory)

<HARD-GATE> BEFORE the first scoping question, search for a structured question tool (e.g., `AskUserQuestion` or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment. </HARD-GATE>

If context does not make scope clear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

Do you want to evaluate one trace or a batch?
A) One trace now
B) Small batch (up to 25)
C) Full batch loop

Rules:

• Ask exactly one clarifying question per message.
• Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
• Ask a single follow-up if needed, then proceed.

Workflow

1. Identify target live evaluation:
   • If live evaluation id is unknown, call list_live_evaluations.
   • Select public_id and verify required input_columns.
2. Prepare inputs:
   • Ensure inputs keys exactly match input_columns.
   • Include media_url for multimodal evaluations when needed.
3. Execute evaluation:
   • Use the run_eval tool with live_evaluation_id and inputs for each trace.
4. Return useful outputs:
   • run_id
   • per-judgment scores/outcomes
   • brief interpretation for next action
5. Optional handoff:
   • If human judgment is needed, route to review-and-promote-traces.

Batch mode guidance

• Use deterministic trace ordering and log run_id for each input.
• Apply retries with stable idempotency context in caller logic if needed.
• Summarize failures by category or threshold, then propose review handoff.

Scopes reference

• list_live_evaluations requires live-evaluations:read
• run_eval requires live-evaluations:execute

If a scope error occurs, ask the user to create an API key with the missing scope in Truesight Settings.