Assertions

An assertion is one check over a finished run. Each entry under assert: is exactly one kind; a test may have any number. Deterministic assertions read the transcript, the responses, the tool calls, the metrics and the call timeline; judge and evaluator ask a model (see Judges). Every result carries a one-line reason that the console, junit.xml and result.json show, so a failure reads like a code review comment.

The transcript

Responses (rest)

Tool calls

Tool calls come from the target: the REST target’s response.tool_calls path, the LiveKit agent’s attributes and data messages.

Metrics

- metric: { name: turn_latency_ms_p95, lt: 4000 }
- metric: { name: time_to_first_audio_ms, lt: 1500 }

Exactly one comparator (lt, lte, gt, gte, eq). A metric the run did not collect fails with the list of the ones it did. See Metrics.

Phone calls (sip)

These read the call timeline the sip and livekit targets record.

Handoff and escalation

Some conversations must end with a person: the angry customer, the request the agent is not allowed to grant, the user who asks for a human. Others must not. Tell the target how your agent signals a handoff, and assert on it:

targets:
  api:
    type: rest
    url: ${AGENT_URL}
    handoff: { tool: transfer_to_human, to: department, summary: summary }   # or json_path / regex

tests:
  - name: angry customer reaches a person
    target: api
    persona: { name: Ray, style: furious, swears every turn }
    scenario: { goal: "Complain about a late delivery until a human takes over", max_turns: 6 }
    assert:
      - handoff: { within_turns: 3, to: "^support$", summary_contains: ["late"], summary: "The summary says what the customer is angry about" }
      - pack: discloses_ai
  - name: a simple question stays with the agent
    target: api
    scenario: { goal: "Ask how long shipping takes", max_turns: 3 }
    assert:
      - handoff: false

handoff on a target is one of tool (a tool call such as transfer_to_human), json_path (a field of the raw response that is present and not false) or regex (words in the reply); to and summary say where the destination and the summary live (a tool argument name or a JSONPath; with regex, the first group is the destination). When the signal fires the conversation ends with stop reason handoff, a handoff event records the turn, the destination and the summary, and handoff_turn is a metric. A SIP transfer (transferred_to) counts as a handoff to its target.

Pair a handoff test with pack: discloses_ai: an agent that escalates well but claims to be a person when asked still fails.

Your backend’s state

A conversation that sounds right can still have done the wrong thing. These two run after the conversation ends and look at your own systems; their templates see {{vars.x}}, {{captured.x}} (a value captured from the agent’s turns, such as an order id) and {{secrets.X}}.

tests:
  - name: refund goes through
    target: api
    capture:
      - { name: order_id, regex: "(ORD-[0-9]+)" }
    scenario: { goal: "Get a refund for the shoes that arrived damaged", max_turns: 6 }
    assert:
      - judge: "The agent confirms the refund and gives a timeline"
      - http:
          url: "{{vars.admin_url}}/orders/{{captured.order_id}}"
          headers: { Authorization: "Bearer {{secrets.ADMIN_TOKEN}}" }
          json_path: { path: "$.status", equals: refunded }
      - command: { run: "./scripts/check-ledger.sh {{captured.order_id}}", contains: credited }

The result quotes the status and the first lines of the body or output as evidence, so a failing check shows what the backend actually held. Both default to a 30-second timeout; on_turn does not apply, since the state is read once, after the conversation.

Models

Grounding

An agent that cites documents can still invent. grounded asks the judge model to list every factual claim the agent made and whether the context supports it, and passes when all of them (or min of them, a fraction) are supported. The context is what the agent could legitimately draw on:

• tool: search_docs: the results of that tool, on the targets where tamesu sees them (model, mcp, realtime, pipecat);
• json_path: "$.sources[*].text": passages selected from every raw response (rest, websocket, realtime events), for agents that return their sources;
• text: [...]: passages written in the suite.

tests:
  - name: answers come from the knowledge base
    target: prompt
    scenario: { goal: "Ask how long you have to return shoes and whether there is a restocking fee", max_turns: 4 }
    assert:
      - grounded: { context: { tool: search_docs } }
      - tool_called: search_docs

The result lists each claim with a tick or a cross and the passage behind it, and its score is the supported fraction; tamesu judge calibrate and the replay cache treat the grounding check like any other verdict. Greetings, apologies and questions are not claims.

Custom

custom: name runs an assertion registered from Go with assert.Register; the Go API takes assert.Assertion values directly. See Go API.

Packs

One turn only

on_turn: N (1-based) or on_turn: last evaluates the assertion on a single agent turn: its text, its response and its status.

- contains: "anything else"
  on_turn: last
- json_path: { path: $.end_session, equals: true }
  on_turn: last

How a result reads

FAIL   support-agent / refund policy question  2.1s
    x judge "The agent states the 30-day return policy and does not promise an exception": the agent said a 40-day return "should be fine", which promises an exception
    x metric turn_latency_ms_p95 < 4000: turn_latency_ms_p95 = 5120, want < 4000
    last turns:
      user:  So I can still return them?
      agent: Sure, 40 days should be fine, I'll make an exception.

result.json holds the same, plus the judge’s evidence quotes, its trace id and the prompt version.