Concepts

tamesu has a small vocabulary that is the same for every kind of target. This page defines it, then shows how a conversational test flows and how a result is graded.

Vocabulary

Targets

A target is declared once under targets: and named by tests. The type decides which keys apply; tamesu validate warns when a key belongs to another type.

• browser: a URL, browser options (engine, viewport, locale), an optional pre-authenticated storage state and allowed_domains. Tests are a list of steps: goal (the browser agent plans and acts until the goal is met), expect (an agentic assertion against the page) and action (deterministic: navigate, click, fill, press, wait, screenshot, eval_js, assert_url, assert_text, mock, a11y, visual).
• rest: a URL and method, headers, how a user turn is sent (request.body, with {{session}} and {{message}}), how the agent turn is read back (JSONPaths in response.text, response.tool_calls, response.done), streaming: none|sse|websocket and a session id template. When the reply arrives somewhere else, session.open creates the conversation, response.poll reads the reply from a second endpoint and response.listen reads it from the agent’s own calls to an API tamesu serves, which is how a Slack, Teams, Telegram or WhatsApp bot is tested.
• livekit: the LiveKit server URL, api_key/api_secret (as {{secrets.X}}), a room template, an optional agent dispatch, how to recognise the agent participant, mode: text|audio and the text-mode data topic.
• sip: direction: outbound (tamesu calls the agent at to, presenting from) or inbound (a trigger makes the agent call sip.inbound_number), the trunk, ring timeout and maximum call duration. A SIP call is bridged into a LiveKit room, so everything audio mode does applies.
• realtime: a websocket URL speaking the OpenAI Realtime protocol (the API or your own server), headers, model, a session_update merged into the session, tool_results that answer the agent’s function calls, done_tool, mode: text|audio and audio_format.
• grpc: grpc://host:port of an agent implementing the turn API, with headers as call metadata and turn_timeout.
• pipecat: the websocket URL of a Pipecat bot’s transport, mode: text|audio, sample_rate, idle, tool_results for client-side function calls.
• twilio: the stream URL of a Twilio Media Streams agent, with from, to and parameters as the custom parameters the agent sees; tamesu plays Twilio’s side of the socket.
• model: a system prompt (or system_file) and tools with their result, run directly against a model (with fallback, temperature, max_tokens, max_tool_rounds), no agent server.
• mcp: an MCP server, command (with args and env) over stdio or url (with headers) over streamable HTTP; its tools are offered to a driving model with a short system, narrowed by allow_tools, each call bounded by call_timeout.
• a2a: an agent speaking the Agent2Agent protocol at url over any of its bindings (transport: jsonrpc, http or grpc; the agent card is read from the URL’s origin or with GetAgentCard), with headers, stream: true to read the turn from message/stream, push: true to receive task updates as push notifications, a skill the card (or the authenticated extended card) must offer and metadata sent on every message. Each user turn is one message, with a script step’s data and files as parts; a task that asks for input is continued, a completed one is followed by a new task in the same context, and a failed, canceled or rejected one ends the session.
• voice: any websocket that carries audio: url (with the {{session.NAME}} captures of a session.open request that creates the call), audio (encoding: pcm16|mulaw|alaw, sample_rate, frames: binary|json with send and receive) and messages (the protocol’s JSON control messages: open, transcript, done, interrupted, tool_calls, dtmf, close). Hosted voice platforms such as Vapi or Deepgram Voice Agent are recipes on it.

Any conversational target may also declare connect (how reaching the agent is retried), handoff (how the agent signals that a person took over) and session.open (a request made once per run before the target starts, whose captured values render as {{session.NAME}} in the target’s configuration: create a conversation, mint a token). The full key list is in Suites.

Personas and scenarios

A conversational test (rest, grpc, livekit, sip, realtime, pipecat, voice, model, mcp, a2a) is driven by a persona and a scenario, or by a counterpart: another target of the suite that plays the user, so two agents talk (see A2A). A suite’s remote agents (remote_agents) are A2A agents tamesu runs for the agent under test to delegate to; what it sends them is recorded for delegated_to.

persona:
  name: Alice
  style: impatient
  language: en-GB
  voice: en-GB-female                 # TTS voice for audio mode
  behaviors:
    interrupt_probability: 0.3        # talk over the agent sometimes
    silence_after_turn_ms: [300, 1500]
    background_noise: cafe
scenario:
  goal: "You booked a room for Friday; move it to Saturday"
  max_turns: 12
  stop_when: "the agent confirmed the new date"
  script:                             # sip and livekit audio: deterministic steps mixed in
    - wait_for: "press 2 for billing"
    - dtmf: "2"
    - say: "Hi, my last invoice is higher than usual"

The simulated user, given the persona, the scenario and the conversation so far, produces the next utterance (or a DTMF digit, or decides to hang up or stop). Its model is models.simulator; a seed per test keeps it as repeatable as the provider allows.

Suite-wide assertions and SLOs

Rules that every test must meet live once, under defaults in tamesu.yaml or in a suite, instead of in every test. defaults.assert is a list of assertions (packs included) that runs before each test’s own, reported as defaults/<assertion>; defaults.slo caps metrics by name, reported as slo/<metric>, and a metric a test does not measure, such as an audio metric on a text test, passes. A suite’s entries run after the project’s and its SLOs override them.

defaults:                                   # tamesu.yaml, or a suite
  assert:
    - pack: pii_leakage
    - not_contains: "as an AI language model"
  slo:
    turn_latency_ms_p95: 3000
    time_to_first_token_ms_p95: 1500
    llm_cost_usd: 0.05

Hooks

hooks in the project or the suite run commands or HTTP requests before and after the whole suite (before_all, after_all) and around every run (before_each, after_each): seeding, resets and cleanup live next to the tests. See Configuration.

Parameterised tests

A test with cases runs once per case, each case giving values to {{vars.<key>}} in the goal, the stop condition, the target’s request or anywhere else a template is accepted. Each case becomes a test of its own, named <name> [<case>] and tagged case:<name>, so --filter, --tags, repeats and reports treat it like any other test.

tests:
  - name: return window
    target: api
    vars: { lang: en }                          # this test's values, under the cases
    cases:
      - { name: late, vars: { days: "40", expect: "not possible" } }
      - { name: early, vars: { days: "10", expect: "can return" } }
    persona: { name: Marta, language: "{{vars.lang}}" }
    scenario: { goal: "Ask whether shoes bought {{vars.days}} days ago can be returned" }
    assert:
      - contains: "{{vars.expect}}"

cases_file: cases.csv (a header row; a name column names the case) or cases_file: cases.json (an array of objects; a name key names the case) reads the cases from a file next to the suite, so a spreadsheet of questions becomes a suite.

Capturing values across turns

capture takes values from the agent’s turns as they arrive, by a regular expression on the agent’s text (the first group, else the whole match), a JSONPath on the turn’s raw response, or extract: a plain-language description of the value (the order number the agent mentioned) that a model takes out of the turn as written, for values with no fixed shape. The first turn that has the value sets it; from then on it is {{captured.<name>}} in later script say steps, in tool_results, in hooks and in assertions, and it is written to result.json.

tests:
  - name: order status
    target: api
    capture:
      - { name: order_id, regex: "order (ORD-[0-9]+)" }
      - { name: eta, json_path: "$.eta" }
      - { name: carrier, extract: "the delivery company the agent named" }
    scenario:
      goal: "Place an order for the blue sneakers, then ask when it arrives"
      script:
        - say: "I'd like the blue sneakers in a 42"
        - say: "What is the status of order {{captured.order_id}}?"
    assert:
      - captured: { name: order_id, matches: "^ORD-" }
      - contains: "{{captured.eta}}"

extract runs on the judge model (one short call per agent turn until the value is found), is cached by the replay cache like a verdict, and counts in the run’s model usage. Prefer a regex or json_path when the value has a shape; extract is for the rest: a carrier name, a reason, a date the agent words differently every time.

How a conversational test flows

1. tamesu opens a session on the target.
2. The persona loop runs: the simulator produces the next user message from persona, scenario and history; the target sends it and waits for the agent’s turn; the turn is appended to the transcript; stop_when, max_turns and the agent’s own end-of-session are checked.
3. Metrics are collected throughout. Timestamps come from the transport layer, never from a model.
4. The session closes and the artifacts are saved under the run directory.
5. Assertions are evaluated. Judges get the transcript, the tool calls and the metrics as structured input and answer with pass, score, reason and evidence.
6. The result is aggregated across repeat runs and compared with pass_threshold.

Assertions

Every assertion is one key under assert:; a test may have any number, and on_turn: N or on_turn: last scopes one to a single agent turn.

Judges return structured output only (pass, score, reason, evidence[]), run at temperature 0 when the provider supports it, and their prompt version is recorded with every result so judge drift is traceable.

Metrics

Metric names follow the pattern name or name_p95.

Runs, repeats and exit codes

A test runs repeat times (default 1); the fraction of runs that must pass is pass_threshold (default 1.0). retries re-runs a test whose infrastructure failed, never one whose assertions failed; before it comes to that, every target retries reaching the agent on its own. Every run gets its own directory with transcript.json, result.json and the artifacts of its target.

Quarantining a flaky test

A test that fails for reasons you already know about can be quarantined instead of deleted or skipped. It still runs on every invocation and its result is still written and reported, but a failure neither fails the run nor counts as a regression against the baseline:

tests:
  - name: refund status
    target: api
    quarantine: { reason: "SUP-412 flaky since the payments migration", until: 2026-12-31 }
    scenario: { goal: "Ask where the refund for order 1234 is", max_turns: 3 }
    assert:
      - judge: "The agent gives a refund date or asks for the order"

reason is required, so a quarantine always points at the ticket that tracks it. until is the last day of the quarantine; from the next day the test counts again, and tamesu validate warns that it expired. Without until, the quarantine lasts until you remove it. A quarantined test that passes just passes. The console prints QUAR instead of FAIL, the summary line and every report count quarantined tests apart from failures, the JUnit file reports them as skipped with the failure in the message, and the GitHub reporter annotates them as warnings.