Suites: the YAML reference

A suite is one YAML file with the targets under test, the suite’s defaults and its tests. This page lists every key, always in step with what tamesu validate accepts. The JSON Schema for editors is published at schema/tamesu.schema.json; point your editor’s YAML extension at it for completion and inline errors. See Concepts for the model and Configuration for tamesu.yaml.

Strings accept ${VAR} and ${VAR:-default} (environment, non-secret values) and the templates {{uuid}}, {{now}}, {{env.X}}, {{vars.x}}, {{secrets.X}} and {{faker.email}}; REST bodies also take {{session}} and {{message}}. Durations are strings such as 30s, 3m or 1h30m.

Suite

The top level of the file.

Example:

suite: support
tags: [smoke]
vars: { base_url: "https://api.example.com" }
secrets: { API_TOKEN: env:API_TOKEN }
targets:
  api:
    type: rest
    url: "{{vars.base_url}}/chat"
    headers: { Authorization: "Bearer {{secrets.API_TOKEN}}" }
    response: { text: "$.reply" }
tests:
  - name: refund policy
    target: api
    persona: { name: Marta, style: polite }
    scenario: { goal: "Find out if shoes bought 40 days ago can be returned", max_turns: 4 }
    assert:
      - judge: "The agent states the 30-day return policy"
      - contains: "30 days"

Target

Example:

targets:
  web:  { type: browser, url: "https://shop.example.com", browser: { viewport: 1280x800 } }
  api:  { type: rest, url: "https://api.example.com/chat", response: { text: "$.reply", done: "$.end_session" } }
  grpc: { type: grpc, url: "grpc://agent.internal:9090" }
  phone:
    type: twilio
    url: "${AGENT_STREAM_URL:-wss://agent.example.com/media}"
    parameters: { caller_tier: gold }
    from: "+15550100"
  prompt:
    type: model
    model: googleai/gemini-2.5-flash
    system_file: prompts/support.md
    tools:
      - name: lookup_policy
        description: The policy for a topic
        parameters: { type: object, properties: { topic: { type: string } } }
        result: { topic: "{{args.topic}}", window_days: 30 }
  voice: { type: livekit, url: "${LIVEKIT_URL}", api_key: "{{secrets.LIVEKIT_API_KEY}}", api_secret: "{{secrets.LIVEKIT_API_SECRET}}", mode: audio }
  phone: { type: sip, direction: outbound, to: "+15550001", from: "+15550002" }
  rt:   { type: realtime, model: gpt-realtime, headers: { Authorization: "Bearer {{secrets.OPENAI_API_KEY}}" }, mode: text }
  bot:  { type: pipecat, url: "wss://bot.example.com/ws", mode: audio }

ToolDef

Example:

tools:
  - name: lookup_policy
    description: The return policy for a topic
    parameters: { type: object, properties: { topic: { type: string } }, required: [topic] }
    result: { topic: "{{args.topic}}", window_days: 30 }
  - name: end_session
    description: Call when the customer has no more questions

BrowserOptions

Example:

browser:
  viewport: 1280x800
  locale: en-US
  recording: webm
  mocks:
    - { url: "**/api/recommendations", body: { items: [] } }

BrowserMock

Example:

mocks:
  - { url: "**/api/payment", method: POST, status: 502, delay_ms: 2000 }
  - { url: "**/analytics.js", abort: true }

BrowserAuth

Example:

auth: { storage_state: auth/customer.json }

RESTRequest

Example:

request:
  body: { session_id: "{{session}}", message: "{{message}}" }
  open: { type: hello, session: "{{session}}" }        # websocket only

RESTResponse

Example:

response:
  text: "$.reply"
  tool_calls: "$.tool_calls[*]"
  done: "$.end_session"
  turn_done: "$.done"                                    # websocket: the message that ends a turn
  idle: 2s

RESTSession

Example:

session: { id: "{{uuid}}" }

Dispatch

Example:

dispatch: { agent_name: support-agent }

Trigger

Example:

trigger:
  http: { url: "https://agent.example.com/callbacks", method: POST, body: { number: "{{sip.inbound_number}}" } }

HTTPTrigger

Example:

http: { url: "https://agent.example.com/dial", method: POST, headers: { Authorization: "Bearer {{secrets.AGENT_TOKEN}}" } }

Test

Example:

tests:
  - name: return window
    target: api
    tags: [returns]
    repeat: 3
    pass_threshold: 0.67
    vars: { days: "40" }
    capture:
      - { name: order_id, regex: "order (ORD-[0-9]+)" }
    persona: { name: Marta, style: polite, language: en }
    scenario: { goal: "Ask whether shoes bought {{vars.days}} days ago can be returned", max_turns: 4 }
    assert:
      - judge: "The agent states the 30-day return policy"
      - tool_called: lookup_policy
  - name: refund status
    target: api
    quarantine: { reason: "SUP-412 flaky since the payments migration", until: 2026-12-31 }
    scenario: { goal: "Ask where the refund for order 1234 is", max_turns: 3 }
    assert:
      - judge: "The agent gives a refund date or asks for the order"

Case

Example:

cases:
  - { name: late, vars: { days: "40", expect: "not possible" } }
  - { name: early, vars: { days: "10", expect: "can return" } }

Capture

Example:

capture:
  - { name: order_id, regex: "order (ORD-[0-9]+)" }
  - { name: eta, json_path: "$.eta" }

Step

Example:

steps:
  - goal: "Log in as {{vars.user}} with password {{secrets.PASSWORD}}"
  - expect: "The account page shows the user's name"
  - action: { click: "#add-to-cart" }
  - action: { a11y: { impact: serious } }
  - action: { visual: { name: cart, mask: ["#date"] } }

Persona

Example:

persona:
  name: Marta
  style: "polite, slightly confused"
  language: es-ES
  voice: es-ES-female
  behaviors: { interrupt_probability: 0.2, background_noise: cafe }

Behaviors

Example:

behaviors:
  interrupt_probability: 0.2
  silence_after_turn_ms: [300, 900]
  background_noise: street
  background_noise_snr_db: 15
  line_quality: mobile_poor

Scenario

Example:

scenario:
  goal: "Reach the billing department and ask about a double charge"
  max_turns: 6
  stop_when: "the agent has confirmed a refund or clearly refused one"
  script:
    - wait_for: "press 2 for billing"
    - dtmf: "2"
    - say: "I was charged twice for order {{captured.order_id}}"

ScriptStep

Example:

script:
  - wait_for: "press 2 for billing"
  - dtmf: "2"
  - say: "Hello, I was charged twice"

Assertion

Example:

assert:
  - judge: "The agent states the 30-day return policy and does not promise an exception"
  - judge: { rubric: "The tone stays warm", scale: "1-5", min: 4 }
  - contains: "30 days"
  - not_regex: "[0-9]{16}"
  - tool_called: { name: lookup_policy, args: { topic: returns } }
  - tool_sequence: { calls: [lookup_policy, end_session] }
  - metric: { name: turn_latency_ms_p95, lt: 3000 }
  - turns: { lte: 6 }
  - captured: { name: order_id, matches: "^ORD-" }
  - golden: { file: golden/refund.json, mode: similar, min: 0.8 }
  - handoff: { within_turns: 3, to: "^billing$" }
  - grounded: { context: { tool: search_docs } }
  - http: { url: "{{vars.admin_url}}/orders/{{captured.order_id}}", json_path: { path: "$.status", equals: refunded } }
  - command: { run: "./scripts/check-refund.sh {{captured.order_id}}", contains: refunded }
  - pack: pii_leakage
  - { contains: "goodbye", on_turn: last }

Judge

Either a string (The rubric; the judge model and a binary scale are the defaults), or an object with:

Example:

judge: { rubric: "The agent apologises before explaining the policy", scale: "1-5", min: 4, model: anthropic/claude-sonnet-4-6 }

ToolCall

Either a string (The tool name), or an object with:

Example:

tool_called: { name: lookup_policy, args: { topic: returns } }

ToolSequence

Example:

tool_sequence: { calls: [lookup_policy, issue_refund], exact: false }

Evaluator

Either a string (The evaluator name), or an object with:

Example:

evaluator: { name: genkitEval/faithfulness, min: 0.8 }

JSONPathAssertion

Example:

json_path: { path: "$.status", equals: ok }

DTMFAssertion

Example:

dtmf_received: { equals: "9" }

Matches

Example:

transferred_to: { matches: "^sip:billing@" }

MetricAssertion

Example:

metric: { name: time_to_first_audio_ms_p95, lt: 1500 }

IntCompare

Example:

turns: { lte: 6 }

CapturedAssertion

Example:

captured: { name: order_id, matches: "^ORD-[0-9]+$" }

GoldenAssertion

Example:

golden: { file: golden/refund.json, mode: judge }

OnTurn

Evaluate the assertion on one agent turn: a 1-based number or last.

One of: integer, one of last.

Example:

- { contains: "goodbye", on_turn: last }
- { json_path: { path: "$.end_session", equals: true }, on_turn: 3 }

StringList

One of: string, list of string.

Example:

call_ended_by: [agent, caller]

Hooks

Example:

hooks:
  before_all: [ { command: ./scripts/seed.sh } ]
  before_each: [ { http: { url: "{{vars.admin_url}}/reset", method: POST } } ]
  after_each: [ { command: "./scripts/cleanup.sh $TAMESU_SESSION", timeout: 30s } ]

Hook

Example:

- command: ./scripts/seed.sh
- http: { url: "https://admin.example.com/reset", method: POST, headers: { Authorization: "Bearer {{secrets.ADMIN_TOKEN}}" } }

Defaults

Example:

defaults:
  timeout: 3m
  repeat: 1
  retries: 1
  pass_threshold: 1
  assert:
    - pack: pii_leakage
  slo:
    turn_latency_ms_p95: 3000
    llm_cost_usd: 0.05

Models

Example:

models:
  default: googleai/gemini-2.5-flash
  judge: anthropic/claude-sonnet-4-6
  simulator: openai/gpt-4.1-mini
  fallback:
    default: [openai/gpt-4.1-mini]
    judge: [googleai/gemini-2.5-pro]
  retry: { attempts: 3, backoff: 500ms, max_backoff: 10s }

Budgets

Example:

budgets: { max_cost_usd: 2, max_tokens: 500000 }

Duration

A duration such as 30s, 3m or 1h30m.

Type: string.

Example:

timeout: 90s

ConnectConfig

Example:

connect: { attempts: 5, backoff: 1s, max_backoff: 10s }

Counterpart

Delegation

Either a string (The remote agent’s name), or an object with:

Fuzz

Either a string (The fuzz mode: error, empty, null, malformed, hostile, huge, wrong_type or delay), or an object with:

Example:

fuzz_tools:
  lookup_order: error                      # the mode alone
  get_balance: { mode: delay, delay: 8s }  # the normal answer, late
  refund: { mode: custom, result: { ok: false, reason: "insufficient funds" } }

GroundedAssertion

Example:

grounded: { context: { tool: search_docs }, min: 0.9 }

GroundedContext

Example:

context: { json_path: "$.sources[*].text" }

HandoffAssertion

Either a string (Whether the agent must have handed the conversation to a person), or an object with:

Example:

handoff: { within_turns: 3, to: "^billing$", summary_contains: ["ORD-"], summary: "The summary names the order and the problem" }

HandoffSignal

Example:

handoff: { tool: transfer_to_human, to: department, summary: summary }

HandshakeStep

ModelFallback

Example:

fallback:
  default: [openai/gpt-4.1-mini]
  judge: [googleai/gemini-2.5-pro]
  simulator: [anthropic/claude-sonnet-4-6]

ModelRetry

Example:

retry: { attempts: 3, backoff: 500ms, max_backoff: 10s }

Push

Either a string (true listens on a free local port), or an object with:

Quarantine

Example:

quarantine: { reason: "SUP-412 flaky since the payments migration", until: 2026-12-31 }

RESTListen

RESTOpen

RESTPoll

RedTeam

Example:

red_team:
  target: api
  agent: "Mochi Shoes support: never more than 30-day returns, no other customers' data, no refunds without verification"
  playbooks: [prompt_injection, pii_extraction, policy_drift]
  max_turns: 8
  assert:
    - not_contains: "90 days"

RemoteAgent

RemoteExtended

RemoteReply

RemoteSkill

Either a string (The skill id), or an object with:

ScriptFile

Either a string (A path relative to the suite, or a URI), or an object with:

Similar

Either a string (The text some agent turn should mean; the configured embedding model and a 0.8 minimum are the defaults), or an object with:

Example:

- similar_to: "The return window is 30 days from delivery"
- similar_to: { text: "Returning after 40 days is not possible", min: 0.7, model: googleai/text-embedding-004 }

StateCommand

Example:

command: { run: "psql -tAc \"select status from orders where id = '{{captured.order_id}}'\"", contains: refunded }

StateHTTP

Example:

http: { url: "{{vars.admin_url}}/orders/{{captured.order_id}}", headers: { Authorization: "Bearer {{secrets.ADMIN_TOKEN}}" }, json_path: { path: "$.status", equals: refunded } }

VoiceAudio

VoiceMessages