Reports

Every tamesu run writes a run directory and feeds the reporters named in reporters: (or --reporter): console, json, junit, html, markdown, github, slack and sarif.

The run directory

.tamesu/runs/2026-10-31T14-20-00/
  summary.json
  junit.xml
  tamesu.sarif         (sarif reporter: the red-team findings for a security dashboard)
  report.html          (html reporter, or tamesu report <run-dir>)
  coverage.md          what the run exercised: targets, tags, tools, assertion kinds (also coverage.json; tamesu coverage prints it)
  matrix.md            (tamesu run --matrix: the variants compared; each variant's run directory sits next to it)
  tests/<test-slug>/run-<n>/
    transcript.json      the turns, the tool calls and why the conversation stopped
    result.json          everything about the run: assertions with reasons, metrics, trace ids, events, steps
    audio.wav            (voice audio mode and sip: left the caller, right the agent)
    steps.json           (browser: each step with its outcome, actions and screenshot)
    summary.md           (markdown and github reporters: the run as Markdown)
    comparison.json      (--baseline / tamesu compare: the run against the baseline)
    screenshots/*.png    (browser: one per step, plus screenshot actions)
    recording.webm       (browser: the whole session; recording.gif without ffmpeg)
    trace.har            (browser)
    console.log          (browser: console errors and exceptions)

--out picks another directory. The slug is the test name lowercased with punctuation as dashes; a second test with the same name gets -2.

result.json

{
  "index": 1,
  "passed": false,
  "assertions": [
    { "name": "judge \"...\"", "pass": false, "reason": "the agent promised an exception", "evidence": ["Sure, 40 days should be fine"], "trace_id": "9b1c..." },
    { "name": "metric turn_latency_ms_p95 < 4000", "pass": true, "reason": "turn_latency_ms_p95 = 1240, want < 4000" }
  ],
  "metrics": { "turn_latency_ms": 1100, "turn_latency_ms_p50": 1050, "turn_latency_ms_p95": 1240, "turns": 3, "llm_cost_usd": 0.0021, "llm_tokens": 4120 },
  "transcript": [ { "role": "user", "text": "...", "at": "..." }, { "role": "agent", "text": "...", "index": 1 } ],
  "tool_calls": [ { "Name": "lookup_policy", "Args": { "topic": "returns" } } ],
  "trace_ids": ["3f2a...", "9b1c..."],
  "stop_reason": "simulator: the agent answered the question",
  "usage": { "calls": 3, "input_tokens": 3900, "output_tokens": 220, "cost_usd": 0.0021 }
}

error and error_kind (timeout, infra, budget, cancelled) say why a run did not complete. trace_ids are the Genkit traces of the simulator and judge calls, which the Genkit Developer UI shows when genkit.dev_ui is on (see AI).

console

Live progress on stdout: a header, one PASS/FAIL/ERROR line per test with its duration (and, with repeat, how many runs passed against the threshold), and on failure the failing assertions’ reasons and the last three turns. -v adds a START line per test and a line per run. The footer counts tests, time, model calls, tokens and cost.

tamesu run: 2 tests, output in .tamesu/runs/2026-10-31T14-20-00
PASS   support-agent / refund policy question  2.1s
FAIL   support-agent / reschedule booking by voice  14.8s  (3/5 runs passed, threshold 80%)
    run 2:
      x judge "The agent confirmed the new date": the agent proposed Sunday instead of Saturday
      last turns:
        user:  Saturday, please.
        agent: Done, I moved it to Sunday the 8th.

1 passed, 1 failed, 0 errors in 17.2s · 14 model calls, 21034 tokens, $0.0112

junit

junit.xml, one testsuite per suite and one testcase per test, repeats aggregated: the failure message carries the pass rate or the first failing assertion, the body every failing run’s reasons, and system-out the first failing transcript. An infrastructure error is an error, not a failure. Each testcase carries the suite file and line of its test, so a reporter that annotates sources (dorny/test-reporter, GitLab) points at the test. GitHub Actions, GitLab and Jenkins test reporters read it as is; see CI for the workflow step.

sarif

tamesu.sarif, the run’s security findings as SARIF 2.1.0, the format GitHub code scanning, Azure DevOps and most security dashboards ingest. A finding is a red-team attack that got through (level error, rule tamesu/playbook/<name>, with the attack’s name, the playbook’s OWASP category and the judge’s reasons), an attack that could not run (warning), or a failed check of a built-in red-team pack on any test (error, rule tamesu/pack/<name>). Each result points at the test’s line in its suite file and carries a stable fingerprint, so an alert stays the same alert run after run and closes when the attack holds again. A run without findings writes an empty result set, which closes the open alerts. Plain test failures are not findings; they are in junit.xml. On GitHub, upload the file with github/codeql-action/upload-sarif and the findings appear under Security → Code scanning (see CI).

json

summary.json, the whole invocation: every test with every run (result.json inlined), in load mode a load profile per test (users, runs, runs per second, error rate, turn latency percentiles; see CI: load mode), the totals and the model usage. It is always written, whatever --reporter says, so a script can read one file.

Exit codes

Tests that never ran because --fail-fast stopped the run are reported as skipped (SKIP in the console, a skipped element in JUnit, their own count elsewhere) and do not change the exit code; summary.json carries stopped_early: fail_fast.

A test with a quarantine block never changes the exit code: its failure is reported as quarantined (QUAR in the console, a skipped element in JUnit, a warning annotation in GitHub, its own count in the Markdown, Slack and HTML summaries) and is not a regression in a baseline comparison. See quarantining a flaky test.

Merging shards and repeated runs

tamesu merge turns several run directories into one: the shards of a suite (tamesu run --shard 1/4 to 4/4 in parallel jobs), or repeated runs of the same tests, whose runs are joined into one outcome with its pass rate recomputed. The run directories are copied under the merged one, every report is written again, labels are the union of the inputs, and the exit code is the merged verdict, so a pipeline gets a single result after its shards and can compare it with a baseline or save it as one.

tamesu merge out/shard-1 out/shard-2 out/shard-3 out/shard-4 --out out/all
tamesu compare .tamesu/baseline out/all

History

Every run under .tamesu/runs (the default output directory) adds a History section to its report.html: one row per test with a square per run over the last 20, the pass rate, a p95 latency sparkline, a flaky mark when the verdict changed two or more times, and the current streak. tamesu history prints the same from the terminal:

5 runs, 2026-11-03 09:00 to 2026-11-03 13:00

TEST                       RUNS  PASS RATE  RESULTS  P95             NOTE
support / refund policy    5     100%       ✓✓✓✓✓    820 → 960 ms
support / booking          5     60%        ✓✗✓✗✓    1100 → 1250 ms  flaky (4 flips)
support / cancel order     5     40%        ✓✓✗✗✗                    fail 3× in a row

A test that flips is a quarantine candidate; a p95 that climbs run after run is a regression a single baseline comparison would not catch until it crossed the limit. --flaky lists the candidates, --json feeds a dashboard.

Coverage

Every run writes coverage.md and coverage.json: the targets and tags the tests reached (with how many passed), every tool the agent called (calls and tests, and whether a suite declared it), the assertion kinds used, how the conversations ended and, for adversarial personas, the tactics tried. Tools the suites declare (a model target’s tools, the names in tool_called and tool_sequence assertions) that no test made the agent call are listed as never called: the cheapest way to see that a suite stopped exercising a tool after a prompt change. tamesu coverage <run-dir> prints the same for any run directory.

Comparing with a baseline

A run on its own says whether the tests passed; against a baseline it says whether anything got worse. tamesu baseline save <run-dir> keeps a trusted run’s summary.json (in .tamesu/baseline, or baseline.dir); then tamesu run --baseline compares the new run with it and tamesu compare <baseline-dir> <run-dir> compares any two runs after the fact.

Label the runs so the comparison says which builds it compared: tamesu run --label agent=$GIT_SHA --label env=staging, or labels: in tamesu.yaml. Labels appear in summary.json, in the console and HTML headers, in the Markdown and Slack summaries and next to the baseline’s date in the comparison.

The comparison matches tests by suite and name and reports:

• new failures: tests that passed in the baseline and fail now (a regression by default);
• fixed, added and removed tests;
• per test, the growth of turn_latency_ms_p95, time_to_first_token_ms_p95, time_to_first_audio_ms_p95 (medians over the test’s repeats, so one slow run does not decide) against max_latency_increase_pct (default 20), and of llm_cost_usd against max_cost_increase_pct (default 50); growth under 50 ms or $0.001 is never a regression, whatever the percentage.

baseline:                                   # tamesu.yaml
  dir: .tamesu/baseline
  max_latency_increase_pct: 20
  max_cost_increase_pct: 50
  fail_on_new_failures: true

The verdict is printed after the summary line, written as comparison.json in the run directory and into summary.json, and shown at the top of the HTML report. A regression exits 1 even when every test passed.

html

report.html is one self-contained file (no external assets): the totals, then every test with its runs: the assertions and their reasons and evidence, the transcript (or the browser steps), the metrics, the event timeline, a turn-latency histogram for a test run in load mode (with p50 and p95 marked), the session recording and a strip of step screenshots for browser tests, an audio player for audio.wav with a button per caller and agent utterance (and DTMF) that seeks to that moment, the artifacts and the Genkit trace ids. Artifacts are linked relative to the file, so the run directory can be zipped and shared. tamesu report <run-dir> rebuilds it from summary.json (for example on a CI artifact).

markdown

summary.md in the run directory: the headline, the table of tests and a collapsed list of the failures with their reasons, in GitHub-flavoured Markdown. Post it as it is:

tamesu run --reporter console,markdown --out .tamesu/runs/pr
gh pr comment "$PR" --body-file .tamesu/runs/pr/summary.md

github

For GitHub Actions. It writes a workflow annotation per failed or errored test, pointing at the test’s line in its suite file (::error file=tests/support.yaml,line=30,title=tamesu: refund policy::the reason), so failures show inline on the pull request’s files; it appends the Markdown summary to the job summary (GITHUB_STEP_SUMMARY) and writes summary.md in the run directory. Nothing to configure: tamesu run --reporter console,github.

slack

Posts one message to a Slack incoming webhook when the run ends: the headline, the baseline verdict when there is one, and the failures with their reasons (ten at most).

reporters: [console, slack]
slack:
  webhook: "{{secrets.SLACK_WEBHOOK}}"           # declare SLACK_WEBHOOK under secrets:
  only_failures: true                            # stay quiet when everything passed
  link: "${CI_JOB_URL:-}"                        # put at the end of the message
secrets:
  SLACK_WEBHOOK: env:SLACK_WEBHOOK

A webhook that answers an error fails the run with an infrastructure error, so a broken notification is noticed.