CI

tamesu is built for pipelines: deterministic exit codes, file:line errors, secrets that come from your CI’s secret store through mamori rather than from the environment, JUnit output your CI’s test reporter reads, and a Docker image with Chromium preinstalled. This page shows how to validate, check and run suites in GitHub Actions, GitLab CI and Docker.

GitHub Actions

name: agent tests
on: [push, pull_request]
jobs:
  tamesu:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-go@v7
        with:
          go-version: "1.27"
      - run: go install github.com/xavidop/tamesu/cmd/tamesu@latest
      - run: tamesu validate --strict
      - run: tamesu doctor --var ENV=ci
        env:
          SUPPORT_API_TOKEN: ${{ secrets.SUPPORT_API_TOKEN }}
          AWS_REGION: eu-west-1
      - run: tamesu run --var ENV=ci --out out
        env:
          SUPPORT_API_TOKEN: ${{ secrets.SUPPORT_API_TOKEN }}
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
      - uses: dorny/test-reporter@v2
        if: always()
        with:
          name: agent tests
          path: out/junit.xml
          reporter: java-junit

validate --strict fails the job on any warning, including a ${VAR} used for a secret. doctor resolves every declared secret: with a chain such as env:SUPPORT_API_TOKEN,aws-sm://${ENV}/support-agent#api_token, the job above takes the GitHub secret, and a workflow without it falls through to AWS Secrets Manager for the ci environment (--var ENV=ci), using the runner’s AWS credentials.

Output in a job is not a terminal, so it is plain by default; GitHub Actions renders ANSI colours, so tamesu run --color always (or FORCE_COLOR=1 in the job’s environment) brings the green and red verdicts back in the log.

GitLab CI

tamesu:
  image: ghcr.io/xavidop/tamesu:latest
  script:
    - tamesu validate --strict
    - tamesu doctor --var ENV=ci
  variables:
    SUPPORT_API_TOKEN: $SUPPORT_API_TOKEN

The image’s working directory is /work; GitLab mounts the checkout there when image: is the job’s image.

Splitting a suite across jobs

A long suite runs in parallel jobs with --shard i/n: each job runs the i-th of n equal slices of the selected tests, dealt in suite and test order, so the n jobs cover every test exactly once and the split does not change between runs unless the suite does. Each shard writes its own run directory and junit.xml; the run is labelled shard=i/n, and a shard that ends up with no tests exits 0.

jobs:
  tamesu:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v7
      - run: go install github.com/xavidop/tamesu/cmd/tamesu@latest
      - run: tamesu run --shard ${{ matrix.shard }}/4 --out out
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: junit-${{ matrix.shard }}
          path: out/junit.xml

tamesu list --shard 2/4 prints the tests a shard would run. A final job downloads the shards’ run directories and merges them into one result, which is what you compare with the baseline:

  verdict:
    needs: tamesu
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v8
        with:
          pattern: run-*
          path: runs
      - run: tamesu merge runs/* --out merged && tamesu compare .tamesu/baseline merged

(upload each shard’s whole --out directory as run-${{ matrix.shard }} for that to work).

For a quick local loop, tamesu run --fail-fast stops starting new tests after the first failure; the tests that never ran are reported as skipped and the exit code is still 1.

Docker

docker run --rm -v "$PWD:/work" -e SUPPORT_API_TOKEN ghcr.io/xavidop/tamesu:latest validate --strict
docker run --rm -v "$PWD:/work" --env-file .env ghcr.io/xavidop/tamesu:latest doctor

ghcr.io/xavidop/tamesu ships the pure-Go CLI with Chromium preinstalled. Pass secrets as environment variables or a dotenv file; the image never bakes them in.

Secrets in CI

Declare secrets as mamori refs and let the pipeline’s identity do the rest:

A chain env:NAME,aws-sm://... lets developers use a local variable and CI the cloud store. See Configuration: secrets.

Exit codes

A known flaky test should not turn the pipeline red while someone works on it: give it a quarantine with the ticket and an until date, and its failure is reported but does not fail the run (see concepts).

Reports

On GitHub Actions add the github reporter: every failed test becomes an annotation on its line in the suite file and the run’s table lands in the job summary; markdown writes the same table as summary.md for a pull request comment, and slack posts to a webhook. See Reports.

- run: tamesu run --reporter console,github,junit

Every run writes a run directory with summary.json, junit.xml for the CI’s test reporter and, per test and run, transcript.json and result.json with the assertions, metrics and judge reasoning. Upload the run directory as an artifact to read a failure after the fact. See Reports.

The security tab

The sarif reporter writes the red-team findings of a run as tamesu.sarif: every attack that got through, every attack that could not run, and every failed check of a built-in red-team pack. Upload it with the code-scanning action and the findings appear under Security → Code scanning of the repository, one alert per attack, pointing at the test’s line in the suite, with the playbook’s OWASP category as a tag. An alert closes on the first run where the attack holds again.

permissions:
  contents: read
  security-events: write
steps:
  - run: tamesu run --tags red-team --reporter console,junit,sarif --out out
    env:
      GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
  - uses: github/codeql-action/upload-sarif@v3
    if: always()
    with:
      sarif_file: out/tamesu.sarif
      category: tamesu-red-team

category keeps tamesu’s alerts apart from other scanners’ and lets several workflows (one per environment, one per agent) upload without closing each other’s alerts.

Load mode

tamesu run --load N runs every selected test with N concurrent virtual users, each looping runs for --load-duration (one run each without it), so a conversational agent can be watched under modest concurrency with the same suites and assertions: pass/fail still comes from the assertions and pass_threshold over all the runs, and the target caps (livekit.max_concurrent_rooms, sip.max_concurrent_calls) still hold, so a phone suite cannot place more calls than the project allows.

tamesu run --tags smoke --load 10 --load-duration 2m --max-cost-usd 5

Each test’s entry in summary.json gets a load profile: users, runs, duration_ms, runs_per_second, error_rate (runs that did not pass) and turn_latency_ms (count, p50, p95, max over every agent turn of every run, from transport timestamps). The console prints the same line under the test and the HTML report shows it next to the runs. Load mode is a smoke-level tool: the simulator and the judges cost model calls per run, so set a budget.

Budgets

budgets.max_cost_usd (or --max-cost-usd) caps what a pipeline run may spend on models and exits 4 when reached, so a looping agent cannot run up a bill overnight.

Running without model keys

With the caches committed (cache.dir, default .tamesu/cache), tamesu run --replay plays the simulated user from cached turns and the browser agent from cached action plans, so a pull request pipeline needs no model key for the conversations that were recorded; see Replay. Record on a trusted branch with the keys and refresh with --update-cache when the scenarios change.

Gating on a baseline

Save a green run from your main branch as the baseline (tamesu baseline save <run-dir>, kept as an artifact or committed) and run pull requests with tamesu run --baseline: a test that starts failing, or a p95 latency or cost that grows beyond the baseline: limits, fails the job with exit 1 even when every assertion passed. See Comparing with a baseline.