tamesu

A tamesu seal character shaking hands with a chat robot over a ticked checklist

tamesu (試す, “to try, to test”) is an open-source, agentic testing framework and CLI written in Go. You describe a test the way you would brief a colleague: a persona, a goal, and what a good outcome looks like. A simulated user then talks to your agent or drives your web app, and deterministic checks and LLM judges grade the result. Your agents, tested like users would.

tamesu init, validate, list and doctor in a terminal

Quick start

brew install xavidop/tap/tamesu        # macOS; Windows: scoop install tamesu; Linux: a .deb, .rpm or .apk
tamesu init                            # tamesu.yaml + examples/support-agent.yaml
tamesu validate                        # file:line errors for anything wrong
tamesu doctor                          # resolve every secret through mamori and check the setup
tamesu run                             # run the suites: transcripts, metrics, junit.xml, exit code

The Quick start walks through a first project.

What you can do

Task	Where to read
Write a suite: targets, personas, scenarios, steps and assertions in YAML	Suites
Start from a complete, working project	Recipes
Describe a REST agent, a LiveKit voice agent, a phone agent or a web app as a target	Suites: Target
Grade a conversation with an LLM judge and deterministic checks	Concepts: assertions
Keep API keys out of suites with mamori secret refs	Configuration: secrets
Validate suites with file and line numbers, in your editor and in CI	`tamesu validate`
Check the whole setup, secrets included, without printing a value	`tamesu doctor`
Run the suites and read the reports	`tamesu run`, Reports
Test a REST agent	REST target
Test an agent over gRPC	gRPC target
Test a Pipecat bot	Pipecat target
Test a LiveKit voice agent, in text or audio	LiveKit target
Pick models, override prompts, see traces, count cost	AI (Genkit)
Drive tests from `go test`	Go API
Run tamesu in GitHub Actions, GitLab CI or Docker	CI

Seven targets, one model

Target	What it is	Transport
`browser`	A web app, driven by an agent from natural-language goals	Chromium over CDP (chromedp)
`rest`	An HTTP agent that answers one user turn per request	HTTP, SSE or WebSocket
`livekit`	A realtime voice agent in a LiveKit room, in text or audio	WebRTC
`sip`	A phone agent reached through a LiveKit SIP service and your trunk	SIP / PSTN
`realtime`	An agent speaking the OpenAI Realtime protocol, in text or audio	WebSocket
`grpc`	An agent behind tamesu’s turn API, a small gRPC service you implement	gRPC
`pipecat`	A Pipecat bot through its websocket transport, in text or audio	WebSocket

All seven share the suite schema, the assertion model, the persona and scenario definitions, the metrics and the report format, and all seven work with the action cache for browser goals, the recorder, the generator, the HTML report, the red-team packs, load mode and the MCP server for coding agents.

All AI through Genkit, all secrets through mamori

Every model call (the simulated user, the judges, scenario generation, speech) is a Genkit flow. Models are Genkit references such as googleai/gemini-2.5-flash or anthropic/claude-sonnet-4-6, so switching provider is a one-line change, and every run is traceable in the Genkit Developer UI.
Every secret is a mamori ref (env:, aws-sm://, vault://, op://, …), resolved once per run, revealed only at the moment of use and replaced by ‹secret:NAME› everywhere else. See Security.

Next steps

Install tamesu.
Follow the Quick start.
Read Concepts for suites, targets, personas, assertions and metrics.
Keep the CLI reference at hand.