試す tamesu · to try, to test

Your agents, tested like users would.

Describe a test the way you would brief a colleague: a persona, a goal and what a good outcome looks like. A simulated user then talks to your agent or drives your web app, and deterministic checks and LLM judges grade the result.

Install Read the docs View on GitHub

brew install xavidop/tap/tamesu

Homebrew is macOS only; on Linux and Windows, use a package or a release binary.

One YAML schema for browser, rest, grpc,livekit, sip, twilio, realtime,pipecat, voice, model, mcp and a2a targets, validated with file and line numbers.tamesu doctor resolves every secret through mamori and checks the setup.

tamesu init, validate, list and doctor in a terminal

Ten targets, one model

The same test for a prompt, a chatbot, a voice agent, a phone line, a realtime API and a web flow

browser

Web apps

Agentic goal and expect steps drive Chromium over CDP; deterministic actions, screenshots and a HAR for every run.

rest

REST agents

A simulated user talks to your HTTP agent turn by turn (JSON, SSE or WebSocket); JSONPaths read the reply, tool calls and the end of the session.

livekit

Voice agents

tamesu joins a LiveKit room as a caller, in text or real audio: TTS in, VAD and STT out, interruptions, silence and background noise included.

sip

Phone agents

Real calls through a LiveKit SIP service and your trunk: dial out or get called, press DTMF, follow IVR scripts, measure ring and setup time.

realtime

OpenAI Realtime

The Realtime protocol over a websocket, in text or audio: the API itself with your instructions and tools, or your own server speaking its events.

grpc

gRPC agents

A small turn API (Converse stream or unary Turn) any gRPC service can implement, with the same latency metrics as REST.

pipecat

Pipecat bots

The websocket transport of a Pipecat bot: protobuf audio frames and RTVI messages, text or real audio, with the bot's words and timing.

twilio

Twilio agents

tamesu plays Twilio's side of Media Streams: μ-law media, DTMF, marks and clear against your stream URL, so a phone agent is tested with no number, trunk or call.

mcp

MCP servers

An MCP server's tools, driven by a model: descriptions, argument schemas, answers and errors tested as an agent built on the server would meet them.

a2a

A2A agents

An agent speaking the Agent2Agent protocol: its card and skills, message/send or a streamed task, input-required turns and artifacts, judged like any other agent.

voice

Any voice websocket

A websocket that carries audio, PCM or G.711, binary or base64 in JSON, with the protocol's control messages read by JSONPath: hosted voice platforms and in-house stacks, from YAML.

model

Prompts, no server yet

A system prompt and its tool definitions straight against a model; tool calls are answered from the suite, so a prompt is regression-tested before the agent exists.

Quick start

A persona, a goal, and what good looks like

suite: support-agent
secrets:
  API_TOKEN: env:SUPPORT_API_TOKEN       # a mamori ref, never a value
targets:
  api:
    type: rest
    url: https://api.example.com/v1/chat
    headers: { Authorization: "Bearer {{secrets.API_TOKEN}}" }
    request: { body: { session_id: "{{session}}", message: "{{message}}" } }
    response: { text: $.reply, tool_calls: $.tool_calls[*].name }
tests:
  - name: refund policy question
    target: api
    persona: { name: Marta, style: "polite, slightly confused" }
    scenario: { goal: "Find out if you can return shoes bought 40 days ago", max_turns: 8 }
    assert:
      - judge: "The agent states the 30-day return policy and does not promise an exception"
      - not_contains: "I'm just an AI"
      - metric: { name: turn_latency_p95_ms, lt: 4000 }

$ tamesu init
$ tamesu validate
1 suite, 2 tests, 0 errors, 0 warnings
$ tamesu doctor
ok   config: tamesu.yaml (version 1)
ok   suites: 1 file, 2 tests
ok   secret API_TOKEN: env (link 1 of 1)

Follow the quick start or read theYAML reference, generated from the schema.

Why tamesu

Built for CI, honest about AI

assertions

One assertion model

judge rubrics with structured output next to contains, regex, tool_called, json_path and metric, on the whole test or one turn.

genkit

All AI through Genkit

The simulated user, the judges and speech are Genkit flows. Switch Gemini, Claude, GPT or Ollama with one line and trace every run in the Developer UI.

mamori

Secrets done right

Keys are mamori refs (env:, aws-sm://, vault://, op://), resolved once per run, revealed at the point of use and redacted everywhere else.