# senro documentation > A pipeline engine defined in Go: build a Line of Steps as ordinary Go code, run it, and attach to it live over a unix socket to watch and debug what it's doing. A pipeline engine first - CI/CD is the obvious first use, not the boundary of what it's for. The entire senro documentation, concatenated for agent consumption. --- > Source: https://xavidop.github.io/senro/docs # senro `senro` (線路, "railway track") defines pipelines in Go, executes them, and exposes a live attach protocol so a second process can watch and debug a run in progress. Reach for it when you want a pipeline as real, typed, testable Go code instead of YAML. It is a pipeline engine first, and nothing in the API is CI-specific. CI/CD is the most familiar thing to build on it, but a data pipeline, a batch job, an infrastructure rollout or a release script is built the same way: steps with dependencies, retries and failure handlers, one event stream you can attach to. > The railway metaphor lives in prose and error messages, never in identifiers. A step is a > station, a workflow is a line, the resolved execution graph is a timetable; the Go API says > `Pipeline`, `Workflow`, `Step` and `Plan`. ## Start | Page | What it gives you | |---|---| | [Install](/docs/install/) | `go get`, requirements, platform support | | [Quickstart](/docs/quickstart/) | A pipeline, a run, and a terminal attached to it | | [Concepts](/docs/concepts/) | Pipeline, workflow, step, plan, event stream | ## Steps: the building blocks | Page | What it covers | |---|---| | [Steps](/docs/steps/) | What a step is, `exec.Command`, the two step kinds | | [Ordering](/docs/steps/ordering/) | `Needs` on a step, `senro.Needs` on a workflow | | [Settings](/docs/steps/settings/) | `Env`, `WorkDir`, `Timeout`, `ContinueOnError` | | [Retries](/docs/steps/retries/) | `Retry`, `RetryPolicy`, the predicates, backoff | | [Handlers](/docs/steps/handlers/) | `OnFailure`, `Always`, and what a handler inherits | | [End states](/docs/steps/states/) | The ten end states, and how they propagate | | [Conditions](/docs/steps/conditions/) | `When`, `Branch`, `ParamIs`, `EnvIs` | | [Func steps](/docs/steps/functions/) | `RegisterFunc`, `senro.Func`, `senro.Ctx` | ## Executors: where steps run | Page | What it covers | |---|---| | [Executors](/docs/executors/) | The four targets, `senro.On`, read-only enforcement | | [Containers](/docs/executors/containers/) | `container.Image`, its five properties, `container.User` | | [Kubernetes](/docs/executors/kubernetes/) | A workflow on a pod: setup, behavior, refusals | | [SSH](/docs/executors/ssh/) | A workflow on a remote host: setup, behavior, refusals | | [Func off the coordinator](/docs/executors/func-remote/) | A `Func` step over SSH or in a container | ## Data: workspaces and caching | Page | What it covers | |---|---| | [Workspaces](/docs/data/workspaces/) | `senro.Workspace`, `Mount`, `ScopeRun`, snapshots | | [Persistent workspaces](/docs/data/persistent/) | `ScopePersistent` and its four rules | | [Scratch cache](/docs/data/scratch/) | `ScratchCache`, `Key`, `RestoreKeys` | | [Caching a step](/docs/data/caching/) | `Pure()`, `Inputs`, `Outputs`, `CacheEnv` | | [Cache keys](/docs/data/cache-keys/) | What enters a key, and `cache explain` | | [Shared cache](/docs/data/shared-cache/) | The S3 and OCI tier, config, degradation | | [Archiving](/docs/run/archiving/) | Archiving a run, and `logs fetch` | ## Monorepos | Page | What it covers | |---|---| | [Monorepos](/docs/monorepo/) | The problem, and which tool solves which part | | [Fan-out](/docs/monorepo/fan-out/) | `Expand`, `Template`, `MaxParallel`, `MaxNodes` | | [Per-unit ordering](/docs/monorepo/needs-each/) | Per-unit edges versus the workflow barrier | | [Partitioning](/docs/monorepo/partition/) | `Partition`, `TemplateShard`, the duration history | | [Affected](/docs/monorepo/affected/) | Running only what a change affects | ## Unit graphs | Page | What it covers | |---|---| | [Choosing a graph](/docs/monorepo/unit-graphs/) | The eight graphs, and which support `Affected` | | [`glob`](/docs/monorepo/unit-graphs/glob/) | One unit per directory matching a pattern | | [`gowork`](/docs/monorepo/unit-graphs/gowork/) | Go modules and packages, via `go list` | | [`cargo`](/docs/monorepo/unit-graphs/cargo/) | Rust crates, from the manifests | | [`jswork`](/docs/monorepo/unit-graphs/jswork/) | npm, pnpm, Yarn and Bun workspace packages | | [`maven`](/docs/monorepo/unit-graphs/maven/) | Maven reactor projects | | [`gradle`](/docs/monorepo/unit-graphs/gradle/) | Gradle projects, from the declarative subset | | [`pyproject`](/docs/monorepo/unit-graphs/pyproject/) | Python distributions, and why `Affected` is refused | | [`bazel`](/docs/monorepo/unit-graphs/bazel/) | Bazel packages, with and without running bazel | | [Write your own](/docs/monorepo/unit-graphs/custom/) | `UnitGraph` and `UnitAffector` | ## Secrets | Page | What it covers | |---|---| | [Secrets](/docs/secrets/) | Declaring and using a secret; the file-path rule | | [Channels](/docs/secrets/channels/) | Safe, redacted, refused; per-executor delivery | ## Watch and control a run | Page | What it covers | |---|---| | [Attach](/docs/attach/) | What attach is; one client, two sources; `attach.Listen` | | [The TUI](/docs/attach/tui/) | The terminal UI and its keys | | [The browser UI](/docs/attach/browser/) | `senro ui` | | [Control operations](/docs/attach/control-ops/) | The eleven operations, and the refusal codes | | [The shell](/docs/attach/shell/) | `senro shell` on a live step, `--tty` | | [Security](/docs/attach/security/) | The boundary, tokens, TLS, platform support | ## Triggers | Page | What it covers | |---|---| | [Running on an event](/docs/triggers/) | Wiring `WithTrigger`, the matchers, the three outcomes | | [Run it as a server](/docs/triggers/server/) | Your binary as the webhook endpoint, verified per source | | [The event file](/docs/triggers/events/) | The envelope every source is delivered through | | [GitHub](/docs/triggers/github/) | `push`, `pull_request`, and tags arriving as pushes | | [GitLab](/docs/triggers/gitlab/) | Push, tag push, merge request, and GitLab's own action words | | [Bitbucket](/docs/triggers/bitbucket/) | `repo:push`, `pullrequest:*`, and the missing file lists | | [Gitea](/docs/triggers/gitea/) | `push`, `pull_request`, `create`, and the double tag | | [Schedule & manual](/docs/triggers/manual/) | The neutral shape, for cron and for a button | | [Write your own](/docs/triggers/custom/) | `trigger.Provider` and `trigger.Matcher` | ## Notifications | Page | What it covers | |---|---| | [Sending a result out](/docs/notifications/) | The destinations, every option, delivery and retries | | [Slack](/docs/notifications/slack/) | A line in a channel, and what widening it costs | | [Webhook](/docs/notifications/webhook/) | Raw events as JSON, and verifying a signature | | [GitHub Checks](/docs/notifications/github-checks/) | A check run on the commit, with annotations | | [Write your own](/docs/notifications/custom/) | `notify.Renderer` and `notify.Requester` | ## Failure analyzers | Page | What it covers | |---|---| | [What an analyzer does](/docs/analyzers/) | Explaining a failed step, and the approval gate | | [The AI analyzer](/docs/analyzers/genkit/) | `contrib/genkitanalyzer`, with the model of your choice | | [Write your own](/docs/analyzers/custom/) | `Analyzer`, `api.Failure` and `api.Proposal` | ## Extend | Page | What it covers | |---|---| | [Extension points](/docs/extend/) | Every extension point, one line each | | [A trace exporter](/docs/extend/exporter/) | A trace exporter as a `Sink` | | [A unit graph](/docs/monorepo/unit-graphs/custom/) | Implement `UnitGraph` / `UnitAffector` | | [A trigger source](/docs/triggers/custom/) | Implement `trigger.Provider` | | [A notifier](/docs/notifications/custom/) | Implement a `Renderer` / `Requester` | | [A failure analyzer](/docs/analyzers/custom/) | Implement `Analyzer`, and the proposal gate | ## Reference | Page | What it covers | |---|---| | [CLI](/docs/cli/) | Every command in one table, plus exit codes | | [Running and watching](/docs/cli/run/) | `senro run`, `attach`, `shell`, `ui` | | [Cache commands](/docs/cli/cache/) | `cache gc`, `cache explain`, `verify` | | [Workspace commands](/docs/cli/workspaces/) | `ws ls/pull/diff`, `logs fetch`, `func check` | | [The event stream](/docs/run/event-stream/) | The event envelope, and turning it into live state | | [`api`](/docs/run/api/) | The `api` package as a wire contract | | [Run options and outcomes](/docs/run/options/) | Every `senro.Run` option, and reading `RunError` | | [Reading a failed run](/docs/run/debugging/) | The run directory, file by file | | [Agent skill](/docs/reference/skill/) | The skill that teaches an AI agent senro | --- > Source: https://xavidop.github.io/senro/docs/reference/skill # Agent skill senro ships an [Agent Skill](https://www.skills.sh/) that teaches an AI coding agent (Claude Code, Cursor, Copilot, Windsurf, Gemini, and others) how to use senro: the `Pipeline`/`Workflow`/`Step` model, fan-out, caching, secrets, attach, and the CLI. You'll find the source under [`skills/senro/`](https://github.com/xavidop/senro/tree/main/skills/senro) in the repo. ## Install ```bash npx skills add xavidop/senro ``` This works for any agent skills.sh supports. It fetches the skill into your agent's skills directory, and the agent loads it automatically for tasks like defining a pipeline, wiring retries or failure handlers, workspaces, caching, secrets, attach, or the `senro` CLI. To install manually, copy the folder into your agent's skills directory (Claude Code: `~/.claude/skills/`; other agents use their own location, such as a project-level `.cursor/` or `.github/` skills folder; see your agent's documentation): ```bash git clone https://github.com/xavidop/senro cp -r senro/skills/senro ~/.claude/skills/senro ``` ## What ships One `SKILL.md`, plus two reference files the agent loads only when it needs them: `references/cli.md` (the command reference) and `references/secrets-channels.md` (the channel table). ## What it covers - **The core model**: `Pipeline`, `Workflow`, `Step`, and the two ways steps can depend on each other. - **The two step kinds**: an `exec.Command` that runs on any executor, and a `senro.Func` Go function that also runs on any executor. - **All four executors**: local, containers, Kubernetes, and SSH, and what each does with workspaces and secrets. - **Monorepo tools**: fanning a step out across a unit graph, ordering per-unit work, sharding into balanced partitions, and running only what a change affects. - **Retries and failure handling**: retry policies, timeouts, and `OnFailure`/`Always` handlers. - **Workspaces and caching**: workspaces, persistent workspaces, the scratch cache, and caching a step's output. - **Secrets**: how to pass them in, and which channels are safe, redacted, or refused. - **Attach and control**: connecting to a live run, the operations you can issue, and opening a shell inside a running step. - **The `senro` CLI**: every command, flag, and exit code. ## llms.txt If your agent doesn't support skills, point it at the docs directly instead. This site publishes the [llms.txt convention](https://llmstxt.org/): [`/llms.txt`](/llms.txt) is a short index for an agent to navigate, and [`/llms-full.txt`](/llms-full.txt) is the whole documentation site as one Markdown file. A prompt that works in most coding agents: ```text Add senro to my Go project. Docs: ``` ## See also - [Quickstart](/docs/quickstart/): the same ground for a human reader. - [Steps](/docs/steps/): the two step kinds in full. - [Fan-out](/docs/monorepo/fan-out/) and [Conditions](/docs/steps/conditions/): `Expand` and `When`, the same ground this skill covers. - [Secret channels](/docs/secrets/channels/): the table `references/secrets-channels.md` mirrors. - [CLI](/docs/cli/): the command reference `references/cli.md` mirrors. --- > Source: https://xavidop.github.io/senro/docs/run/api # `api`: the wire contract `github.com/xavidop/senro/api` is a Go package with the types you need to read a run from the outside: the event envelope, the attach protocol's request/response frames, the run and step status enums, and a function that turns a stream of events into live state. Reach for it when you're writing your own client or tool instead of using the shipped TUI or browser UI. It's part of the main `senro` module, so `go get github.com/xavidop/senro` is all you need. ## What's in it - **`Event`** and **`Type`**: the envelope every stream entry uses, and the full list of event types. See [The event stream](/docs/run/event-stream/). - **`Frame`**, **`Kind`**, and the `Op*` constants: the attach wire protocol. See [Control operations](/docs/attach/control-ops/). - **`State`** and **`RunStatus`**: a step's final state and a run's overall outcome, plus `RollUp([]State) RunStatus`, the function that computes one from the other. See [Step states](/docs/steps/states/). - **`RunState`** and **`(*RunState).Apply(Event) error`**: turns a stream of events into a live picture of a run. The attach server, the TUI, the browser UI, and offline replay all use this same function, so they always agree on what a run's state is. - **`Version`**, **`VersionMinor`**, **`CheckVersion`**: for checking that a client and the engine speak compatible versions of the protocol. See [Control operations](/docs/attach/control-ops/). ## Small and dependency-free `api` imports nothing beyond the Go standard library. That keeps it easy to depend on: a Slack bot that posts on `run.finished`, a status-page poller, or your own TUI only needs `api`'s types, not the rest of senro. See [A Func step off the coordinator](/docs/executors/func-remote/) for another place this matters: it bounds what has to be shipped to a remote host. ## Stability Changes to `api` are additive: a type is never renamed, removed, or repurposed within the same major protocol version (the one `api.Version` reports). As a consumer, the rule for you is: **ignore event types and struct fields you don't recognize.** `encoding/json` already skips unknown struct fields, and `Type.Known()` skips unrecognized event types rather than erroring. That's what lets the protocol add new things later without breaking your client. ## Where to go next - **[The event stream](/docs/run/event-stream/)**: the envelope, in depth. - **[Attach](/docs/attach/)**: the protocol built on these types. - **[Step states](/docs/steps/states/)**: what `State` and `RunStatus` mean. --- > Source: https://xavidop.github.io/senro/docs/run/archiving # Archiving a run A CI runner is destroyed when its job ends, and your build's logs go with it. Archiving is what gets them back. While the run is still going, senro uploads each step's output to the [shared cache](/docs/data/shared-cache/), so you can pull the whole run down onto any machine afterwards and read it exactly as if it had run there. ```sh senro logs fetch 20260812T151058-540c8ca44b # into ./runs/20260812T151058-540c8ca44b senro attach --run 20260812T151058-540c8ca44b # read it like any other run ``` There is nothing to configure. Archiving turns itself on whenever the shared cache is on, and `senro logs fetch` reads the same `SENRO_REMOTE_CACHE` variables the original run used, so any machine already set up for the shared cache can already fetch runs. ## What is uploaded, and when - **Each step attempt's `stdout` and `stderr`, uploaded as that attempt finishes**, not at the end of the run. So a run that crashes or gets cancelled halfway through has already archived everything that completed. - **The event ledger, `events.jsonl`, uploaded once at the end**, after it's sealed. This is the file the other logs depend on for meaning: which steps ran, how they ended, and where in each stream every write landed. - **Handler output** (`OnFailure`, `Always`): usually the part someone actually needs, since it's the evidence of whether cleanup ran. - **The logs a cached step replays**, so you can't tell from the archive alone which steps hit the cache. The ledger's `cache.hit` events tell you that instead. ## The write path never touches a step A step's execution never waits on an upload: queuing one is a non-blocking handoff to a background worker. If the upload queue fills up, the oldest uploads are dropped from the **archive**, never from local disk, and the run reports this. ```mermaid flowchart LR Step["step attempt finishes"] --> Queue["upload queue
(non-blocking)"] --> Worker["background worker"] --> Store["shared store"] Queue -- "queue full" --> Drop["oldest dropped
(archive only)"] ``` When the run ends, senro drains the queue with a bounded grace period (`CleanupGrace`, sixty seconds by default). A job that can't exit because a store is slow is worse than a lost upload. ## When the store is unreachable Just like the cache: **nothing fails**. The run's exit code describes the pipeline, never the store. Logs stay on local disk, still stream live to anything attached, and still print to your CI log. One `cache.degraded` event and one line on standard error tell you the archive didn't happen. See [When the store is unreachable](/docs/data/shared-cache/#when-the-store-is-unreachable). > **Why not live logs?** Neither store supports appending. Live upload would mean either multipart > uploads nobody can read until they're complete, or one object per chunk that the reader stitches > back together, and either way, a store outage would sit inside every log line's write path. Live > reading is already handled by the attach server streaming straight from the local file. ## Where it lands Logs are stored content-addressed, like cached objects, with a small mutable pointer naming the digest. In a bucket: ``` /v1/cas/sha256/// the bytes /v1/runs//logs/// the digest /v1/runs//events the ledger's digest ``` and in a registry, where every name is a tag with a restricted alphabet, so run and step ids are hashed (see [what a tag costs](/docs/data/cache-stores/#what-a-tag-costs)): ``` senro-v1-sha256- the bytes senro-v1-log-sha256- the digest senro-v1-run-sha256- the ledger's digest ``` Two runs that produce identical output store it once. A fetched log is verified against its digest with the same code that verifies a cached object, so a truncated download or a substituted object is refused rather than shown to you as if it were your build's real output. ## Reading a run back What `senro logs fetch` writes is an **ordinary run directory**, so every tool senro has for reading a run works on it. `DEST` defaults to `./runs/RUN`, the same path `senro attach --run` resolves on its own, and the fetch prints the command to use once it's done. - **Read permission is all it needs** (`s3:GetObject` on the prefix, or `pull` on the repository). Which streams to fetch comes from the ledger, never from listing the store. That matters more on a registry, since senro doesn't implement tag listing at all. - **A stream the ledger names but the archive doesn't have is reported, not treated as an error.** An upload that never finished and one a lifecycle rule expired look the same from here, and the rest of the run is still worth having. - **`DEST` is replaced, not merged into**, exactly as `senro ws pull`'s is. A non-empty destination is refused unless `--force`. - **The exit code describes the fetch, never the archived run.** Fetching the record of a failed build is a success. `senro attach --run` turns that run's own outcome into an exit code. The destination rules, refusals and the full exit code table are in [`senro logs fetch`](/docs/cli/workspaces/). ## Retention senro never deletes from a shared store, so expiry is the store's job, exactly as for the cache itself. On a bucket that is a lifecycle rule: ```json { "Rules": [ { "ID": "expire-senro-cache", "Status": "Enabled", "Filter": { "Prefix": "senro/" }, "Expiration": { "Days": 30 } } ] } ``` If you archive to a bucket, consider a **longer expiry for the `runs/` prefix than for the cache**. A stale cache entry is worthless, but the log of a build from six weeks ago might be exactly what someone needs. A registry can't express that split, since a tag namespace has no prefix a policy can filter on, so pick the retention that the logs deserve. Registry retention is whatever the repository offers: an age-based policy or a periodic sweep. This is the one place a bucket is meaningfully easier to work with. An expired log is simply missing when a fetch goes looking. The rest of the run still comes back. ## Where to go next - **[Shared cache](/docs/data/shared-cache/)**: turning the store on, and how it degrades. - **[Reading a failed run](/docs/run/debugging/)**: what to do with the run directory once you have it. - **[Attach](/docs/attach/)**: reading a run live instead. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/bazel # `bazel` Two graphs, and the choice between them is whether you are willing to run bazel while senro is still planning. | | Discovery | Affected set | |---|---|---| | `bazel.Packages()` | A tree walk. No bazel needed. | No | | `bazel.Query()` | The same tree walk. | **Yes**, by running `bazel query` | ```go import "github.com/xavidop/senro/unit/bazel" verify.Expand("test", bazel.Packages()). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("bazel", "test", u.Name+"/...")) }) ``` ## What counts as a unit One **Bazel package**: a directory holding a `BUILD` or `BUILD.bazel` file. | Field | For `apps/web` | |---|---| | `Name` | `//apps/web`, the package label (`//` for the root) | | `ID`, `Dir` | `apps/web` | Discovery is a pruned walk plus one small file read. **No bazel is needed and none is run**; a test empties `PATH` and checks exactly that. Left out, because Bazel itself would refuse to build them from this root: directories in `.bazelignore`, directories inside a nested repository (their own `MODULE.bazel`, `REPO.bazel` or `WORKSPACE`), and directories with no `BUILD` file. A root that is not a workspace root, or a workspace with no `BUILD` file anywhere, is an error rather than an empty graph. **Not one unit per target.** A macro computes its targets' names, so enumerating them means evaluating Starlark or running bazel. `bazel test //apps/web/...` covers the package anyway. ## Why `bazel.Packages()` refuses `Affected` The only way to answer without running bazel is to parse `BUILD` files, and that cannot be done correctly. A `BUILD` file is a Starlark program: macros compute their own deps, `glob()`, `select()` and comprehensions build them dynamically, a dep can be a variable or an alias, and edges also come from toolchains, implicit rule dependencies and the `.bzl` files themselves. Each is a missing edge, and a missing edge is a green build for a tree that does not build. ## `bazel.Query()` Answers by **running bazel**, which has no such problem: ```go verify.Expand("test", bazel.Query()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("bazel", "test", u.Name+"/...")) }) ``` One query for the whole workspace, `bazel query --output=xml 'kind(rule, //...)'`, whose target edges are mapped to package edges. Units still come from the tree walk, because a package is a directory holding a `BUILD` file and that needs no toolchain; only the edges need bazel. Labels into other repositories (`@rules_go//...`) are dropped: they are not units of this workspace. Choosing it is choosing to run bazel while senro is still planning, and that is the whole cost: - It is a **build, not a lookup**. A JVM starts, every `BUILD` file is evaluated, and under bzlmod repository rules run, which is arbitrary code executing during planning. - The answer **depends on the machine**: bazel version, `.bazelrc`, `--config`, the module lockfile. **It never skips.** bazel missing, bazel failing, or output it cannot parse is an error and the expansion fails. A graph that skipped cleanly when bazel was absent would compute one set on CI and a different one on a laptop, with nothing saying which happened. ## Which to use **Prefer `bazel.Packages()` and fan out over everything.** It is close to what a Bazel repository wants anyway, since bazel does its own incrementality per invocation: running `bazel test` on a package bazel already has cached costs almost nothing. Reach for `bazel.Query()` when the packages themselves are expensive to even start (a container build per package, a deploy per package) and the planning-time cost is worth paying. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows. - **[Fan-out](/docs/monorepo/fan-out/)**: `MaxParallel` and `MaxNodes` for a large workspace. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/triggers/bitbucket # Bitbucket triggers `provider: "bitbucket"`. Reads Bitbucket Cloud's `repo:push` and every `pullrequest:*` event. ## Wire it ```go ev, err := trigger.LoadEvent(*eventPath) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")), trigger.OnPullRequest(trigger.Actions("created", "updated")), trigger.OnTag(trigger.Semver(">=1.0.0")), )) ``` ## Write the event file `event` is Bitbucket's `X-Event-Key` header, verbatim: ```json {"provider":"bitbucket","event":"repo:push","payload":{ ...Bitbucket's body... }} {"provider":"bitbucket","event":"pullrequest:created","payload":{ ... }} ``` ## What it accepts | `event` | Becomes | |---|---| | `repo:push` | `Push`, or `Tag` for a tag or annotated tag | | `pullrequest:created`, `:updated`, `:fulfilled`, `:rejected`, `:approved`, and the rest | `PullRequest` | ## Worth knowing **No Bitbucket payload carries a changed-file list at all.** A commit here has a hash, a message and an author, and no paths. So `Paths` against **any** Bitbucket event is an [error](/docs/triggers/events/#when-there-is-no-file-list), not a no-match. Narrow with `Branches`, or with an [affected set](/docs/monorepo/affected/) once the run has started. **One delivery may move several refs.** A push payload carries an array of changes, and senro's event is one ref, so a multi-ref delivery is **refused** rather than half read. Split the envelope first, one event per entry: ```sh jq -c '.payload.push.changes[] as $c | .payload.push.changes = [$c]' event.json \ | while read -r one; do echo "$one" | ./pipeline --trigger-event -; done ``` **The action comes from the event key's suffix**, because a pull request body carries none. It is carried through untranslated: ```go trigger.OnPullRequest(trigger.Actions("created", "updated")) ``` **A created or deleted ref is a null `old` or `new`**, not an all-zero SHA. That is the whole of what Bitbucket says about either. A deleted ref never matches. **The repository object names no default branch**, so a Bitbucket push is always mode `affected`. See [what a match carries](/docs/triggers/#what-a-match-carries-into-the-run). **Mercurial references are refused.** A change on a `named_branch` or a `bookmark` reaches neither of senro's kinds; a git repository's changes are on a `branch`, `tag` or `annotated_tag`. **The body is cross-checked.** A payload carrying the other top-level object (a `pullrequest` where the envelope said `repo:push`, or the reverse) is refused. The body names no event of its own, so which object is present is the only check there is. **Hashes are abbreviated to 12 characters** by Bitbucket, and senro reports what the event said, so whatever consumes the base resolves it. A pull request's `Number` is `pullrequest.id`, the per-repository number a person sees. ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers, and what a match carries into the run. - **[The event file](/docs/triggers/events/)**: the envelope every source shares. - **[Affected sets](/docs/monorepo/affected/)**: narrowing a run when `Paths` cannot help. --- > Source: https://xavidop.github.io/senro/docs/data/cache-keys # Cache keys The [action cache](/docs/data/caching/) skips a step when its key matches a result senro already has, so the key is what decides whether a step reruns or repeats work you already paid for. Every key is built from the same twelve components, on every machine, whether the result is stored locally or in the [shared cache](/docs/data/shared-cache/). This page covers what's in a key, what isn't, and how to read a miss. ## The twelve components | Component | What it covers | | --- | --- | | `command` | The step's kind, argument vector and working directory | | `env` | The allowlisted environment, as name plus value digest pairs, never a value | | `secrets` | The declared secrets' identity: name, source, version and a salted digest, never a value | | `executor_class` | The executor's cache equivalence class, deliberately not host identity | | `platform` | The declared platform | | `input_digests` | The sorted paths and digests of the step's declared inputs | | `workspace_digests` | The sorted names and digests of the workspaces the step mounts | | `mount_shape` | The same mounts' name, mode and path, without their content | | `step_shape` | `NoSnapshot` and the declared `Outputs`, which decide what a saved result contains | | `func_identity` | A `Func` step's binary digest, registered name and parameter digest | | `tool_versions` | The declared toolchain fingerprint | | `version` | The key format's own version | Each component comes from something you wrote: `Inputs` feeds `input_digests`, `Mount` feeds `workspace_digests` and `mount_shape`, `CacheEnv` feeds `env`. See [Caching a step](/docs/data/caching/). ```mermaid flowchart LR Inputs["Inputs(...)"] --> ID["input_digests"] --> Key["cache key"] Mount["Mount(...)"] --> WD["workspace_digests"] --> Key Mount --> MS["mount_shape"] --> Key CacheEnv["CacheEnv(...)"] --> ENV["env"] --> Key Other["+ 8 more components:
command, secrets, platform..."] --> Key ``` ## What never enters a key - **Any environment variable you didn't name in `CacheEnv`.** The allowlist is all of `env`, and even an allowlisted variable enters as a digest of its value, never the value itself. - **A secret's value, ever.** `secrets` carries a secret's *identity*: its name, source, version and a salted digest. That holds for the local cache, the shared cache, a bucket key and a registry tag alike. See [Secrets](/docs/secrets/). - **Host identity.** `executor_class` is an equivalence class, so a fleet of interchangeable machines can share entries. If the class were built from hostnames instead, forty machines would never share a single entry, and nothing would tell you why. On Kubernetes, the namespace is deliberately left out of the class too. - **A [scratch cache](/docs/data/scratch/)**: not its content, not its key, not its mounts. - **Which store you use.** A [shared cache](/docs/data/shared-cache/) changes where a result is kept, never what it is keyed by. Two machines share an entry only when they would have computed the identical thing. - **How you grouped your steps.** Moving steps between workflows does not change the plan's digest. Adding a workflow-level `Needs` does, because that adds real edges. See [Ordering](/docs/steps/ordering/). Two details about `executor_class` are worth knowing. A declared [`container.User`](/docs/executors/containers/) enters the class, but the default doesn't, since the default just names the coordinator's own identity rather than anything about the pipeline. And [`ssh.CacheClass`](/docs/executors/ssh/) is yours to keep accurate: senro has no way to tell that one host quietly picked up a different toolchain. ## Reading a miss `senro cache explain` diffs a step's current key against the most recent recorded entry for that step, field by field: ``` MISS measure key e126dad1 (previous 2ba03dd0) ✗ input_digests: greeting.txt 86c9c55c → 37e3516a ✗ workspace_digests: src 37931680 → d8ded6fe ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity, tool_versions, version unchanged ``` ```sh senro cache explain # every Pure() step and scratch cache the latest run touched senro cache explain build/test # one step's own key, hit or miss, field by field ``` - Only a step marked `Pure()` has a cache record. A step skipped because a dependency failed never reaches the cache and has none either. - A run with no `Pure()` steps and no scratch cache says so explicitly, rather than printing nothing, and still exits `0`. - `workspace_digests` moving tells you a workspace changed, not what changed inside it. [`senro ws diff`](/docs/cli/workspaces/) answers that. - A key that did **not** change is not proof the step was pure. That is what [`senro verify --recheck-pure`](/docs/data/caching/) is for. Full flags are in [Cache commands](/docs/cli/cache/). A worked example of a miss is in [Reading a failed run](/docs/run/debugging/). ## Where to go next - **[Caching a step](/docs/data/caching/)**: declaring the things this page keys on. - **[Shared cache](/docs/data/shared-cache/)**: the same keys, a second tier. - **[Persistent workspaces](/docs/data/persistent/)**: the one workspace whose content is measured before the run starts. --- > Source: https://xavidop.github.io/senro/docs/data/cache-stores # Cache stores Turning on the [shared cache](/docs/data/shared-cache/) means picking somewhere for it to live: anything that speaks the S3 API, or any OCI registry. Both hold the same objects, action cache entries, and [archived runs](/docs/run/archiving/), verify them the same way, and degrade the same way when the store is unreachable. What differs is what you already run, what it costs to expire, and how legible the store is when you go looking through it. This page helps you pick one and configure it. ## Which one - **A bucket is cheaper to expire.** Object stores have lifecycle rules; a registry wants a retention policy per repository or a periodic sweep. - **A registry is easier to already have.** A robot account and a retention policy is the whole setup: no region to guess at, no addressing style, no second system to provision. - **A bucket lists more legibly.** A key holds the step id and run id in the clear; a registry tag cannot. See [what a tag costs](#what-a-tag-costs). - **Only a bucket can share scratch caches.** `SENRO_REMOTE_SCRATCH` needs a prefix listing for the `RestoreKeys` fallback, and the registry API cannot list by prefix, so an `oci://` target ignores it. This is the one thing the two backends do not both do. See [Sharing scratch caches](/docs/data/scratch-sharing/). ## Buckets senro signs its own requests with AWS Signature Version 4 and doesn't depend on an SDK. Known to work: Amazon S3, MinIO, Cloudflare R2, Backblaze B2, Ceph RADOS Gateway, and Google Cloud Storage's S3 interoperability endpoint. Amazon S3 is configured exactly as in [Turn it on](/docs/data/shared-cache/#turn-it-on). Two stores want something different: ```sh # MinIO, or anything self-hosted. Bucket-in-path addressing is chosen automatically for any # endpoint that is not Amazon's; override with SENRO_REMOTE_CACHE_PATH_STYLE if yours disagrees. export SENRO_REMOTE_CACHE_ENDPOINT="http://minio.internal:9000" export SENRO_REMOTE_CACHE_REGION="us-east-1" # Cloudflare R2 has one region and calls it auto. export SENRO_REMOTE_CACHE_ENDPOINT="https://.r2.cloudflarestorage.com" export SENRO_REMOTE_CACHE_REGION="auto" ``` ## A registry instead of a bucket `oci:///` names the registry host (with a port if it isn't 443, e.g. `registry.internal:5000`, which is a host, not a URL) and the repository path inside it (e.g. `acme/senro-cache`, lowercase as the spec requires; it's created on first push). One repository holds the objects, the action cache, and the archived runs, so a fleet configured this way needs no bucket at all. ### Authentication senro supports one authentication flow: **the OCI token challenge**, which every hosted registry serves. The registry answers `401` with a challenge naming a realm. senro fetches a token from that realm, presenting your username and password as HTTP Basic, then repeats the request with the token. Your credential goes only to the token endpoint, nowhere else. Where the username and password come from is up to you. senro doesn't run a credential helper, doesn't read `~/.docker/config.json`, and doesn't contact any metadata service. | Registry | Username | Password | | --- | --- | --- | | GitHub Container Registry | your GitHub username, or `x-access-token` | a PAT with `write:packages`, or `${{ secrets.GITHUB_TOKEN }}` | | GitLab | `gitlab-ci-token` | `$CI_JOB_TOKEN` | | Quay, Harbor, Artifactory | the robot or service account | its token | | Amazon ECR | `AWS` | `$(aws ecr get-login-password)` | | Google Artifact Registry | `oauth2accesstoken` | `$(gcloud auth print-access-token)` | **Every other scheme is refused, by name.** If a registry answers with `Basic`, `Negotiate`, or a Bearer challenge that names no realm, senro prints one clear line saying what the registry asked for and that senro doesn't support it. A self-hosted registry using `htpasswd` Basic authentication needs a token endpoint in front of it, or you should use a bucket instead. A registry that demands no credential at all works fine with none. That's convenient on a laptop, but not something you should share as a team cache. ### How an object is stored Each object is a small OCI artifact, tagged with the digest of its **plaintext** and holding the compressed bytes as its single layer: ``` senro-v1-sha256- the tag └── manifest application/vnd.oci.image.manifest.v1+json └── layer application/vnd.senro.cache.object.v1+zstd ``` This indirection lets a senro digest stay the digest of the plain content, while the registry addresses blobs by the digest of the compressed bytes. It also means every blob is referenced by a manifest, so a registry's garbage collector treats the cache as something to keep, and a retention policy has something to act on. Concurrent uploads need no coordination. Two runners finishing the same step push identical bytes under identical digests, and write byte-identical manifests to one tag. ### What a tag costs An action cache is a **mutable key-to-value mapping**. A registry has exactly one mutable name, a **tag** (at most 128 characters from `[A-Za-z0-9_.-]`), so a tag isn't the obvious way to bridge the two: it's the only way. That splits the mapping in two pieces. **The entry itself costs nothing**: a cache key is already a digest, so it goes into a tag basically unaltered (except for the one character a tag can't hold). **Everything else has to be hashed, which costs legibility**: a step id is arbitrary text (like `build/test[os=linux]`), and run ids and stream names aren't digests either. ``` senro-v1-action-sha256- the entry for that key hex is the cache key itself senro-v1-recent-sha256- a step's most recent key ... sha256 of the step id senro-v1-log-sha256- one archived log stream ... of run, step, attempt, stream senro-v1-run-sha256- a run's event ledger ... of the run id ``` > **`crane ls` on a senro cache repository shows a wall of hex**, and nothing in it says which step > or run a tag belongs to. A bucket keeps the step id and run id in the key itself, percent-encoded, > so if you expect to debug a cache by browsing it, a bucket is kinder. Going from a step id you > already have to its tag works fine. Only the reverse is closed off. The hashing is **not** a privacy measure: a secret's value never reaches a tag, a key, or an object, on either backend. It costs nothing in correctness, either. A tag is just a name anyone with push access can write, so senro never trusts a tag on its own: - The manifest records which document it is. - The bytes are hashed against the digest the manifest names. - An action-cache entry additionally carries the key it was filed under. The registry's own digest check on upload only proves a blob is the bytes it claims to be. It says nothing about whether the manifest pointing at it names the right object, so it isn't one of the checks senro relies on here. **Two machines writing one key** is a case a bucket doesn't have. Both writes succeed, and the later one stays. That's what a mutable name means. Both machines ran the same action under the same key, so either result is legitimate. The loser's blob is simply left unreferenced, for the registry's own garbage collector to clean up. ## Running it in CI A trunk build fills the cache; a pull-request build reads it. Give the pull-request build a read-only credential and set `SENRO_REMOTE_CACHE_READ_ONLY=1`: ```yaml # after aws-actions/configure-aws-credentials has assumed the cache role via OIDC - run: go run ./ci env: SENRO_REMOTE_CACHE: s3://acme-senro-cache SENRO_REMOTE_CACHE_ENDPOINT: https://s3.eu-west-1.amazonaws.com SENRO_REMOTE_CACHE_REGION: eu-west-1 SENRO_REMOTE_CACHE_READ_ONLY: ${{ github.event_name == 'pull_request' && '1' || '0' }} ``` The same job against a registry, where the forge already issues the credential: ```yaml SENRO_REMOTE_CACHE: oci://ghcr.io/acme/senro-cache SENRO_REMOTE_CACHE_USERNAME: x-access-token SENRO_REMOTE_CACHE_PASSWORD: ${{ secrets.GITHUB_TOKEN }} SENRO_REMOTE_CACHE_READ_ONLY: ${{ github.event_name == 'pull_request' && '1' || '0' }} ``` > `SENRO_REMOTE_CACHE_READ_ONLY` is a courtesy, not a real control. Back it up with the store's own > permissions: what actually stops an untrusted build from writing should be the credential it was > given, not a variable it could unset itself. For a registry, that means a pull-only robot account > or a job token without `write:packages`. Nothing needs a delete permission, because senro never deletes from a shared cache: - **Bucket:** `s3:GetObject` and `s3:PutObject` on the prefix for a writing build; `s3:GetObject` alone for a reading one. - **Registry:** `pull` and `push` on the one repository; `pull` alone for a reading build. No catalog listing, no tag listing, no delete: senro addresses everything it stored by name. ## Where to go next - **[Shared cache](/docs/data/shared-cache/)**: turning the tier on, its variables, and how it degrades. - **[Archiving a run](/docs/run/archiving/)**: the run records this store also holds. --- > Source: https://xavidop.github.io/senro/docs/data/caching # Caching a step The action cache skips a step if its inputs haven't changed, and restores what it produced last time. It's **opt-in per step**: mark the step `Pure()`, declare what it reads with `Inputs`, and optionally declare what it produces with `Outputs`. ## Which cache is which senro says "cache" in three places, and only two of them are caches. The third is a place the first one lives. | Name | What it does | Where it lives | | --- | --- | --- | | **Action cache** (this page) | Skips a step entirely. Opt in with `Pure()` and `Inputs`, restores its `Outputs` | Local disk, plus the shared store when one is configured | | **[Scratch cache](/docs/data/scratch/)** | Restores a *directory* by key before a step runs. Never skips anything | Local disk, and a bucket only if you set `SENRO_REMOTE_SCRATCH` | | **[Shared cache](/docs/data/shared-cache/)** (`SENRO_REMOTE_CACHE`) | Not a cache. The S3 or OCI store the action cache is kept in, so other machines can reuse it | It *is* the off-machine part | One sentence each: the action cache answers "can I skip this work?", the scratch cache answers "can I avoid re-downloading this?", and the shared cache answers "where does the action cache live?". Two consequences worth knowing before you read further. `SENRO_REMOTE_CACHE` on its own does nothing for scratch caches: they travel only when you also set `SENRO_REMOTE_SCRATCH` ([why it is separate](/docs/data/scratch-sharing/)). And a scratch cache's contents never enter an action cache key, so the two are fully independent. A [workspace](/docs/data/workspaces/) is not a cache at all, though it occupies the same mental slot. It's how data moves between steps, and unlike a scratch cache it crosses into pods and onto ssh hosts. ```go verify.Step("test", exec.Command("go", "test", "./...")). Needs("compile"). Mount(src.At("/src", senro.RO), gomod.At("/root/go/pkg/mod")). Pure(). // eligible for the action cache Inputs(artifact.Glob("**/*.go"), artifact.File("go.sum")). Outputs(artifact.File("coverage.out")). CacheEnv("GOFLAGS") // by digest, never by value ``` ## 1. Mark the step `Pure()` `Pure()` marks a step as **eligible** for the action cache. Steps are impure by default: senro can also SSH into production and restart a service, so caching has to be something you explicitly opt into, not something that happens without your knowledge. A `Pure()` step must declare `Inputs`. `Build()` rejects one that doesn't, because a cache key that can't change when the sources change is worse than no cache at all. ## 2. Declare `Inputs`, and `Outputs` if you have them `Inputs(sel ...artifact.Selector)` and `Outputs(sel ...artifact.Selector)` take `artifact.File(path)` and `artifact.Glob(pattern)`, with the [pattern syntax](/docs/data/workspaces/#pattern-syntax) senro uses everywhere. - **Inputs are hashed into the cache key.** If the step reads something you didn't declare, the key can't see it. This is the usual way a wrongly-marked-`Pure()` step causes problems. - **Outputs are stored on a save and restored on a hit.** They also affect the key's shape, since they determine what a saved result contains. - **`Outputs` needs a mounted workspace.** Otherwise `Build()` refuses it: `plan: step "compile" declares Outputs but mounts no workspace, so nothing would survive the step to be stored: mount a workspace and write the outputs into it`. If a step's outputs land in a workspace, mount that workspace with `senro.RW`. ## 3. Add `CacheEnv` for variables that matter `CacheEnv(names ...string)` names environment variables that enter the cache key **by digest, never by value**. That way a credential in a step's environment can never leak into a cache entry. No other environment variable affects the key at all. Declaring the same variable in both `SecretEnv` and `CacheEnv` is refused at `Build()`. ## What a hit does A cache hit **skips the step entirely**. senro restores its declared outputs and mounted workspaces from the store, replays its recorded logs, and records a `cache.hit` event in the ledger. A miss just runs the step normally and saves the result. Two things are worth knowing about the boundary: - A step skipped because a dependency failed never reaches the cache, so it has no cache record. - A [scratch cache](/docs/data/scratch/) is not part of any of this. It is restored best-effort and is never an input to the key. ## Verify the claim with `senro verify --recheck-pure` `Pure()` is **trusted, not enforced**. senro doesn't sandbox a step's network access, so if a step claims purity but downloads something anyway, senro believes it and serves that result to every future run with the same key. ```sh senro verify --recheck-pure # what it WOULD re-run; nothing executes senro verify --recheck-pure --rerun # re-run the latest run's cached Pure() steps senro verify --recheck-pure --rerun --fail-on-mismatch # exit 1 on a finding, for CI ``` This command re-runs a cached step against the exact input its own cache key recorded, then compares the digests of the declared outputs, the mounted workspaces, and the exit code. Here's what that does and doesn't prove: - **`verified`**: the re-run reproduced the entry exactly. - **`mismatch`**: the re-run differed from the entry, **and a second re-run agreed with the first**. The step is deterministic, but it still didn't reproduce what's in the cache. That means it depends on something its key doesn't cover: the network, a file outside its workspace, an environment variable it never declared in `CacheEnv`, or the clock. - **`nondeterministic`**: the re-run differed from the entry, **and from a second re-run of itself**. This disagreement isn't evidence about purity: it's just the step being nondeterministic. An archive that embeds a build timestamp lands here, for example. This category keeps the report free of false alarms. - **Logs are never compared**, since a step's output legitimately carries timestamps, durations, PIDs and temp paths. - A step with **no workspace**, a **`Func` step**, or an entry whose workspace data `cache gc` has since collected is reported as `skipped`, with the reason given. It is never silently passed. `senro cache explain` will still call that step a clean `HIT`, because it is one: the key didn't change. That gap between "the key matched" and "the step actually reproduced" is the whole reason this command exists. The flags, bounds, and verdict table are in [Cache commands](/docs/cli/cache/). ## Where to go next - **[Cache keys](/docs/data/cache-keys/)**: exactly which components enter a key. - **[Shared cache](/docs/data/shared-cache/)**: making a cache entry reusable across machines. - **[Scratch caches](/docs/data/scratch/)**: the best-effort cache with none of these rules. - **[Secrets](/docs/secrets/)**: what a declared secret contributes to a key. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/cargo # `cargo`: Rust One unit per crate in a Cargo workspace. **Computes an affected set.** ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/cargo" ) verify.Expand("test", cargo.Crates()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("cargo", "test", "-p", u.Name)) }) ``` `Name` is the `[package]` name, which is exactly what `cargo test -p ` takes. ## What counts as a crate Every `Cargo.toml` with a `[package]` name, minus what a `[workspace]` table `exclude`s. **Not the `members` list**: a path dependency inside the workspace becomes a member whether or not `members` names it, so senro finds it either way. ## Where the edges come from `[dependencies]`, `[dev-dependencies]`, `[build-dependencies]`, and each of those under a `[target.]` table. The cfg expression is not evaluated, because the graph cannot know the target you will build for, so a target-specific dependency always draws its edge. A dependency resolves either way round: - by its `path` (and for `dep.workspace = true`, the path in the root's `[workspace.dependencies]`); - by its **crate name** matching a crate in the tree, which covers `[patch]` and `[replace]` redirecting a registry dependency at a local crate. ## No cargo needed The graph reads the manifests. `cargo metadata --no-deps` would be Cargo's own answer, but it needs cargo installed on whatever machine is planning the run; reading the manifests is the trade this graph makes deliberately. ## Worth knowing **The workspace manifest and lockfile affect everything.** A change to the root `Cargo.toml` or `Cargo.lock` runs every crate, because it can change what all of them compile against. **A file no crate owns runs everything**, for the same reason. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows. - **[Sharding](/docs/monorepo/partition/)**: splitting a large crate list across runners. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/cli # CLI This page lists every `senro` command, the conventions they all share, and the exit codes a script can rely on. Full usage, flags, and behavior for each command live on the three pages linked below. ```bash git clone https://github.com/xavidop/senro cd senro go build -o senro ./cmd/senro ``` senro supports Linux and macOS. Windows is not supported; see [Attach security](/docs/attach/security/) for why. ## Every command | Command | What it does | Detail | |---|---|---| | `senro run ` | Build a pipeline package, exec it, attach and render | [Run and watch](/docs/cli/run/) | | `senro attach` | Watch a live run, or replay a finished one from disk | [Run and watch](/docs/cli/run/) | | `senro ui` | Serve a browser view of a live run on loopback | [Run and watch](/docs/cli/run/) | | `senro shell` | Open a session inside a live run's step | [Run and watch](/docs/cli/run/) | | `senro runs` | List runs under `./runs`, newest first | [Workspaces and runs](/docs/cli/workspaces/) | | `senro cache gc` | Reclaim disk in the local content-addressed store | [Cache and verify](/docs/cli/cache/) | | `senro cache explain` | Why a `Pure()` step hit or missed the action cache | [Cache and verify](/docs/cli/cache/) | | `senro cache scratch` | List the scratch entries in the shared bucket | [Cache and verify](/docs/cli/cache/) | | `senro verify --recheck-pure` | Re-run cached `Pure()` steps and compare their digests | [Cache and verify](/docs/cli/cache/) | | `senro ws ls` | List a run's workspaces, or one workspace's files | [Workspaces and runs](/docs/cli/workspaces/) | | `senro ws pull` | Write a workspace's stored body out to a directory | [Workspaces and runs](/docs/cli/workspaces/) | | `senro ws diff` | Compare two runs' workspaces from their stored indexes | [Workspaces and runs](/docs/cli/workspaces/) | | `senro logs fetch` | Bring an archived run back from the shared cache | [Workspaces and runs](/docs/cli/workspaces/) | | `senro func check` | Report cgo in a `Func` step's dependency graph | [Workspaces and runs](/docs/cli/workspaces/) | | `senro help` | Print the full synopsis to stdout and exit `0` | | ## Conventions Four things hold across the whole CLI. **Help and version.** `senro help`, `senro -h`, and `senro --help` print the full command list to stdout and exit `0`. Subcommands don't repeat that help. `senro run --help` fails with `senro run: unknown flag "--help"`. `senro attach --help`, `senro shell --help`, and `senro ui --help` each print their own flag list, but to stderr. All three exit `2`. There's no `senro version` or `--version` flag. Both fail with `senro: unknown command`, exit `2`. **A run ID looks like `20260812T151058-540c8ca44b`.** It's a UTC timestamp plus a short random suffix. This is also the directory name under `runs/`. **Naming a run.** Any command that takes a run accepts a run ID, a path to a run directory, or nothing at all. Leave it out and senro uses the newest directory under `./runs`. Don't have an ID yet? `senro runs` lists what's there. `senro logs fetch` is the exception. Its `RUN` argument names a key in the shared store, not anything on your machine, so a path is refused there. **Credentials never come from a flag.** A TCP attach server's bearer token comes from `$SENRO_ATTACH_TOKEN`, never `--token`. A flag value would show up in `ps(1)` output for every user on the machine, and in shell history. TLS connections always verify against the system's root certificates. There's no `--insecure` flag. If you need a private CA, set `$SSL_CERT_FILE` or `$SSL_CERT_DIR` instead. See [Attach security](/docs/attach/security/). ## Choosing a renderer: `--ui` `senro run` and `senro attach` both take `--ui=auto|tui|plain|none`, defaulting to `auto`. | Value | What you get | |---|---| | `auto` | The terminal UI on a TTY, plain streaming lines otherwise | | `tui` | The terminal UI. A **hard error** on a non-TTY, never a silent downgrade | | `plain` | One line per event, no escape sequences | | `none` | No rendering at all; the exit code is still the run's | If you pass `--ui=tui` without a real terminal, senro fails with `senro: --ui=tui requires a terminal, but stdout is not a TTY`. This is intentional: in a CI log, the TUI's escape sequences would look like garbage, or worse, like a run that succeeded when it didn't. See [The TUI](/docs/attach/tui/). ## Exit codes These exit codes are a stable contract. A script wrapping `senro` can depend on these values meaning exactly this. Each code covers more than just "the run failed" though, so check the value, not a specific cause. | Code | Meaning | |---|---| | `0` | Success | | `1` | The run failed, or one of the other causes below | | `2` | Usage error, or one of the other causes below | | `78` | No trigger matched the event (`EX_CONFIG`): nothing to run | | `130` | Cancelled (`Ctrl-C`, or an external `SIGINT`/`SIGTERM`) | `senro ws diff` and `senro verify` always exit `0`, whether or not they find anything. A finding is an answer, not a failure. Besides a failed run, exit `1` also covers: - `func check` found cgo - `verify --fail-on-mismatch` found a step that did not reproduce its cached result - `cache gc` failed - `ws ls` could not load an index - `ws pull` refused a tar entry that escapes its destination - `logs fetch` could not reach the shared store, or was handed an object that did not match its digest - a write to stdout failed - an attach watch errored - the pipeline process was killed by a signal Besides a usage error, exit `2` also covers: - `go build` of the pipeline package failed - the pipeline process would not start - the attach socket would not connect - a cache record or workspace index is missing - `ws pull` or `logs fetch` found a non-empty destination without `--force` - `ws diff` could not compare a workspace - `logs fetch` found no shared cache configured, no such run in the store, or credentials the store refused - `func check`'s own analysis failed to run ### About `78` Exit `78` is neither success nor failure. It means the pipeline was asked whether an event was its business, and it said no. A dispatcher can tell this apart from a real success or failure without parsing any output. senro itself never makes this decision. The pipeline binary decides whether an event matches, and `senro run` passes its exit code through unchanged. On a `78`, senro also prints one line, `senro run: no trigger matched the event, so there is nothing to run (exit 78)`. Without that line, a bare exit `78` would look like a crash. See [Triggers](/docs/triggers/). ### About detaching Detaching (pressing `q` in the TUI) is not a failure in `senro attach`. Detaching doesn't stop the run, so the exit code reflects the run's actual outcome. If the run hasn't finished yet when you detach, the exit code is `0`. `senro run` works differently, because it owns the pipeline process. It waits for that process to exit and reports its exit code; `--ui=none` behaves the same way. If you want to walk away without waiting, start the pipeline binary yourself and watch it separately with `senro attach`. ## Where to go next - **[Reading a failed run](/docs/run/debugging/)**: the run directory and these errors, explained in context. - **[Attach](/docs/attach/)**: the protocol `senro attach` speaks. - **[Run options and outcomes](/docs/run/options/)**: the `senro.Run` call `senro run` wraps, and its full option list. --- > Source: https://xavidop.github.io/senro/docs/cli/cache # CLI: cache and verify This page covers the four commands for reading and reclaiming the cache: `senro cache gc`, `senro cache explain`, `senro cache scratch`, and `senro verify`. For the full command table and exit codes, see [CLI](/docs/cli/). **Where the cache root is**, for every `--cache-dir` flag below: the flag if you pass it, otherwise `$SENRO_CACHE_DIR`, otherwise `os.UserCacheDir()/senro`. This is the same root that `senro.WithCacheDir` sets for a library caller. It is not the run directory, which senro never deletes. ## `senro cache gc` ```bash senro cache gc [--max-size 50G] [--keep-failed 168h] [--dry-run] [--cache-dir DIR] ``` Reclaims disk space in the local content-addressed store, evicting least-recently-used entries first. - `--max-size` has no default. Running bare `senro cache gc` collects expired pins and unreferenced objects, and sweeps failed-run workspaces older than `--keep-failed`, but it won't evict anything just to free up size. The `50G` in the usage line above is an example, not a default senro sets. - `--max-size` takes a plain byte count, or a number with a `K`, `M`, or `G` suffix. It must be an integer: `1.5G` is refused rather than rounded, and so is a negative size. - `--keep-failed` defaults to `168h` (a week). Failed runs keep their workspace snapshots for that long, so the filesystem state you're debugging is still around. Only failed runs get this protection, which is why an old successful run's index can already be gone. - `--dry-run` reports what would be deleted, without deleting anything. Output is prefixed `dry run:`. A sweep prints a single summary line: objects deleted out of scanned, bytes freed out of total, entries evicted out of scanned, objects kept because they're pinned or scratch-referenced, pins expired, and leaked temp files swept. If a scratch cache save or a pipeline run is in progress against the same root, `gc` adds a note and deletes nothing. Just run it again once that finishes. There's no remote backend for this to sweep. See [Shared cache](/docs/data/shared-cache/). ## `senro cache explain` ```bash senro cache explain [--run RUN] [STEP] senro cache explain # every Pure() step and scratch cache the latest run touched senro cache explain build/test # one step's own cache key, hit or miss, field by field ``` Diffs a step's current cache key against its most recent recorded entry, so you can read exactly why a step missed the cache instead of guessing: ``` MISS measure key e126dad1 (previous 2ba03dd0) ✗ input_digests: greeting.txt 86c9c55c → 37e3516a ✗ workspace_digests: src 37931680 → d8ded6fe ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity, tool_versions, version unchanged ``` - **There's no `--cache-dir` flag here**, unlike `cache gc`. This command just formats what the engine already recorded to `/cache`. It doesn't re-plan or re-hash anything, so everything it reads lives inside the run directory. - Only steps marked `Pure()` (which opt into the action cache as safe to skip when their inputs haven't changed; see [Caching a step](/docs/data/caching/)) get a cache record, and only once they're actually attempted. A step skipped because a dependency failed never reaches the cache, so it has no record either. - A `STEP` argument can carry an attempt suffix (`build@2`); senro strips it before looking anything up, and it's never stored in a record. - A run with no `Pure()` step and no scratch cache says so, rather than printing nothing (`no cache activity recorded ...: no step declared Pure() and no scratch cache was mounted`). It still exits `0`. A [scratch cache](/docs/data/scratch/) is a mutable directory restored best-effort by key (think a package manager's download cache), never part of a `Pure()` step's cache key. Scratch caches don't emit events anywhere else, so this is the one place you can see their behavior: one line per cache, reporting `cold`, `cold, saved`, `restored (exact)`, or `restored from `. See [Cache keys](/docs/data/cache-keys/) for what each component means. ## `senro cache scratch` ```bash senro cache scratch [--pipeline NAME] [--limit N] [KEY-PREFIX] senro cache scratch # every pipeline sharing the store, newest first senro cache scratch --pipeline acme-ci # just this pipeline's entries senro cache scratch --pipeline acme-ci gomod- # and just the keys a RestoreKeys prefix would match ``` Lists what the **shared bucket** holds, which nothing else can show you. `cache explain` reports one run's view of its own scratch caches; this reports the store itself, so it answers "is anything in there at all" and "would my `RestoreKeys` prefix match it". ``` PIPELINE KEY SIZE STORED acme-ci gomod-9f2c1e8b 412.7 MiB 2026-08-24T11:02:19Z acme-ci npm-4d1a77c0 1.8 GiB 2026-08-23T18:40:02Z ``` - **Reads the same `SENRO_REMOTE_*` environment a run does**, so a shell already set up for the shared cache needs nothing extra beyond `SENRO_REMOTE_SCRATCH=1`. - **`--pipeline` is the namespace** entries are stored under, which is the name passed to `senro.New`. Omit it to see every pipeline sharing the bucket. - **Needs `s3:ListBucket`**, which nothing else in senro uses. A credential scoped to `GetObject` and `PutObject` alone reaches everything else and fails here. - **Buckets only.** An `oci://` target refuses, because the registry API cannot list by prefix. See [Sharing scratch caches](/docs/data/scratch-sharing/). ## `senro verify` ```bash senro verify --recheck-pure [--run RUN] [--rerun] [--step STEP] [--limit N] [--json] [--no-classify] [--keep] [--fail-on-mismatch] [--cache-dir DIR] [--local-class CLASS] senro verify --recheck-pure # what it WOULD re-run; nothing is executed senro verify --recheck-pure --rerun # re-run the latest run's cached Pure() steps senro verify --recheck-pure --rerun --fail-on-mismatch # exit 1 on a finding, for CI ``` `Pure()` is trusted, not enforced. Nothing sandboxes a step's network access, so if a step claims to be pure and then downloads something anyway, senro believes it, and serves that result to every future run with the same key. `senro verify` is how you check that claim. It puts a cached step back in front of the exact input its cache key recorded, runs it again, and compares the results. You have to name which check you want: a bare `senro verify` is a usage error. This also means adding a second kind of check later won't change what this invocation does. It reads `/plan.json` instead of rebuilding the pipeline. That means it needs no Go toolchain and no pipeline source, but it also can't re-resolve a pipeline that's been edited since the run. ### What it re-runs Only one run's cached `Pure()` steps, never their impure neighbors and never an upstream step. It doesn't need to re-run upstream steps, because a step's cache key already records the content digest of every workspace it mounted before it ran. `verify` restores that content straight from the store. `--limit N` checks only the first N steps in plan order (`0` means no limit). `--step` checks one named step. ### Nothing runs without `--rerun` This command exists because a `Pure()` claim might be false, so it doesn't assume the claim is safe either. Without `--rerun`, every step is just reported as `planned`, and nothing actually executes. When you do pass `--rerun`, four things limit the risk: - Every re-run happens in a throwaway directory tree restored from a content address. It never touches the run's own workspaces, and never touches your checkout. - A step that declares secrets is never re-run. The secret values live in the struct the pipeline passed to `senro.WithSecrets`, which isn't in the run directory and shouldn't be. - A step on a non-local executor is never re-run, since doing so would mean pulling an image or spinning up a pod. - A scratch cache is always realized cold, as an empty directory, since a scratch cache is never part of a cache key. A re-run also never writes an action cache entry, so it can't overwrite the entry it's comparing against. It does add objects to the content store when it snapshots a re-run's workspace; those are immutable, unreferenced, and get cleaned up by `senro cache gc` later. ### What "the same" means senro compares declared `Outputs`, mounted workspaces, and the exit code. Logs are never compared: a step's output legitimately contains timestamps, durations, PIDs, and temp paths, and flagging those would just produce noise you'd learn to ignore. A workspace is only compared when the `Needs` graph puts this step in a fixed order relative to every other step that mounts it read-write. A `ScopeRun` workspace is shared, so a post-step snapshot can contain whatever an unordered sibling step had written by that point, while an isolated re-run has no siblings at all. When that happens, the report names the sibling and explains why. The verdict is then decided by the step's declared outputs, since no sibling writes to those. ### Verdicts | Verdict | Means | |---|---| | `verified` | The re-run reproduced the entry exactly | | `mismatch` | The re-run differed from the entry **and a second re-run agreed with the first**: the step is deterministic and still did not reproduce what the cache holds, so it depends on something its key does not cover | | `nondeterministic` | The re-run differed from the entry **and from a second re-run of itself**: the step cannot produce the same bytes twice, so its disagreement is not evidence about purity | | `planned` | Would be re-run; `--rerun` was not given | | `skipped` | Cannot be checked, for a reason the report names | | `error` | The check itself broke, so nothing was learned | The split between `mismatch` and `nondeterministic` exists specifically to avoid false alarms. For example, an archive that embeds a build timestamp will disagree with its cache entry on every re-run, but that doesn't mean the step is impure. senro only spends a second re-run on a step that already disagreed once, so a clean pass costs just one execution per step. `--no-classify` skips this second re-run and merges both verdicts into `mismatch`. A caught step: ``` MISMATCH codegen key f33a005ee7e4 entry from run 20260813T105427-a248f55faa (hermeticity: trusted) ✗ output schema.gen cached 13080eb7 re-run 9639198b again 9639198b ✗ workspace src cached 80e6e936 re-run c00ef77f again c00ef77f declared inputs glob:*.go declared outputs file:schema.gen both re-runs agreed with each other and neither reproduced the entry, so this step depends on something its key does not cover: the network, a file outside its workspace, an environment variable it never declared in CacheEnv, or the clock ``` `cache explain` would report that same `codegen` step as a clean `HIT`, because it is one: the key never changed. That's exactly the kind of failure this command exists to catch. `hermeticity: trusted` appears on every entry senro writes today. It means `Pure()` was taken at its word. This label makes room for a future where purity is actually enforced, without needing a migration to tell old entries apart from new ones. Verifying a step doesn't upgrade its entry: a passing check is evidence about that one moment, not a permanent property of the cache entry. ### What it cannot check Each of these is reported as `skipped` with the reason, never silently passed: - A step that mounts no workspace. Its `Inputs` resolve against the working directory the pipeline ran in, which can't be reconstructed from a content address. - A `Func` step, since its body is compiled into the pipeline binary rather than described in the plan. - An entry whose workspace bodies a `cache gc` sweep has already collected. ### Exit codes and output Exits `0` whether or not it finds anything, just like `senro ws diff`. A finding is an answer, not a failed run. `--fail-on-mismatch` turns this from a report into a gate: it exits `1` if any step failed to reproduce its cache entry, or if the check itself broke. A `skipped` step never changes the exit code, since skips are a normal part of running this over a real pipeline. `--json` emits the whole report as one document; new fields may be added later, but existing ones won't change. `--keep` leaves the re-run trees on disk and prints where; without it, the report skips printing paths that are about to be deleted anyway. `--local-class CLASS` mirrors `senro.WithLocalClass` for the pipeline being verified. ## Where to go next - **[Caching a step](/docs/data/caching/)**: `Pure()`, `Inputs`, `Outputs`, `CacheEnv`. - **[Cache keys](/docs/data/cache-keys/)**: exactly what enters a key. - **[Scratch caches](/docs/data/scratch/)**: the mutable, best-effort cache `cache explain` also reports on. - **[Workspaces and runs](/docs/cli/workspaces/)**: `ws ls/pull/diff`, `logs fetch`, `func check`. --- > Source: https://xavidop.github.io/senro/docs/cli/run # CLI: run and watch This page covers the four commands that start a run or connect to one: `senro run`, `senro attach`, `senro shell`, and `senro ui`. For the full command table and exit codes, see [CLI](/docs/cli/). ## `senro run` ```bash senro run [--ui=auto|tui|plain|none] [--trigger-event PATH] [-- pipeline-args...] senro run ./ci # build, exec, auto-attach, render senro run ./ci -- --env=staging # flags after -- go to the pipeline, not to senro ``` `senro run` builds the named package with `go build` into a temporary binary, then runs it. If the pipeline registered an attach server (by calling `attach.Listen`, see [Run options and outcomes](/docs/run/options/)), senro attaches to it and renders the run exactly like `senro attach` would. - `--trigger-event PATH` is passed straight to the pipeline binary, which decides for itself whether the event applies. `PATH` can be `-` to read from stdin. `--trigger-event=` with no value is refused as a typo. If you want no event, just leave the flag off. - A pipeline that never calls `attach.Listen` still runs, and its exit code is passed through as-is. Its stdout and stderr are relayed when the UI mode is `plain` or `none`. Under `tui` (what `--ui=auto` picks on a real terminal), they're not connected at all, because the TUI owns the terminal. Pass `--ui=plain` if you need to see a non-attach pipeline's output. - **A Go toolchain must be on `PATH`.** Without one, `senro run` stops before it even builds, with `senro run: no Go toolchain found on PATH`. It'll tell you to build the binary yourself and run `./pipeline --tui` instead. Running an already-built binary needs no toolchain at all. - A package that fails to compile stops there too. You'll see `go build`'s own errors, followed by `senro run: go build ./ci: exit status 1`, and an exit code of `2`. - `senro run` sets `$SENRO_FUNC_PKG` on the pipeline process. This lets a `Func` step be cross-compiled for another platform without adding anything to the pipeline's source. An explicit `senro.WithFuncBuild` overrides it. See [A Func step off the coordinator](/docs/executors/func-remote/). `Ctrl-C` asks the engine to cancel gracefully instead of killing the process outright, so cleanup and `Always` handlers still get to run. If a pipeline ignores the request, senro kills it after five minutes so the CLI never hangs forever. ## `senro attach` ```bash senro attach [--pid | --run | --addr ] [--follow] [--tls] [--ui=auto|tui|plain|none] senro attach # auto-discover the one live run senro attach --run 20260812T151058-540c8ca44b --follow # tail a finished run from disk senro attach --addr 127.0.0.1:8443 --tls # a TCP attach server directly ``` Running `senro attach` with no flags discovers every live run registered on the machine, cleaning up any whose process has already died, and attaches to the one it finds. If there's more than one, it won't guess: it lists them all, with pid, run, pipeline, working directory, and start time, so you can pick one with `--pid`. That listing is printed to stderr as an error, with exit `2`. If there are no live runs at all, senro prints `senro: no live senro runs found` along with how to start one, also exit `2`. Which flag combination you need depends on whether the run you want is still live or already finished: ```mermaid flowchart TD Q{Which run?} Q -- "no flags" --> L{How many live runs?} L -- "0" --> E["error: no live senro runs found"] L -- "1" --> One["attach to it"] L -- "more than 1" --> List["list them all, exit 2:
pick one with --pid"] Q -- "--pid N" --> Pid["attach to that live run"] Q -- "--run ID" --> Live{"Live entry with that ID?"} Live -- yes --> LiveAttach["attach live
(handoff to disk on exit)"] Live -- no --> Disk["fall back to runs/ID on disk"] Q -- "--run ID --follow" --> Follow["replay from disk only,
no live lookup at all"] Q -- "--addr host:port" --> Addr["dial that TCP attach server directly"] ``` | Flag | Behavior | |---|---| | `--pid N` | A specific live run by process id. A pid that was registered and whose process has since died gets its own message, not "never existed" | | `--run ID` | Prefers a live entry with that ID, so you get the live-to-disk handoff on exit for free, and otherwise falls back to the recorded run under `runs//` | | `--follow` | Tails a run **from disk only**, no socket needed. Requires `--run`, and skips the live lookup entirely | | `--addr host:port` | Dials a TCP attach server directly, taking its token from `$SENRO_ATTACH_TOKEN` | | `--tls` | Says the `--addr` endpoint speaks TLS. Meaningless without `--addr`, and refused there | | `--ui MODE` | The renderer; see [CLI](/docs/cli/) | - `--pid` and `--run` can't be used together. `--addr` can't be combined with `--pid`, `--run`, or `--follow` either: `--addr` names an endpoint directly, while the other flags search for one. - `--run ` resolves `runs/` relative to your current directory. Run this command from the same place the pipeline ran. - On first contact, the client checks protocol versions. A matching major version is silent. A minor version mismatch prints a warning to stderr. A major version mismatch stops the connection instead of producing garbled output. Recorded runs and live sockets both implement the same interface internally, so the client renders them identically. Replaying a finished run isn't a second-class experience. See [Attach](/docs/attach/). ## `senro shell` ```bash senro shell [--pid | --run | --addr ] [--tls] [--tty] --step ID [-- cmd...] senro shell --step build # a session on the step's workspaces senro shell --step build -- cat build.log # one command, non-interactively senro shell --step build --tty # a real terminal ``` `senro shell` opens a session inside a step of a live run. You get its workspaces (read-only), at the same paths the step saw them, in the step's own working directory, on the step's own executor. Pair it with a breakpoint to pause a run before a step and inspect what it's about to run against. See [The shell](/docs/attach/shell/) for the full picture. Here are the essentials: - **`--step` is required.** A session always runs inside one specific step's workspaces, so there's no default to fall back on. - **No secrets reach a session.** Not the files, not the `SENRO_SECRET_*` variables, not any alias a step declared. A session can stay open indefinitely, and senro won't put a credential back on disk for that long. - **Pipes by default, a real terminal with `--tty`.** Without `--tty`, there's no prompt and no line editing: you type a line and press enter. With it, you get a real pty, job control, `^C` as a signal, and a window size that follows your own. - **`local` and `container` executors support a terminal; `ssh` does not.** On `ssh`, `--tty` is refused outright rather than silently downgraded (`executor_no_terminal`). Either way, a banner is printed to stderr, so a redirected stdout only captures what the session itself printed. - **It needs a live run.** For a finished run, use [`senro ws pull`](/docs/cli/workspaces/) instead. If you try `senro shell` against a finished run, it'll tell you and name the run. - **It reaches a remote run the same way `senro attach` does**: `--addr` plus `--tls`, with the token from `$SENRO_ATTACH_TOKEN`. There's no `--token` flag. - **A read-only attach refuses it.** If the attach server was started with `attach.Options{ReadOnly: true}`, a shell request gets a 403. A shared dashboard never hands out a command prompt. Everything after `--` runs instead of the default shell, and the session's exit code is that command's exit code. That makes `senro shell --step build -- test -f out/app` usable in a script, just like running the command directly. Refusals print a short reason and exit `1`, using the same vocabulary as [control operations](/docs/attach/control-ops/): `unknown_step`, `run_not_active`, `executor_no_shell` (an executor whose sandbox can't host a session at all; this build has none), `executor_no_terminal`, and `sandbox_failed`. ## `senro ui` ```bash senro ui [--pid | --run | --addr ] [--tls] [--port N] senro ui # the one live run on this machine senro ui --addr 127.0.0.1:9944 # a run reached through a port-forward senro ui --port 8730 # pin the loopback port instead of taking a free one ``` `senro ui` serves a browser view of a live run on loopback. It prints a one-time link to stdout and keeps running until interrupted. The page itself is a Go client compiled to WebAssembly, and it processes the run's events using the same logic the TUI does, so the two views never disagree about what a stream means. - It offers the same controls as the TUI: cancel, pause, resume, retry, skip, set and clear breakpoints, rerun-from, and accept/reject an analysis. `ws.snapshot` is available but has no button for it (see [the browser UI](/docs/attach/browser/#controls)). It does not offer `senro shell`. - The run's bearer token stays in this process and never reaches the browser. senro only accepts a control request from the page itself, checked by its session cookie and a matching `Origin`. - **Loopback only**, with no flag to widen that. `--port 0` (the default) picks a free port for you. - There's no `--follow` flag here. A finished run has no attach server to connect to; read one with `senro attach --run --follow` instead. See [The browser UI](/docs/attach/browser/) for more, including where the one-time link's nonce does and doesn't end up. ## Where to go next - **[Cache and verify](/docs/cli/cache/)**: `cache gc`, `cache explain`, `verify`. - **[Workspaces and runs](/docs/cli/workspaces/)**: `ws ls/pull/diff`, `logs fetch`, `func check`. - **[Reading a failed run](/docs/run/debugging/)**: what to do when one of these reports a failure. --- > Source: https://xavidop.github.io/senro/docs/cli/workspaces # CLI: workspaces and runs This page covers `senro runs` and the commands for reading what a run left behind: `ws ls`, `ws pull`, `ws diff`, `logs fetch`, and `func check`. For the full command table and exit codes, see [CLI](/docs/cli/). `--cache-dir` resolves the same way in every command below: the flag if given, else `$SENRO_CACHE_DIR`, else `os.UserCacheDir()/senro`. It only matters for a run whose pipeline used `senro.WithCacheDir`. ## `senro runs` ```bash senro runs [-n LIMIT] senro runs # the 20 most recent runs under ./runs, newest first senro runs -n 100 # more of them ``` Lists what's under `./runs` without your having to already know a run ID: each run's ID, pipeline name, status, when it started, and how long it took (or `running` for one still in progress). Every other command on this page and on [Cache and verify](/docs/cli/cache/) takes a `RUN` argument; this is where that ID comes from when you don't already have it pasted somewhere. Reads the same `events.jsonl` fold every other view of a run uses, so its status column can never disagree with what `senro attach --run` would show for the same run. An empty `./runs` prints `no runs under ./runs` and exits `0`; a directory with no `./runs` at all is a usage error, the same message `senro attach` and `senro ws ls` give you when they default to "the newest run" and find nothing to default to. Which of these to reach for depends on where the run lives and what you're trying to answer: ```mermaid flowchart TD Q{Is the run directory on this machine?} Q -- no --> F["senro logs fetch RUN:
pull the archived run down first"] Q -- yes --> W{What do you need?} F --> W W -- "which workspaces exist, or a file listing" --> L["senro ws ls"] W -- "the actual files a step left behind" --> P["senro ws pull"] W -- "what changed between two runs" --> D["senro ws diff"] ``` > **One note that applies to all three `ws` commands.** If a workspace's most recent state came > from a cache hit, it has no recorded file index; a cache entry only stores a body digest. > `ws ls` and `ws diff` will report this rather than erroring out. `ws pull` isn't affected, since > a body digest is all it ever needed. ## `senro ws ls` ```bash senro ws ls [--cache-dir DIR] [RUN] [NAME] senro ws ls # every workspace the latest run declared senro ws ls 20260812T151058-540c8ca44b src # one workspace's files, from its stored index ``` Lists a run's workspace snapshots: name, content digest, and size, read from the run's own event log. If you name a workspace, it lists that workspace's files, reading only the index object. That's what makes it instant regardless of workspace size: `ws ls` never pulls the actual snapshot body. - A workspace over 2 GiB is flagged `LARGE`, along with the command to list what's inside it. - `--cache-dir` only matters for the file-listing form. It points at the storage root that holds the index. - If a `senro cache gc` sweep already collected the index, `ws ls` reports the body digest that's still known and exits `2`. Only a failed run's workspaces are protected from being swept. ## `senro ws pull` ```bash senro ws pull [--cache-dir DIR] [--force] RUN NAME [DEST] ``` Writes a workspace's stored body out to a directory (default `./NAME`), so you can read whatever files a failed step left behind with ordinary tools. Unlike `ws ls`, both `RUN` and `NAME` are required here. Since `DEST` is also optional, a bare pair of arguments would otherwise be ambiguous. ``` pulled workspace "src" from /path/to/runs/20260812T151058-540c8ca44b into /tmp/broken body sha256:1468842d3dcfd3bda403aa4362f2c137b16694b16b055aa80a084a45731fc72d restored 7 entries, 71 B modes 0644 or 0755 for files, 0755 for directories, 0777 for symlinks: a snapshot carries the executable bit and nothing else mtimes 1970-01-01T00:00:00Z on every restored file and directory, fixed so a digest cannot depend on when a compiler ran dropped uid, gid, extended attributes, ACLs, hard links, devices, sockets and fifos are not stored by a snapshot at all, so they are not restored ``` Those last three lines print on every successful pull on purpose. Without them, the first thing anyone concludes from `ls -l` is that senro mangled their permissions, when it didn't. This normalization makes a workspace's digest depend on what a file actually contains, not which machine produced it, which is essential for every cache key downstream. See [Cache keys](/docs/data/cache-keys/). - The destination is replaced, not merged into. A merged tree wouldn't actually be the snapshot it claims to be. If the destination already holds anything, `ws pull` refuses with exit `2`; `--force` replaces it anyway. A destination that exists but isn't a directory is refused even with `--force`. - If a tar entry's path would escape the destination (a `..` component, an absolute path, a symlink pointing outside the workspace), the command refuses it: nothing is written, and it exits `1`. A snapshot senro produced can't contain such an entry, so seeing one means something else altered the body. Extraction stages files beside the destination first and only moves them into place once the whole body is read and verified, so a refusal never leaves a half-populated directory. - The reported file count comes from the extraction itself, since the ledger doesn't record one for a restored workspace. ## `senro ws diff` ```bash senro ws diff [--cache-dir DIR] [--json] RUN-A RUN-B [NAME] ``` Answers "what did this step actually do to the tree". It reads the two stored indexes and never opens a body, so diffing two multi-gigabyte workspaces only costs two small JSON reads. ``` + added - removed M content changed P mode changed K kind changed workspace "src" M VERSION 3 B -> 3 B 2d27fbdf -> 81db67b6 P build.sh 0644 -> 0755 + cmd/app.bin 0644 4 B 54034ac5 M current symlink target VERSION -> build.sh - docs/notes.md 0644 10 B 54d048ab 1 added, 1 removed, 3 modified, 1 mode, 0 kind, 2 unchanged ``` `P` means `chmod +x` with byte-identical content: a real change, and the one most easily missed just by looking. It's the only permission a snapshot tracks. `K` means the path changed kind entirely, like a file replaced by a directory or a symlink. It won't tell you what changed inside a file. Pull both sides with `senro ws pull` for that. - **It exits `0` whether or not there are differences**, unlike `diff(1)`. Exit `1` means "the run failed" everywhere else in this CLI, so it can't be reused here. Exit `2` means at least one named workspace couldn't be compared at all; the ones that could be are still reported. - With no `NAME`, every workspace both runs share is compared. A workspace that only one run has is reported, not silently dropped. Naming a workspace that one run doesn't have is a usage error, and the error lists what each run does have. The same happens if the two runs share no workspace at all. - A cache-restored workspace is the one case this command can't work around, since avoiding the download is the whole point of `ws diff`. It reports this on stderr with exit `2` and points you at `senro ws pull`: pull both sides and compare the trees yourself. - Two snapshots with the same body digest are reported as identical even without reading either index, since identical content addresses mean identical trees. `--json` emits one document: `{"workspaces": [...]}`. Each workspace has `name`, `a`/`b` (with `run`, `dir`, `digest`, `index`), `identical`, `changes`, `summary`, and where relevant a `note` (for a one-sided workspace) or `error` (couldn't be compared). Each change carries `path`, `status`, and an `a`/`b` entry with `kind`, `mode`, `size`, `digest`, and `link`. `mode` is an octal string (`"0644"`), since a plain number like `420` isn't something anyone would recognize. ## `senro logs fetch` ```bash senro logs fetch [--force] RUN [DEST] senro logs fetch 20260812T151058-540c8ca44b # into ./runs/20260812T151058-540c8ca44b ``` Fetches a run archived in the [shared cache](/docs/run/archiving/) back onto this machine. This is useful for a run whose CI runner no longer exists. `RUN` here is the run ID it was archived under, not a path, unlike every other run-taking command on this page. It reads the same environment variables that the original archiving run used: `SENRO_REMOTE_CACHE`, then either `SENRO_REMOTE_CACHE_ENDPOINT`, `SENRO_REMOTE_CACHE_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` for a bucket, or `SENRO_REMOTE_CACHE_USERNAME` and `SENRO_REMOTE_CACHE_PASSWORD` for a registry. Read permission is all it needs (`s3:GetObject` on the prefix, or `pull` on the repository), since the streams to fetch come from the run's own ledger, never from listing the store. What it writes is an ordinary run directory, so every other senro command that reads a run works on it too. `DEST` defaults to `./runs/RUN`, which is exactly the path `senro attach --run RUN` resolves on its own. The fetch prints that command when it's done: ``` fetched run 20260812T151058-540c8ca44b from s3 bucket acme-senro-cache at s3.eu-west-1.amazonaws.com into /home/you/ci/runs/20260812T151058-540c8ca44b ledger 7 steps, run failed logs 12 of 12 streams, 84.1 KiB read it with senro attach --run 20260812T151058-540c8ca44b ``` - If `DEST` is somewhere else, the printed command adjusts to match: a `cd` first, or a note naming the files if the path is one `senro attach --run` can't resolve on its own. Nothing here is guessed. - If the ledger names a stream the archive doesn't have, that's reported, not treated as an error. A step that never wrote to a stream never uploaded it, and there's no way to tell that apart here from an upload that didn't finish or one a lifecycle rule expired. - `DEST` is replaced, not merged into, just like `senro ws pull`'s. A directory holding one run's ledger and another run's logs would be an accurate record of neither. A non-empty `DEST` is refused with exit `2` unless you pass `--force`; a `DEST` that exists but isn't a directory is refused even with `--force`. - A failed fetch leaves nothing behind. Any directories it created get removed, so you'll never find an empty `runs//` sitting around for `senro attach` to mistake for a broken run. **The exit code describes the fetch itself, never the archived run.** Fetching the record of a failed build is still a success, exit `0`. It's `senro attach --run` that turns the archived run's own outcome into an exit code. | Condition | Exit | Because | |---|---|---| | No shared cache configured, or a variable missing | `2` | Nothing can be fetched until you set them; the message names them | | The run is not in the store, or the bucket is not there | `2` | The store answered, and no retry changes the answer. Check the run ID, the bucket/prefix or repository, or an expiry rule | | The store refused the credentials | `2` | Also an answer, and a different thing to fix: the key, the password, the session token, or the policy | | The store did not answer at all | `1` | Unreachable, timed out, or a 5xx. Nothing is shown to be wrong; try again | | An object did not match its digest | `1` | Refused rather than written: a log that is not what was uploaded is worse than no log | | Interrupted | `130` | What was already written stays and is readable, and incomplete | ## `senro func check` ```bash senro func check [--dir DIR] [packages...] senro func check # walk the module in the current directory senro func check --dir ./cmd/pipeline # walk a different module directory ``` Walks a module's dependency graph and reports every package that pulls in cgo, along with the import chain that pulled each one in. `--dir` defaults to `.`. Exits `1` if it finds any, `0` if the module is clean (`no cgo in the dependency graph of .`). A `Func` step running on an ssh host or in a container, targeting a different platform, gets cross-compiled with `CGO_ENABLED=0`. A cross-compiled binary can't link a C library for a platform it isn't building on. A container image is always Linux, so a macOS coordinator cross-compiles for every containerized `Func` step. senro refuses such a run before it even starts, using this same check, so the two can't drift apart. Run `func check` in CI to catch this before a deploy does. Steps on the coordinator, and steps on a target that matches the coordinator's own platform, ship the binary unchanged and are unaffected. Common causes the report will name: `os/user` (fix by building with `-tags osusergo`), `net` (`-tags netgo`), and any package wrapping a C library. See [A Func step off the coordinator](/docs/executors/func-remote/). ## Where to go next - **[Workspaces](/docs/data/workspaces/)**: what `ws ls`, `ws pull` and `ws diff` are reading. - **[Archiving a run](/docs/run/archiving/)**: what puts a run's logs in the store. - **[Reading a failed run](/docs/run/debugging/)**: these commands in a walkthrough. --- > Source: https://xavidop.github.io/senro/docs/concepts # Concepts Five ideas. Everything else in the docs is a detail of one of them. ## The vocabulary | Term | What it is | |---|---| | `Pipeline` | A name and a set of workflows: `senro.New("ci")` | | `Workflow` | A named group of steps: `p.Workflow("verify")` | | `Step` | One action: a command, or a registered Go function | | `Plan` | The validated, immutable graph that `Build()` resolves a pipeline into | | Event stream | The append-only record of everything a run did | **Step ids are unique across the whole pipeline**, not per workflow, because a plan is flat. The full surface is on [Steps](/docs/steps/). ## 1. You describe the work; senro runs it Your code builds a graph. Nothing in it executes anything, and your code never drives a step directly. ```go p := senro.New("ci") verify := p.Workflow("verify") verify.Step("test", exec.Command("go", "test", "./...")) if err := senro.Run(ctx, p); err != nil { // builds p, then executes the result log.Fatal(err) } ``` ```mermaid flowchart LR Def["Definition: plain Go"] -->|"Build()"| Plan["Plan: immutable snapshot"] Def -->|"senro.Run"| Exec["Execution"] Plan -->|"senro.RunPlan"| Exec Exec --> Stream["append-only event stream"] Stream --> Live["senro attach (live)"] Stream --> Disk["events.jsonl (post-mortem)"] ``` **Because `Run` builds first, mistakes surface before anything executes.** A dangling `Needs`, a duplicate step id or an empty command comes back as an error, not as a half-finished run. If you want to inspect the plan before running it, build it yourself: ```go plan, err := p.Build() // a validated snapshot; a later Step(...) does not touch plan if err != nil { log.Fatal(err) } fmt.Println("about to run", plan.Digest()) if err := senro.RunPlan(ctx, plan); err != nil { log.Fatal(err) } ``` The separation is what lets a **second process** watch the run: an attached terminal reads the same `Plan` the engine is executing, which it could never reconstruct from watching your function calls. ## 2. One event stream, and nothing else Every observable fact about a run (a step starting, finishing, being retried) is an `api.Event`, appended in order, never rewritten. ```jsonl {"seq":1,"type":"run.started","payload":{"pipeline":"ci"}} {"seq":2,"type":"step.started","step":"test"} {"seq":3,"type":"step.finished","step":"test","payload":{"state":"failed","exit_code":1}} {"seq":4,"type":"run.finished","payload":{"status":"failed"}} ``` The live TUI, the browser UI, and reading `events.jsonl` a week later all build their view by folding **that same list** through one function. So: - If the TUI shows `test` as failed, event 3 above arrived. There is no other way for it to know. - Replaying the file offline gives you byte-for-byte the screen the TUI showed at the time. - Your own code can fold the same events (`senro.WithSink`) and reach the same conclusions. That is why there is no "the UI is out of date" state in senro: there is nothing for it to be out of date with. [The event stream](/docs/run/event-stream/) lists the envelope and every event sent today. [Attach](/docs/attach/) is the protocol built on top of it. ## 3. Two kinds of step - **A command** (`exec.Command`) runs anywhere. `exec.Command("go", "test", "./...")` means the same thing on any machine and on any of the [four executors](/docs/executors/). - **Go code** (`senro.Func`) is a typed function registered by name, turned into a step by `senro.Func(name, params)`. See [Func steps](/docs/steps/functions/). Both are built, scheduled, retried, cached and handled by exactly the same code. A function's body is compiled into your binary and no plan can describe it, so running one elsewhere means moving the binary, not the plan. Off the coordinator, senro puts a copy of your binary on the SSH host, in the container or in the pod and re-enters it as a step child; [Func off the coordinator](/docs/executors/func-remote/) covers what that costs. ## 4. Two caches, for two different jobs Both are opt-in. Neither is on unless you ask. ### The action cache: skip a step entirely **What it is for:** not running work you have already run. Mark a step `Pure()` and declare its inputs, and senro hashes those inputs plus the command plus the environment into a key. On a second run with the same key, the step is **not executed at all** and its recorded outputs are restored from the store. ```go verify.Step("build", exec.Command("go", "build", "-o", "bin/app", "./cmd/app")). Pure(). Inputs(artifact.Glob("**/*.go"), artifact.File("go.sum")). Outputs(artifact.File("bin/app")) ``` Change one `.go` file and the key changes, so it rebuilds. Change nothing and it is a hit: the step ends in the state [`cached`](/docs/steps/states/), with `bin/app` restored. **Nothing is cached by default**, because `Pure()` is a promise only you can make. A tool that also SSHes into production is not pure, and senro cannot tell. See [Caching a step](/docs/data/caching/) and [What's in a cache key](/docs/data/cache-keys/). ### The scratch cache: start warm, but never be wrong **What it is for:** the mutable directories tools keep for their own speed. `~/.npm`, `~/.cargo`, a Go build cache. You want yesterday's copy if there is one, and you do not care if there is not. ```go gomod := senro.ScratchCache("gomod", senro.Key(`gomod-{{ hashFiles "go.sum" }}`), senro.RestoreKeys("gomod-")) verify.Step("test", exec.Command("go", "test", "./...")). Mount(gomod.At("/root/go/pkg/mod")) ``` A scratch cache is restored **best-effort by key** and **never enters a cache key**. `RestoreKeys` is the fallback: a `go.sum` change misses the exact key but still starts from the last module cache rather than from nothing. A miss costs you time and nothing else, and a stale entry cannot make a build produce the wrong answer, because nothing downstream is keyed on it. See [Scratch caches](/docs/data/scratch/). ### They share one store Artifacts, workspaces, cached results and staged binaries all live in one content-addressed store: everything is addressed by the digest of its content, normalized so the same bytes hash the same way on any machine. That is what makes the caches shareable. Point `SENRO_REMOTE_CACHE` at an S3-compatible bucket or an OCI registry and a fresh CI runner starts warm on what another machine already built. An unreachable store degrades the run to local disk; it never fails the run. See [Sharing a cache](/docs/data/shared-cache/). ## 5. Secrets are files, not strings You declare credentials as a typed struct, hand it to `senro.WithSecrets`, and a step asks for one by field name. What it receives is a **file path**, never the value. ```go type Config struct { RegistryToken secret.String `source:"env:NPM_TOKEN"` } cfg, err := mamori.Load[Config](ctx) // resolved once, before the run if err != nil { return err } setup.Step("install", exec.Command("pnpm", "install")). SecretEnv("NPM_TOKEN", "RegistryToken") // env var name, then the struct field senro.Run(ctx, p, senro.WithSecrets(cfg)) ``` Inside the step, `$NPM_TOKEN` holds a **path**, not the token, so the command reads the file: ```sh npm config set //registry.npmjs.org/:_authToken="$(cat "$NPM_TOKEN")" ``` A [func step](/docs/steps/functions/) does the same through `ctx.Secret("RegistryToken")`. **Why a file:** argv is world-readable in `/proc`, environment values leak into crash dumps and child processes, and both end up in logs. A file has an owner and a mode, and it disappears with the sandbox. **What senro refuses:** a plan that would route a resolved value through argv, an environment *value*, `WorkDir`, `Inputs`, `Outputs` or a mount is rejected **before the run starts**, not redacted afterwards. Whatever a step does print is redacted on the way out. See [Secrets](/docs/secrets/). ## Where to go next - [Quickstart](/docs/quickstart/): the shortest pipeline that exercises all of this. - [Steps](/docs/steps/): `Pipeline`, `Workflow` and `Step` in full. - [Executors](/docs/executors/): the four places a step can run. - [Monorepos](/docs/monorepo/): running only the units a change affects. - [Step states](/docs/steps/states/): why `recovered` is not `succeeded`. --- > Source: https://xavidop.github.io/senro/docs/steps/conditions # Conditions `When` prunes part of the graph at run start, so a pull-request run and a main-branch run can share one pipeline. ```go deploy := p.Workflow("deploy", senro.Needs("build"), senro.When(senro.Branch("main"))) deploy.Step("apply", exec.Command("sh", "-c", "make deploy")) senro.Run(ctx, p, senro.WithParams(senro.Params{"branch": currentBranch})) ``` `When` exists at three levels, all taking the same `Condition`: | Call | Gates | |---|---| | `senro.When(cond)`, passed to `Workflow` | Every step of the workflow | | `(*StepBuilder).When(cond)` | That one step | | `(*ExpandBuilder).When(cond)` | Every child of an expansion | **A condition is evaluated once, at the start of the run**, against facts the run already has, never against anything a step produces. ## The three conditions - **`senro.Branch(name)`**: true when the run's `"branch"` parameter equals `name`. - **`senro.ParamIs(name, value)`**: true when the named run parameter equals `value`. `Branch` is this with the parameter fixed to `"branch"`. - **`senro.EnvIs(name, value)`**: true when an environment variable equals `value`, read from **the coordinator's own process**, not the step's. Conditions run before any sandbox exists. Parameters come from `senro.WithParams`; see [Run options and outcomes](/docs/run/options/), or [Triggers](/docs/triggers/) for where `branch` comes from when an event started the run. **There is deliberately no `And`, `Or` or `Not`.** Calling `When` more than once, at any level or mix of levels, already means AND. ## senro does not read `git` for you `currentBranch` above is whatever your pipeline binary decided, read from `git` or from CI's environment. senro deliberately does not shell out to `git` itself: a plan depending on ambient repository state would behave differently in a container or a detached checkout. ## A gated step is skipped, not failed A step whose conditions are not all true settles as `skipped_condition`, and its dependents settle the same way, cascading transitively. The difference from a failure is the run's result: - A run made entirely of `skipped_condition` steps still reports `succeeded`, which leaves a pull request's run green when its `Branch("main")`-gated deploy does not fire. - Dependents of an actually failed step get `skipped_upstream_failed` instead, which makes the run `partial` or `failed`. - **`ContinueOnError` does not rescue a `skipped_condition` dependent.** It promises a dependent survives a failure, not that it runs against output that was never produced. See [Step states](/docs/steps/states/) for the whole taxonomy. ## Where to go next - **[Step states](/docs/steps/states/)**: `skipped_condition` and how it propagates. - **[Fan-out](/docs/monorepo/fan-out/)**: `When` on a whole expansion. - **[Run options and outcomes](/docs/run/options/)**: `senro.WithParams`, which `Branch` and `ParamIs` read. - **[Triggers](/docs/triggers/)**: run parameters an event supplies. --- > Source: https://xavidop.github.io/senro/docs/executors/containers # Containers `container.Image(ref)` targets a workflow at a container on the coordinator's own container daemon. Every step runs as its own container. The step's command becomes the container's command, and its exit code becomes the container's exit code. ```go import ( "github.com/xavidop/senro" "github.com/xavidop/senro/exec" "github.com/xavidop/senro/executor/container" ) node := container.Image("node:22-bookworm-slim") setup := p.Workflow("setup", senro.On(node)) setup.Step("install", exec.Command("pnpm", "install", "--frozen-lockfile")) ``` ## What the machine needs You need a container runtime's daemon running on the same machine as the coordinator, reachable over a local unix socket. Building a pipeline doesn't need a daemon at all; running one does. - **Docker, Podman, colima, OrbStack, and Rancher Desktop need no setup.** senro checks `DOCKER_HOST` first if you set it, then each of their well-known socket locations in turn. If it finds none, the error lists every path it tried and shows how to point at a daemon explicitly, for example `DOCKER_HOST=unix:///path/to/your.sock`. - **A remote daemon won't work.** senro refuses a `DOCKER_HOST` that names `tcp://`, because every mount is a bind mount of a directory the coordinator owns, and a daemon on another host can't see that directory. - **containerd on its own isn't enough.** The runtimes above all speak the Docker Engine API over their socket. containerd speaks a different, gRPC-based API that senro doesn't support. ## Pull from a private registry ```go builder := container.Image("ghcr.io/acme/builder:v3", container.RegistryAuth("acme-ci", "GHCRToken")) ``` `container.RegistryAuth(account, field)` authenticates the pull, so a workflow can run on an image in a private registry without a `docker login` on the machine. - **`account` is the registry account's name, never a credential.** Use `AWS` for Elastic Container Registry, `oauth2accesstoken` for Artifact Registry, or a login for `ghcr.io`. It's recorded in the plan exactly as written, and it can be empty for a registry whose token endpoint takes the password alone. - **`field` is a field name on the struct you handed to `senro.WithSecrets`**, the same way `SecretEnv`'s second argument works ([Secrets](/docs/secrets/)). If you type a password here instead of a field name, senro can't resolve it, and the run is refused right at the start rather than writing a password into `plan.json`. - **Don't pass a resolved secret's value in either argument.** Both arguments are recorded verbatim in the plan and in the executor's instance key (what senro uses to tell whether two `On(...)` targets are the same running executor or two separate ones), and neither place is redacted, so senro refuses this too. - **The value reaches exactly one place**: the `X-Registry-Auth` header of the pull. It never touches argv, an environment value, the plan, a cache key, an event, or a log. It's registered with the run's redactor like every other resolved secret. - **senro runs no credential helper.** It doesn't read `~/.docker/config.json` and doesn't contact any metadata service. If another service issues the credential, resolve it into your configuration struct first. - **It doesn't affect the step's cache key**, which already carries the resolved image digest, so two credentials that fetch the same bytes are the same step, and folding the credential in would make a rotated token invalidate every cache entry for that image. It *does* affect the executor's instance key, so one image under two credentials stays two separate executors with two separate pulls. - **A registry credential on any other executor is refused at `Build()`.** Only the container executor pulls an image itself. A pod's image is pulled by its node from an `imagePullSecret` in the namespace, and ssh and local steps pull nothing at all. When a pull is refused, the error tells you which of two things happened: either the registry wouldn't serve the image and the pipeline declared no credential, or the credential was presented for a given account and rejected. Both count as infrastructure failures, so [`retry.OnInfra()`](/docs/steps/retries/) retries them. ## Run as a different user ```go container.Image("debian:bookworm", container.User("0:0")) ``` `container.User` takes Docker's own `uid:gid` or `name` spelling. By default a container step runs as the coordinator's own uid and gid, not root. That's because a root step leaves root-owned files behind in the run directory, and the coordinator can't clean those up without sudo. If a step genuinely needs root (installing OS packages, for example), declare a user for it and expect that consequence. A declared `User` is part of the step's cache key. The default isn't, since it just names the coordinator's identity rather than anything about the pipeline itself. ## What runs where, at a glance | Behavior | On this executor | |---|---| | Image reference | A tag is fine; it resolves against the daemon once per run | | Workspaces | Bind mounts of the coordinator's own directories; nothing is carried | | `senro.RO` mounts | **Genuinely enforced**, as a read-only bind mount | | Secrets | The step's secret directory, bind-mounted read-only at `/run/senro/secrets` | | Scratch caches | Supported | | `Func` steps | Supported; the binary is bind-mounted, never transferred | | `senro shell` | Supported, with or without `--tty` | | stdout and stderr | Kept apart | | Environment | The image's own environment, with the step's declared variables on top | | Cache class | The platform and the **resolved image digest**, plus a declared `User` | ## The image reference resolves once per run The reference is recorded in the plan exactly as you wrote it, and resolved against the daemon once per run. The digest, not the tag, is what enters the cache key and `step.started`'s `executor_class`. That way, if a tag moves, the cache class changes too, instead of silently reusing an entry computed from different bytes. senro resolves once per run rather than once per step, so a tag can't move mid-run and split one executor into two classes. It resolves at run time rather than at build time, so a plan's identity doesn't depend on one machine's daemon cache. ## Workspaces are bind mounts A mounted workspace is the coordinator's own directory, bound into the container at the path you declared. Nothing is copied in either direction, which makes this the cheapest non-local executor for a large tree. ```go src := senro.Workspace("src", senro.Scope(senro.ScopeRun)) build := p.Workflow("build", senro.On(node)) build.Step("compile", exec.Command("pnpm", "build")). Mount(src.At("/src", senro.RW)). WorkDir("/src") ``` - `senro.RO` is a real read-only bind, so a write through one fails immediately at the write, not afterwards. See [the enforcement table](/docs/executors/#read-only-mounts-are-enforced-on-two-of-the-four). - Excluded paths (`.git` and `node_modules` by default) still sit on disk beside the mount, because the mount *is* your directory: only the snapshot leaves them out. [Workspaces](/docs/data/workspaces/) covers what a snapshot carries. - [Scratch caches](/docs/data/scratch/) work here too, bind-mounted like a workspace with no transfer to pay, unlike on the Kubernetes and SSH executors. They can also be shared **between machines** through the bucket with `SENRO_REMOTE_SCRATCH`, which is what stops a cold CI runner installing dependencies from scratch; see [Sharing scratch caches](/docs/data/scratch-sharing/). A container step counts as running on the coordinator's filesystem, so it may share one cache with a Kubernetes or SSH step **within a run** as long as a `Needs` orders the two ([handing one over](/docs/data/scratch/#handing-one-between-a-remote-step-and-a-local-one)). Unordered, that combination is still refused. ## Secrets never reach the container's configuration A secret's file is the same file the local executor writes. It's bind-mounted read-only into this step's container at `/run/senro/secrets`, with its path in the step's environment. It's never passed with `-e`, `--env-file`, or a build argument. The path is fixed rather than configurable, since a step reads it from `SENRO_SECRET_` anyway, and a single fixed path is simpler to audit. Each step's secret directory is bound only into that step's own container, so a step running at the same time can't reach it through the filesystem. See [Secret channels](/docs/secrets/channels/) for the full comparison. ## The command is arguments to the image's `ENTRYPOINT` senro sends the step's command as the container's `Cmd` and leaves `ENTRYPOINT` alone. An image with a wrapper entrypoint that execs its arguments behaves the way you'd expect. One that ignores or rewrites its arguments changes what your step actually runs. The Kubernetes executor works differently here: there, the command replaces the entrypoint. The step's environment is the image's own environment with your declared variables layered on top, computed the same way the daemon computes it. That means the cache key's environment component reflects what the step actually receives. ## `Func` steps cost nothing to stage The daemon runs on the coordinator's own machine, so the pipeline binary is already there. senro binds it read-only at `/senro/bin/senro-sha256-` in the one container that runs it. Nothing is copied, whether it's the first step or the hundredth. An ordinary `exec` step in the same image gets no such bind. A container image is Linux, so a macOS coordinator has to cross-compile for every func step. See [Func steps off the coordinator](/docs/executors/func-remote/) for that, for the `ENTRYPOINT` trap it shares with an `exec` step, and for the cgo constraint. ## What is not here - **Credential helpers and `~/.docker/config.json`.** Declare a private registry's credential with [`container.RegistryAuth`](#pull-from-a-private-registry) instead, and it's resolved along with everything else. - **Remote daemons**, for the bind-mount reason described above. - **Resource limits, networks, and other daemon-level tuning.** `container.Image` takes an image, a `User`, and a `RegistryAuth`, and nothing else. If a coordinator is killed and leaves an orphan container behind, you can find it: every container senro creates is labelled with the run, the step, and the attempt. ```sh docker ps -a --filter label=senro.run= ``` > senro defaults to the coordinator's own uid instead of root, because root inside a bind-mounted > container directory means root-owned files left behind in your run directory afterwards. --- > Source: https://xavidop.github.io/senro/docs/attach/control-ops # Control operations Attach is not read-only: a connected client can ask the engine to *do* something. This is the reference for the wire shape, the eleven operations this build implements, and the refusal codes. ## Frame shape A control request and its response are one JSON `api.Frame` each, exchanged over a single endpoint, `POST /api/control`, correlated by `id`: ```json {"v":1,"kind":"req","id":"c7","type":"step.retry","payload":{"step":"build"}} {"v":1,"kind":"res","id":"c7","ok":true} ``` - A response never carries a payload. It only says whether the operation was accepted, and if not, why, in `error`. What actually happened shows up in the event stream: that's the only record of it. - A request's payload has exactly one key. A request with any other key is rejected outright. `Frame` is plain JSON on purpose, so you can debug it with `curl` alone. ```mermaid flowchart LR client["Client"] -->|"req: step.retry"| engine["Engine"] engine -->|"res: ok:true / ok:false"| client engine -.->|"what actually happened"| stream["Event stream"] ``` `POST /api/control` is one of six routes: ``` GET /api/state a bare RunState GET /api/plan the resolved plan, the same JSON as the run directory's plan.json GET /api/logs/{step} raw log bytes for one step's one stream GET /api/stream?from=N bare Event values as newline-delimited JSON, resuming at seq N POST /api/control the Frame request/response above POST /api/shell an interactive session on a live step, on a hijacked connection ``` Subscribing and reading logs are not control operations. A [shell session](/docs/attach/shell/) is `POST /api/shell` instead, since it hijacks the connection and can't be expressed as a frame. Note that the [stream's](/docs/attach/#snapshot-then-subscribe) resume parameter is `from`, not `from_seq`. ## The eleven operations These are the only eleven operations this build supports. Any other `type` value is refused with `unknown_op`. Each has a declared constant in [`api`](/docs/run/api/). | Operation | Argument | What it does | |---|---|---| | `run.cancel` | none | Cancels the run. The TUI's `c`/`Ctrl-C` and the CLI's own signal handling (`SIGINT`/`SIGTERM`) issue this, best-effort, with a short timeout | | `step.retry` | `step` | Retries the named step in place, in the same run, incrementing its attempt count | | `step.skip` | `step` | Takes a step out of the run. It settles as `skipped_manual` without being dispatched, and so does every step that needs it. See below | | `breakpoint.set` | `step` | Stops the run before a step. See [Breakpoints](#breakpoints) | | `breakpoint.clear` | `step` | Releases a held step | | `run.rerun_from` | `step` | Re-runs a step and everything downstream of it, in a run that is still live. See below | | `run.pause` | none | Stops the run dispatching anything new. See below | | `run.resume` | none | Lets it dispatch again | | `analysis.accept` | `id` | Accepts a [failure analyzer](/docs/analyzers/custom/)'s proposal and performs its remedy | | `analysis.reject` | `id` | Rejects it; nothing is performed | | `ws.snapshot` | `step` | Captures that step's workspaces now, for inspection. See [Forcing a snapshot](#forcing-a-snapshot) | - `step.retry` is a *live* operation. It's different from the [`Retry` policy](/docs/steps/retries/) you declare at build time: this is something a human or script asks for after a step has failed and exhausted (or never had) an automatic retry. It dispatches one bare attempt directly, so it does not re-run the step's `OnFailure`/`Always` handlers. The ledger is the run's permanent, append-only event record, and it logs this with a `handler.superseded` event. - An analysis `id` is the one carried by an `analysis.proposed` event, in the form `@`. It identifies a proposal, not a step, so a client can't accidentally approve a proposal for one step and have it retry another: the engine always retries the step its own record says the proposal was about. - Accepting emits `analysis.applied`; rejecting emits `analysis.rejected`. Once a proposal is settled, a second decision on it is refused, so two operators both pressing `a` can't retry a step twice. - Accepting a proposal grants no more power than a client already has. The only remedy this build can apply is a retry, using the exact same code path as `step.retry`, including its refusals. - Every accepted operation also emits a `control.applied` lifecycle event carrying the client's identity, so the event stream doubles as an audit trail. ## Refusals are answers, not errors A refused operation comes back as `ok:false` with a short, machine-readable reason: a code, not a prose message, so a client can branch on it reliably across releases. An operation either applies completely or is refused. There's no half-applied state. | Reason | Meaning | |---|---| | `unknown_op` | This build doesn't implement that operation | | `run_finished` | Nothing is left to act on the request; the run is over | | `already_cancelled` | The run is already cancelling | | `run_not_active` | The run is being torn down, so no new work may start | | `missing_step` | The operation needs a `step` argument and got none | | `unknown_step` | No such step in this run's plan | | `step_running` | That step (or, for `run.rerun_from`, something in its closure) is mid-attempt | | `step_not_failed` | `step.retry` only applies to a step that failed | | `step_settled` | That step already ran: `step.skip` can't un-run it, and `ws.snapshot` would be a second, later answer to a question its own snapshot already answered | | `step_not_settled` | `run.rerun_from` has nothing to re-run: that step hasn't run yet | | `breakpoint_exists` | A breakpoint is already armed on that step | | `no_breakpoint` | There's no breakpoint on that step to clear | | `already_paused` | The run is already paused | | `not_paused` | The run isn't paused, so there's nothing to resume | | `missing_proposal` | An analysis operation needs an `id` argument and got none | | `unknown_proposal` | No proposal in this run carries that id | | `proposal_settled` | Somebody has already accepted or rejected that proposal | | `no_remedy` | That proposal asked for nothing this build can perform, so there is nothing to apply | | `no_workspace` | `ws.snapshot` has nothing to capture: that step mounts no workspace, or only cluster-backed ones | | `snapshot_failed` | The capture itself failed. Not a refusal: nothing was wrong with the request | ## Forcing a snapshot `ws.snapshot{step}` captures every workspace the named step mounts, right now, and emits one `ws.snapshot` event per workspace so you can `senro ws pull` the digest and look at the files. It only works on a step that has **not run yet**. The main use case is a step held at a breakpoint: the run has stopped there, so nothing is writing to it, and what you capture is exactly what the step is about to receive. - A step **mid-attempt** is refused with `step_running`: it's actively writing to the directories the capture would read, and a half-written snapshot would be worse than no answer. - A step that has **settled** is refused with `step_settled`. Its own snapshot, taken when it settled, already records what it produced (failures included), and that's the digest `senro ws` reports. - A step with **no workspace**, or only [claim-backed](/docs/executors/kubernetes/) ones whose content lives in the cluster, is refused with `no_workspace` rather than silently accepted as a no-op. **A forced capture is never evidence.** It doesn't enter any cache key, doesn't replace a workspace's recorded state, and doesn't change the plan or what the step's own snapshot will say. The event carries `"forced": true`, and `senro ws ls`, `ws pull` and `ws diff` skip it, because those commands report what the run actually produced. The digest stays pinned for the life of the run, so you can still pull the snapshot later. The operation name matches the event it causes, just as `breakpoint.set` causes `breakpoint.hit`. Operations and event types are never mixed on the same channel, so there's no ambiguity. The capture doesn't block the scheduler's own loop, because a workspace can be gigabytes, and control requests are served one at a time. The response only arrives once the capture is finished, so `ok:true` means the snapshot is already in the ledger, not just that the request was accepted. While a capture runs, the step is treated as busy: a second `ws.snapshot`, a `step.retry`, or a `step.skip` on it will all get `step_running`. ## What happens below a skipped step `step.skip` settles the named step as `skipped_manual`. Every step that depends on it, directly or transitively, also settles as `skipped_manual`, not `skipped_upstream_failed`, and `ContinueOnError` doesn't rescue them. That is the rule the engine applies to a step skipped by a `When` condition too. senro distinguishes two ways a step can stop its dependents: - **The upstream failed.** Dependents are `skipped_upstream_failed`, the run rolls up as `partial`, and `ContinueOnError` is the author's explicit "run anyway" escape hatch. - **The upstream never ran, but nothing broke.** Dependents inherit the same skip state, the run rolls up clean, and `ContinueOnError` doesn't apply here: it promises a dependent survives a *failure*, not that it can run against output that was never produced. A manual skip is always the second case. It doesn't poison the whole graph. Only the transitive dependents are affected, unrelated branches still run to completion, and the run finishes `succeeded`. ## Breakpoints `breakpoint.set{step}` stops the run *before* a step runs. `breakpoint.clear{step}` releases it. `run.cancel` is the only other way out: a run held at a breakpoint waits indefinitely otherwise. Nothing inside the engine blocks while it waits. The scheduler simply declines to dispatch that step, and no parallelism slot is held for it, so the rest of the run keeps making progress elsewhere. - The moment the scheduler first withholds the step, it emits `breakpoint.hit` once, naming the client that armed it. That's what distinguishes a held step from one still waiting on its dependencies: a held step has no `step.started`, no `step.finished`, and no other state change. Clients fold this into `StepState.Paused`, and both shipped renderers show it. - A breakpoint only gates *scheduling*, so it doesn't intercept `step.retry`, which dispatches an attempt directly. It does work together with `run.rerun_from`: arm the breakpoint first, then rerun. - A held step is the one state [`ws.snapshot`](#forcing-a-snapshot) is for: nothing is writing the step's workspaces, so a capture taken then is exactly what the step is about to be given. ## Pausing the whole run `run.pause` stops the run from dispatching anything new. `run.resume` releases it. Neither takes an argument, and `run.cancel` is the only other way out: a paused run otherwise waits indefinitely. **A pause is not a breakpoint.** A breakpoint withholds one named step; a pause withholds the whole plan. The mechanism is the same either way: the scheduler computes what it would dispatch next and declines, so a paused run answers control requests just as fast as a busy one. A step already mid-flight is **not** touched. It runs to completion and settles normally, and its `step.finished` event lands in the stream even while the run is paused. senro has no way to suspend a running command: there's no checkpoint, and the sandbox, log files, and (on containers) the daemon process are all still live. The only thing "pause the running step" could actually mean is *kill it*, and a pause that killed work would really be a cancel, not a pause. For the same reason, a step's automatic retry policy keeps running under a pause: that's the step's own execution continuing, not new work starting. So the promise is the narrow one, **no new work is dispatched**: - Settling isn't dispatching, so it isn't suppressed by a pause. If a step fails while the run is paused, its dependents still settle as `skipped_upstream_failed` right away. If you paused in order to retry that step, use `run.rerun_from` to put the dependents back. - `step.retry` isn't blocked by a pause, just as it isn't by a breakpoint: it dispatches an attempt directly. A pause still leaves room for `step.retry`, `step.skip`, and `run.rerun_from`. - `run.rerun_from` works the other way, because it hands its steps back to the *scheduler* rather than dispatching directly. Ask for a rerun while paused, and it's queued, starting once you resume. - You can tell a paused run from a hung one by the `control.applied` event recording the accepted `run.pause`. There's no second event, unlike `breakpoint.hit`: a pause takes effect the instant it's accepted, with no gap between arming and acting. Clients fold this into `RunInfo.Paused`, and the TUI's footer reads `run: paused`. ## Rerunning part of a live run `run.rerun_from{step}` puts the named step and its transitive dependents back to pending, and hands them to the scheduler. They run exactly as they did the first time: same parallelism limits, retry policy, timeouts, cache lookup, and handlers. Nothing outside that set is touched. - Each re-run step announces itself with `step.retried` under a *new* attempt number. Attempt numbers never restart, and both a step's events and its log files are filed under that number (`runs//logs///{stdout,stderr}`), so the previous run's record stays intact. - The cache isn't bypassed, and it doesn't need to be. Only a `Pure` step consults the action cache, and a pure step's output is by definition a function of its inputs, so serving the cached result *is* re-running it. If you want a step's side effects repeated, don't mark it `Pure`. - Handlers run again too, because a rerun is a genuine second execution of the step. The previous pass isn't rewritten: `handler.superseded` just marks it as no longer describing the step's current outcome. ## Version negotiation On first contact (`GET /api/state`, before subscribing), a client compares its protocol version to the engine's. Matching major and minor versions: nothing happens. Matching major but different minor: a one-time warning, then it proceeds. Different major version: an error naming which side is out of date, instead of a confusing JSON decode failure: ``` api: engine speaks protocol v2.0, this client speaks v1.0: upgrade your CLI ``` ## Where to go next - **[Security](/docs/attach/security/)**: who is allowed to issue a control operation at all. - **[The TUI](/docs/attach/tui/)**: the keys that map to these operations. - **[The browser UI](/docs/attach/browser/)**: which operations `senro ui` offers, and when. - **[The shell](/docs/attach/shell/)**: `POST /api/shell`, the sixth route. --- > Source: https://xavidop.github.io/senro/docs/steps/settings # Env, dir & timeout Four settings every step has, whatever it runs and wherever it runs. Every method returns the same builder, so calls chain. ```go verify.Step("deploy", exec.Command("./deploy.sh", "prod")). Env("DEPLOY_ENV", "prod"). WorkDir("./infra"). Timeout(5 * time.Minute). ContinueOnError() ``` | Method | What it does | |---|---| | `Env(key, value)` | One environment variable. Call it again for more. | | `WorkDir(dir)` | The directory the command runs in. | | `Timeout(d)` | Bounds **one attempt**, not the whole retry sequence. | | `ContinueOnError()` | Dependents run even if this step fails. | ## `Env` One pair per call: ```go verify.Step("test", exec.Command("go", "test", "./...")). Env("CGO_ENABLED", "0"). Env("GOFLAGS", "-count=1") ``` It is not variadic, because a variadic run of pairs has an arity the compiler cannot check: `Env("A", "1", "B")` would compile and mean nothing. ### Your step gets exactly what you declared **`Build()` adds nothing to a step's environment, not even a `PATH`.** Two developers on the same commit get the same plan, because nothing about their shells leaks into it. That matters most on the exact field a [cache key](/docs/data/cache-keys/) is computed from. Search-path defaults are the executor's job instead: - **The local executor supplies the coordinator's own `PATH`** to a step that declares none, and nothing else from the parent environment. That is why `exec.Command("go", "test", "./...")` finds `go` with no `Env` call at all. - **Declaring a `PATH` yourself replaces that fallback**, it does not add to it. If you set `PATH=/opt/toolchain/bin`, `go` is no longer on it. ### Never put a credential in `Env` Use [`SecretEnv`](/docs/secrets/) instead, which delivers a **file path** rather than a value. senro refuses a plan that would route a resolved secret through an environment value, so this is not a style preference. ## `WorkDir` ```go verify.Step("build", exec.Command("pnpm", "build")).WorkDir("./web") ``` There is no `cd` in a step, because [nothing is shell-interpreted](/docs/steps/#execcommand-interprets-no-shell). `WorkDir` is how you change directory. When a step mounts a [workspace](/docs/data/workspaces/), `WorkDir` is usually the mount path: ```go deploy.Step("build", exec.Command("./build.sh")). Mount(src.At("/repo", senro.RW)). WorkDir("/repo") ``` ## `Timeout` bounds one attempt A step with `Retry(3, ...)` and `Timeout(5*time.Minute)` can take **fifteen minutes** across three attempts. The bound is per attempt, not per step. An attempt that outlives the bound settles the step as [`timed_out`](/docs/steps/states/). > **One exception, on the coordinator.** Nothing can force a Go function to return, so a > [`senro.Func`](/docs/steps/functions/) step running locally that ignores its context keeps > running past the deadline and is merely *filed* as `timed_out` when it eventually finishes. > > Off the coordinator the function has a process of its own, which ends itself at the deadline. > Declare a `Timeout` on every remote func step. ## `ContinueOnError` is for advisory steps `ContinueOnError()` says: if this step **fails**, its dependents should run anyway, against whatever it did produce. Linting is the usual case. ```go verify.Step("lint", exec.Command("golangci-lint", "run")).ContinueOnError() verify.Step("report", exec.Command("./collect-report.sh")).Needs("lint") // runs either way ``` It is not a general rescue, and two limits catch people out: - **It does not apply when the upstream never ran.** A dependent of a `skipped_condition` or `skipped_manual` step is skipped the same way, because nothing failed and nothing is being excused. `ContinueOnError` promises a dependent survives a *failure*, not that it runs against output that was never produced. - **It still changes the run's rollup.** Without it, the failure makes the run `partial` or `failed`. Both are on [Step states](/docs/steps/states/). ## The rest of the builder | I want to... | Call | Page | |---|---|---| | Run this after another step | `Needs` | [Ordering](/docs/steps/ordering/) | | Try again when it breaks | `Retry`, `RetryPolicy` | [Retries](/docs/steps/retries/) | | Clean up or collect logs afterwards | `OnFailure`, `Always` | [Failure handlers](/docs/steps/handlers/) | | Skip it unless something is true | `When` | [Conditions](/docs/steps/conditions/) | | Give it files, and keep what it wrote | `Mount`, `NoSnapshot` | [Workspaces](/docs/data/workspaces/) | | Skip it when nothing changed | `Pure`, `Inputs`, `Outputs`, `CacheEnv` | [Caching a step](/docs/data/caching/) | | Give it a credential | `SecretEnv` | [Secrets](/docs/secrets/) | A [handler](/docs/steps/handlers/) is a `*senro.StepBuilder` too, and `Env`, `SecretEnv`, `WorkDir` and `Timeout` are exactly the four settings that work on one. --- > Source: https://xavidop.github.io/senro/docs/executors # Executors An executor is where a workflow's steps run. You pick one with `senro.On`, and every step in that workflow runs there. ```go deploy := p.Workflow("deploy", senro.Needs("verify"), senro.On(senro.Local())) ``` The executor belongs to the workflow, not to individual steps. Every target below is described relative to **the coordinator**, the machine that calls `senro.Run` and hosts the engine, whether that's your laptop in local development or the CI runner in a pipeline job. There are four targets today: | Target | Package | Runs every step of the workflow | |---|---|---| | `senro.Local()` | built in; the default | as processes on the coordinator's own machine | | `container.Image(ref)` | `github.com/xavidop/senro/executor/container` | in a container on a local Docker daemon | | `k8s.Pod(ref, k8s.Namespace(ns))` | `github.com/xavidop/senro/executor/k8s` | as a pod in a Kubernetes cluster | | `ssh.Host(dest)` | `github.com/xavidop/senro/executor/ssh` | as a process on a remote machine | ## Choose one | | Local | [Container](/docs/executors/containers/) | [Kubernetes](/docs/executors/kubernetes/) | [SSH](/docs/executors/ssh/) | |---|---|---|---|---| | Setup | none | a container daemon on this machine | five environment variables and RBAC | your own `~/.ssh/config` | | Workspaces | the coordinator's own directories | bind mounts of them | carried in and back, `tar` over the apiserver | carried in and back, `tar` over the connection | | `senro.RO` enforced | no, detected afterwards | **yes**, a read-only bind | **yes**, `readOnly` in the pod | no, detected on read-back | | Secrets | a file under the runtime directory | that file, bind-mounted at `/run/senro/secrets` | a namespaced `Secret`, projected at `0400` | a file on the host, delivered over stdin | | Scratch caches | the coordinator's own directory | that directory, bind-mounted | carried in and back, two full transfers per step | carried in and back, two full transfers per step | | `Func` steps | in this process | yes, the binary is bind-mounted | yes, the binary is sent in per pod | yes, the binary is staged per host | | `senro shell` | yes | yes | yes, in a pod of its own | yes | | `senro shell --tty` | yes | yes | yes, a pty the runtime allocates | `executor_no_terminal` | | stdout and stderr | kept apart | kept apart | merged into one stream | kept apart | | Private registries | nothing to pull | `container.RegistryAuth` | the node pulls, via an `imagePullSecret` you set | nothing to pull | | Cache class | `local//` | platform and resolved image digest | image digest and platform, never the namespace | `ssh//`, or your `ssh.CacheClass` | For a full comparison of secret delivery, see [Secret channels](/docs/secrets/channels/). For `senro shell` and its refusal codes, see [Shell](/docs/attach/shell/). ## Read-only mounts are enforced on two of the four `ws.At(path, senro.RO)` means the same thing on every executor, but each one enforces it differently. Don't assume the strongest behavior applies everywhere. This table shows what actually happens on each one. | Executor | What `senro.RO` does | When a write through one is caught | |---|---|---| | Container | A real read-only bind mount | At the write. The write fails | | Kubernetes | `readOnly` on the pod's volume mount; the kubelet refuses | At the write. The write fails | | Local | Nothing at mount time; a workspace is a directory with no per-step mode | Right afterwards, when the workspace's content is found changed under a mount that promised it would not be | | SSH | Nothing on the far side, which senro is not root on | On read-back. The copy is hashed but not written over yours, so the write is reported rather than carried home | On the local and SSH executors, read-only is a request that senro checks after the fact, not a rule the kernel enforces. Keep credentials and other sensitive input out of any workspace a step could overwrite by mistake. ## What every executor shares A step behaves the same no matter where it runs. These things are handled above the executor level, so no target re-implements them and they can't differ between executors: - retries, `Timeout`, `ContinueOnError`, and the end-state taxonomy ([Failure states](/docs/steps/states/)) - `OnFailure` and `Always` handlers, which run on their parent's executor and can't declare their own ([Handlers](/docs/steps/handlers/)) - workspace snapshots and the action cache ([Caching](/docs/data/caching/)) - secret resolution, delivery as a file, and redaction of every log stream ([Secrets](/docs/secrets/)) - the trace context (`TRACEPARENT`, and `TRACESTATE` when the run has one) If your build doesn't include a given executor, targeting it is rejected at `Build()`. It never falls back to running locally. ## Refusals worth knowing before you pick | What you wrote | Where it is refused | |---|---| | One scratch cache mounted by a remote step and by a local or container step, **with nothing ordering them** | `Build()`. The local step would write that directory while the remote step is tarring it. A `Needs` between them makes it a hand-off, and legal | | An unpinned image tag on `k8s.Pod` | `Build()`. Pin it to a digest | | `k8s.Pod` with no `k8s.Namespace` | `Build()`. There is no fallback to `default` | | An executor on a handler | `Build()`. A handler runs where its parent ran | | `container.RegistryAuth` on any other executor | `Build()`. Only the container executor pulls an image itself | | `DOCKER_HOST` naming `tcp://` | Run start. Every container mount is a bind mount of a coordinator directory | > The executor is set per workflow, not per step, because it decides what a mount, a secret, and a > shell actually mean on that machine. Steps that share a machine share an executor. Grouping steps > by where they run costs nothing: see [Ordering](/docs/steps/ordering/). ## Where to go next - **[Containers](/docs/executors/containers/)**: `container.Image`, the daemon it needs, `container.User`, and private registries. - **[Kubernetes](/docs/executors/kubernetes/)**: cluster configuration, workspaces across the apiserver, and delegated secrets. - **[SSH](/docs/executors/ssh/)**: your own SSH configuration, where things land on the host, and the cache class. - **[Func steps off the coordinator](/docs/executors/func-remote/)**: staging the pipeline binary, cross-compiling, and cgo. --- > Source: https://xavidop.github.io/senro/docs/extend # Extending senro Every seam in senro is a small Go interface you implement in your own module. There is nothing to register, no plugin loader, and nothing under `internal/` to import. ## The five extension points Each one lives next to the built-ins it extends, so you can check whether something already does the job before writing anything. | You want to | Implement | Page | | --- | --- | --- | | Fan out over a layout no [shipped graph](/docs/monorepo/unit-graphs/) reads | `senro.UnitGraph`, `senro.UnitAffector` | [Write a unit graph](/docs/monorepo/unit-graphs/custom/) | | Trigger on an event source [senro does not parse](/docs/triggers/events/) | `trigger.Provider`, `trigger.Matcher` | [Write a trigger source](/docs/triggers/custom/) | | Send a run's result somewhere [senro has no destination for](/docs/notifications/) | `notify.Renderer`, `notify.Requester` | [Write a destination](/docs/notifications/custom/) | | Turn the event stream into traces, metrics or anything else | `senro.Sink` | [Write a trace exporter](/docs/extend/exporter/) | | Have a program [explain a failed step](/docs/analyzers/) | `senro.Analyzer` | [Write an analyzer](/docs/analyzers/custom/) | Each page ends with a worked example. The analyzer has two: a provider-free one you can run with no key, and [`contrib/genkitanalyzer`](/docs/analyzers/genkit/), a shipping package backed by a real model that you install rather than copy. ## What they have in common - **Structural satisfaction.** A type with the right methods is the interface. No registry, no build tag, no `init`. - **A narrow import.** Each one needs `github.com/xavidop/senro` or one of its public subpackages, and nothing else of senro's. The test suite checks that mechanically for the worked examples rather than taking it on trust. - **Your errors are senro's errors.** Every seam has exactly one way to say "I could not answer", and senro turns it into a message naming your implementation. A panic becomes the same error; none of them can end a run. - **The built-ins take the same path.** `trigger.GitHub()` is a `Provider`, `notify.Webhook` is a `To(url, EventJSON(), ...)`, `notify.GitHubChecks` is a `Requester`. There is no private shortcut, so the public path is the tested one. - **A worked example that compiles.** Each page links one under [`examples/`](https://github.com/xavidop/senro/tree/main/examples), driven end to end by senro's own tests. ## The smaller seams These are extension points too, but they are small enough to be documented where they are used: | Seam | What it is | Where | | --- | --- | --- | | `senro.RegisterFunc` | A Go function as a step kind, instead of a command | [Function steps](/docs/steps/functions/) | | `retry.RegisterPredicate` | A Go function deciding whether a failed attempt is worth retrying, under a name a plan can record | [Retries](/docs/steps/retries/#deciding-in-go-retryregisterpredicate) | | `change.Source` | Where "what changed" comes from, when it is not a trigger | [Affected sets](/docs/monorepo/affected/) | | `senro.DurationHistory` | How long each unit took last time, which is what `Partition` balances by | [Partitioning](/docs/monorepo/partition/) | | `senro.Flusher`, `senro.Reporter` | Optional interfaces a `Sink` may also implement | [Trace exporter](/docs/extend/exporter/) | | `notify.ResponseReader` | A `Requester` that needs to read the response it got | [Notifier](/docs/notifications/custom/) | ## What is deliberately not a seam - **The executor.** Local, container, Kubernetes and ssh are the four, and `senro.ExecutorTarget` is closed. See [Executors](/docs/executors/). - **`trigger.Option`.** Its method is unexported and `trigger.Matcher` is the way in, because the set of questions a trigger can ask has to stay the set senro can render into a run's record. - **`api.Remedy`.** An analyzer's remedy comes from a closed vocabulary of one, so the most an unsupervised run can do is retry a step. See [Failure analyzer](/docs/analyzers/custom/). ## Where to go next - **[The event stream](/docs/run/event-stream/)**: what a `Sink` and a notifier are reading. - **[Run options and outcomes](/docs/run/options/)**: `WithSink` and the other options most of these get wired through. - **[The `api` package](/docs/run/api/)**: the wire contract every extension shares. --- > Source: https://xavidop.github.io/senro/docs/analyzers # Failure analyzers When a step fails, senro can hand that failure to a program you supply, which answers with one line saying what broke. That line lands in the run's event stream, so it shows up in the TUI, in `events.jsonl` and in anything else watching the run. That program is an **analyzer**. It can be a model, a `strings.Contains` over the log, or a lookup in your team's runbook. senro does not care which. ``` ✗ fetch failed exit 1 proposed fetch failed on the network, not on its own work press a to retry this step, A to dismiss ``` ## What you get out of it Nothing runs an analyzer for you by default. You add one option to `senro.Run`: ```go senro.Run(ctx, p, senro.WithAnalyzer(myAnalyzer, senro.AnalyzerName("runbook"))) ``` From then on, every failed step gets: - **An explanation** in the run's ledger, as an `analysis.proposed` event. - **Optionally, one offered action**: retry this step. That is the whole vocabulary, and it is offered, not taken. See [the gate](#a-proposal-never-applies-itself). A step that was skipped because something upstream failed is never sent to your analyzer. Only steps that actually broke are. ## Your two options | | | |---|---| | **[The AI analyzer](/docs/analyzers/genkit/)** | `contrib/genkitanalyzer`, ready to install. You give it a [Genkit](https://genkit.dev) instance, it explains failures with the model of your choice: Gemini, OpenAI, Anthropic, Ollama, anything Genkit has a plugin for. | | **[Write your own](/docs/analyzers/custom/)** | One method, `Analyze(ctx, api.Failure) (api.Proposal, error)`. Reach for it when the answer is in your logs or your runbook rather than in a model. | senro itself holds no API key and depends on no AI SDK. The Genkit analyzer is a separate module you install on purpose. ## A proposal never applies itself An explanation is a suggestion. It becomes an action only when somebody decides: - **You decide**, in [the TUI](/docs/attach/tui/): `a` accepts the focused step's proposal, `A` dismisses it. Accepting a retry runs the same code path `r` (retry) already does, refusals included. - **A policy you wrote decides**, for a run nobody is watching: ```go senro.WithAnalyzer(a, senro.AcceptWithoutHumanApproval( func(f api.Failure, p api.Proposal) bool { return p.Remedy == api.RemedyRetry && f.Attempt == 1 })) ``` Anything a policy applies is recorded with `policy: true`, so you can tell from the ledger alone that no person was involved. A policy applies at most once per step per run. With neither, a proposal stays a proposal: an explanation you read, and nothing more. ## What it costs the run Nothing. `Analyze` runs off the engine's goroutine, one call is bounded by `senro.AnalyzeTimeout` (30s by default), and a slow analyzer never delays a step. An analyzer that errors, panics or has no answer is treated as "no comment": the run's own result is unchanged. ## Where to go next - **[The AI analyzer](/docs/analyzers/genkit/)**: install `contrib/genkitanalyzer` and wire it up. - **[Write your own](/docs/analyzers/custom/)**: the interface, with a worked example. - **[The TUI](/docs/attach/tui/)**: `a` and `A`, and what the footer shows before you press one. - **[Reading a failed run](/docs/run/debugging/)**: what senro tells you with no analyzer at all. --- > Source: https://xavidop.github.io/senro/docs/monorepo/fan-out # Fan out with `Expand` `(*WorkflowBuilder).Expand(id, graph)` adds one step per unit that a graph discovers. Add a new app under `apps/`, and you get a new step for it automatically. Nobody has to write it. ```go import ( "github.com/xavidop/senro" "github.com/xavidop/senro/exec" "github.com/xavidop/senro/unit/glob" ) verify := p.Workflow("verify") verify.Expand("lint", glob.Dirs("apps/*")). MaxParallel(4). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("pnpm", "--filter", u.Name, "lint")). Pure().Inputs(u.Sources()...) }) ``` `Expand` walks the repository once. It builds the same steps you'd get from calling `verify.Step(...)` once per directory. ## Discovering units with `glob` The graph decides what counts as a unit. `github.com/xavidop/senro/unit/glob` is the simplest graph: it just matches paths. - **`glob.Dirs(pattern)`**: one unit per matching **directory**. - **`glob.Files(pattern)`**: one unit per directory that **contains** a matching file, so `glob.Files("services/*/go.mod")` is one unit per service with a `go.mod`. Two matches in one directory still produce one unit. - Patterns use senro's standard [path pattern syntax](/docs/data/workspaces/#pattern-syntax). - `ID` and `Name` are both the slash-separated path relative to the root. A path alone doesn't say what it imports, so a glob expansion always covers every unit. To run only what a change affects, fan out over a graph that reads the ecosystem's manifests instead. All eight shipped graphs are listed in [The shipped unit graphs](/docs/monorepo/unit-graphs/). ## What a `Template` receives `Template` is called once per unit, always in the same sorted order. Each call must return a **fresh** `*senro.StepBuilder`, built with `senro.NewStep`: - Not `Workflow.Step`: that would attach the step to the workflow a second time. - Not `senro.Handler`: that marks it as a failure or always handler. | On `Unit` | What it is | |---|---| | `u.ID`, `u.Name` | The unit's stable identity. For `glob` both are the slash-separated path relative to the root. | | `u.Dir` | The unit's directory, relative to the root. What a template passes to `WorkDir`. | | `u.Base()` | The last path segment: `"web"` for `"apps/web"`. Usually what a deployment step names. | | `u.Sources()` | Every file under the unit's directory, as `[]artifact.Selector`, ready for `.Inputs(...)` on a [`Pure()`](/docs/data/caching/) template. Declare narrower `Inputs` yourself if you need them. | ## Child ids are deterministic Each child's id comes from the expansion id and the unit, like `lint[unit=apps/web]`. It's never a name the template picks, because a picked name couldn't be guaranteed unique across units. Since ids come only from the sorted unit set, the same repository always builds the same children. A re-run produces exactly the same step graph. ## `MaxParallel` and `MaxNodes` - **`.MaxParallel(n)`** limits how many children of *this expansion* run at once, within the run's overall limit. That overall limit is the number of CPUs on the coordinator machine, and there's no option to change it for the whole run. - **`.MaxNodes(n)`** rejects an expansion wider than `n` (default 500) at **build** time, and names the pattern and count that caused it. This stops a scheduler from discovering, mid-run, that it has forty thousand sandboxes to hold open. `MaxNodes` checks the **whole** graph, so neither [`Affected`](/docs/monorepo/affected/) nor [`Partition`](/docs/monorepo/partition/) can get around it. ## `Needs`: order the whole expansion `.Needs(ids ...string)` on an `ExpandBuilder` declares upstream steps that **every child** waits for. It's the same kind of dependency that [`(*StepBuilder).Needs`](/docs/steps/ordering/) declares for a single step. `Expand("lint", ...).Needs("install")` makes the whole fan-out wait for the step that produces what its children read. This is a **barrier**. When the downstream work is itself per unit, you want [`NeedsEach`](/docs/monorepo/needs-each/) instead. ## Expansion happens once, at build time `Expand` resolves when `Build()` runs, not while the pipeline is running. Every unit is discovered and written into the plan before the first step starts. - A repository with three matching directories and one with four produce two **different** pipelines, with two different plan digests. - Nothing a step produces can add a node to a graph that's already running. ## What `Expand` doesn't do (yet) - If a step's own output needs to add new nodes mid-run, that's a **generator**, not an expansion: see [Generated subgraphs](/docs/monorepo/generators/). For control flow that isn't a graph at all, a function can run one directly with `senro.RunSubgraph`. - There's no `FailFast` option for an expansion, by design. If one child fails, senro reports it and keeps running the rest of the fan-out instead of cancelling everything. ## Where to go next - **[The shipped unit graphs](/docs/monorepo/unit-graphs/)**: what each graph calls a unit. - **[Per-unit edges](/docs/monorepo/needs-each/)**: `NeedsEach`, so fast tests don't wait behind slow builds. - **[Partition](/docs/monorepo/partition/)**: fewer steps than units, balanced by duration. - **[Running only what changed](/docs/monorepo/affected/)**: `Affected`. - **[Conditions](/docs/steps/conditions/)**: `(*ExpandBuilder).When`, which prunes children the plan still contains. --- > Source: https://xavidop.github.io/senro/docs/steps/functions # Func steps `senro.Func(name, params)` runs a registered Go function in place of a command. Reach for it whenever the work is "call this Go function", not "shell out to a program". ```go type DeployParams struct { App string `json:"app"` } func init() { senro.RegisterFunc("deploy/notify", Notify) } func Notify(ctx senro.Ctx, p DeployParams) error { token, err := os.ReadFile(ctx.Secret("SlackToken")) if err != nil { return err } _, err = fmt.Fprintf(ctx.Stdout(), "deployed %s, %d byte credential in hand\n", p.App, len(token)) return err } // in the pipeline: deploy.Step("notify", senro.Func("deploy/notify", DeployParams{App: "web"})). SecretEnv("SlackToken", "SlackToken") ``` A func step is built, scheduled, retried, cached and handled by exactly the same code an `exec.Command` step is. ## `RegisterFunc` `RegisterFunc[P](name, fn)` registers `fn` under `name`, once, from an `init` function of the defining package. - **Registering the same name twice panics.** - **The name is the function's identity.** A closure has none of its own, so the name is what a plan records and what feeds the step's cache key, exactly like a command's argument list. Renaming it invalidates the cache and breaks any recorded plan that still names it, as renaming a command would. - **`P` must be JSON-serializable, and decoding is strict.** A recorded field `P` does not have is an error, so a renamed parameter field fails loudly instead of running with a zero value. ## `senro.Ctx` `senro.Ctx` is what a function receives in place of a working directory and `argv`. It embeds `context.Context`, so it passes straight into any library call that takes one. | Method | What it gives you | |---|---| | `ctx.Workspace(name)` | `(senro.WorkspacePath, bool)`: a mounted workspace's path (the same path an `exec.Command` mount resolves to) and whether this step mounted it. `WorkspacePath` is a named string type: assign it with `:=` or convert it | | `ctx.Secret(name)` | A delivered secret's file path, by the same field name `SecretEnv` used, or `""` if this step didn't declare it. The value lives in the file; this string is never the value | | `ctx.Stdout()`, `ctx.Stderr()` | The step's log streams, redacted and recorded exactly as a command's output is. Writing to `os.Stdout` instead reaches the coordinator's terminal and no log file | | `ctx.Logger()` | Structured lines to `Stderr` | | `ctx.RunID()`, `ctx.StepID()`, `ctx.Attempt()` | This invocation's identity in the event stream. `Attempt()` is `1` on the first try, which is what an idempotency key needs to know before retrying against a remote API | | `ctx.Failure()` | `(senro.StepFailure, bool)`: what this function is cleaning up after, when it is running as a [handler](/docs/steps/handlers/). `ok` is false for an ordinary step | **`Ctx` carries no working directory**, because the coordinator's is process-global: changing it would change it for every concurrent step. ## Where a func step can run | Executor | A func step there | |---|---| | `senro.Local()` | Runs on the coordinator, in this same process | | `ssh.Host(dest)` | Runs on the host. senro stages a copy of your pipeline binary and re-enters it there | | `container.Image(ref)` | Runs in the container, from a read-only bind of your pipeline binary | | `k8s.Pod(...)` | Runs in the pod. senro sends your pipeline binary in over the apiserver's `exec` subresource and re-enters it there | ### On the coordinator The function runs in this process, in the same kind of sandbox a command gets: its own mounts, secrets and log files. Nothing is copied anywhere. ### Anywhere else: senro sends your binary A Go function's body only exists inside your compiled binary. A `Plan` is JSON and cannot describe it, so running one on an SSH host means **moving the binary, not the plan**. That is exactly what senro does, and you write no code for it: 1. It puts a copy of your pipeline binary on the target: over SSH, as a read-only bind into a container, or in through the apiserver's `exec` subresource for a pod. 2. It re-enters that copy as a child process, telling it which registered function to call and with which params. 3. Your function runs there. `ctx.Workspace(...)` and `ctx.Secret(...)` return paths **on the target**, not on your machine, so the same function body works either way. Three things to know before you rely on it: - **Your module has to cross-compile.** The binary sent to a Linux host has to be built for it, which in practice means `CGO_ENABLED=0`. Run `senro func check [--dir DIR] [packages...]` to find out whether yours can be, before a run tells you on a Friday. See [CLI](/docs/cli/workspaces/). - **A pod's image must carry `sh` and `tar`**, exactly as carrying a workspace does: the binary arrives as a `tar` into a container that is holding open for it. A `FROM scratch` image cannot receive one. - **A func step cannot run on a target that delegates secrets.** The two deliver different things, and only one of them is something a Go function can read. See [below](#why-delegated-secrets-and-func-steps-cannot-mix). The staging, its cost and its caching are covered in [Func steps off the coordinator](/docs/executors/func-remote/). #### Why delegated secrets and func steps cannot mix A secret reaches a pod in one of two ways, and the target decides which: | On the target | What lands in the pod | Who turns it into a credential | |---|---|---| | Default | `SENRO_SECRET_KUBECONFIG=/run/senro/secrets/Kubeconfig`, the path of a **file senro already wrote** | senro, before the step starts | | [`k8s.DelegateSecrets()`](/docs/executors/kubernetes/) | `SENRO_SECRET_KUBECONFIG_SOURCE=aws-sm://prod/ci/kubeconfig`, a **source URI** and nothing else | your command, while it runs | A function never sees either variable: it is handed a `senro.Ctx`, not an environment. `ctx.Secret("Kubeconfig")` is a lookup of the files senro wrote for this step, and under delegation senro wrote none, so the call would return `""` and your function would deploy with an empty kubeconfig. `Build()` refuses the pipeline instead: ```go // Refused runner := k8s.Pod(img, k8s.Namespace("ci"), k8s.ServiceAccount("senro-ci"), k8s.DelegateSecrets(), // the pod fetches its own secrets... ) deploy := p.Workflow("deploy", senro.On(runner)) deploy.Step("deploy", senro.Func("deploy/apply", nil)). // ...but this is a function SecretEnv("KUBECONFIG", "Kubeconfig") ``` ``` plan: step "deploy" is a func step on a target that delegates secrets, and the two cannot both hold: delegation delivers secret "Kubeconfig" to the pod as SENRO_SECRET_KUBECONFIG_SOURCE, a source URI for the step's own COMMAND to resolve, while a function reads ctx.Secret("Kubeconfig") ``` There are two ways out. **Drop the delegation**, so senro delivers a file, which is what a function wants: ```go runner := k8s.Pod(img, k8s.Namespace("ci")) // no DelegateSecrets deploy := p.Workflow("deploy", senro.On(runner)) deploy.Step("deploy", senro.Func("deploy/apply", nil)). SecretEnv("KUBECONFIG", "Kubeconfig") // inside deploy/apply: // kubeconfig, err := os.ReadFile(ctx.Secret("Kubeconfig")) ``` Or **keep the delegation and write the step as a command**, which can resolve the URI itself: ```go deploy.Step("deploy", exec.Command("sh", "-c", ` aws secretsmanager get-secret-value \ --secret-id "${SENRO_SECRET_KUBECONFIG_SOURCE#aws-sm://}" \ --query SecretString --output text > /tmp/kubeconfig KUBECONFIG=/tmp/kubeconfig kubectl apply -f k8s/ `)). SecretEnv("KUBECONFIG", "Kubeconfig") ``` The rule in one line: delegation means *the step fetches its own credential*, and only a command can do that. A function can only read a file senro already put there. ### As a handler, on the same target A [handler](/docs/steps/handlers/) declares no executor of its own and runs wherever its parent ran. A `senro.Func` handler gets the same treatment a func step does: on an ssh host, in a container or in a pod, senro stages the binary on the parent's target and re-enters it there. It reuses the copy the parent step already staged, so a handler on a remote step costs no second transfer. ```go deploy.Step("apply", exec.Command("./deploy.sh")). OnFailure(senro.Handler("collect", senro.Func("ci/collect", CollectParams{}))) ``` Inside `ci/collect`, `ctx.Failure()` says what broke, and `ctx.Workspace(...)` reports paths on the target. See [Failure handlers](/docs/steps/handlers/#a-handler-can-be-a-go-function). ## Panics and timeouts - **A panic is caught** and reported as the step state `panicked` rather than crashing the run. The stack is in the step's stderr log, and panics are not retried. See [Step states](/docs/steps/states/). - **`Timeout` on the coordinator bounds only reporting.** Nothing can force a Go function to return, so one that ignores its context keeps running and is merely filed as `timed_out` when it finishes. Off the coordinator the function has a process of its own, which ends itself at the deadline. Declare a `Timeout` on every remote func step. ## Where to go next - **[Func steps off the coordinator](/docs/executors/func-remote/)**: staging, cross-compiling, cgo. - **[Executors](/docs/executors/)**: picking the target with `senro.On`. - **[Secrets](/docs/secrets/)**: what `ctx.Secret` hands you and why it is a path. - **[Step states](/docs/steps/states/)**: `panicked` and the other nine. --- > Source: https://xavidop.github.io/senro/docs/executors/func-remote # Func steps off the coordinator A [`senro.Func` step](/docs/steps/functions/) is a Go function compiled into your pipeline binary, not a command any shell can run, so a plan can name it but can't describe what it does. Point one at an SSH host, a container image, or a Kubernetes pod, and senro puts a copy of your pipeline binary there and re-enters it, rather than trying to ship the function itself. On an SSH host, the function then runs on that machine's filesystem, against that machine's network: ```go func HelmUpgrade(ctx senro.Ctx, p DeployParams) error { charts, _ := ctx.Workspace("charts") // a directory on build-07, not on your laptop return helm.Upgrade(ctx, p.App, charts.Path("apps", p.App), ctx.Secret("Kubeconfig")) } release := p.Workflow("release", senro.On(ssh.Host("deploy@build-07.internal"))) release.Step("deploy", senro.Func("deploy/helm", DeployParams{App: "web"})). Mount(senro.Workspace("charts").At("/charts", senro.RO)). SecretEnv("KUBECONFIG", "Kubeconfig").Timeout(10 * time.Minute) ``` Point the same step at `container.Image("golang:1.26")` or at `k8s.Pod(ref, k8s.Namespace("ci"))` instead, and it runs there, in that image's filesystem, as a process of its own. Nothing else in the pipeline changes. ## What you have to set up Nothing, if the target's platform matches the coordinator's and your `main` does nothing unusual. Otherwise there are three things to do: name the package to build and make sure the coordinator has a Go toolchain (any container or pod step run from macOS needs both, since an image is Linux), call `senro.StepChild` if your `main` parses flags, and keep cgo out of your module's dependency graph ([below](#cross-compiling-and-the-cgo-constraint)). **Name the package.** A Go program doesn't record where its own source lives, so tell senro yourself with `senro.Run(ctx, pipeline(), senro.WithFuncBuild("./ci"))` or the `SENRO_FUNC_PKG` environment variable. `senro run ./ci` sets this for you, so your local dev loop needs nothing extra. In CI, where the binary is built once and then run, set the variable in the job or pass the option instead. Without either, a run that needs a cross-build fails right away, naming both fixes, the same as a missing Go toolchain would. **Call `senro.StepChild`** if your `main` would exit on the arguments senro re-enters it with: ```go func main() { if handled, err := senro.StepChild(context.Background()); handled { if err != nil { log.Fatal(err) } return } // ... your own main, flag parsing and all } ``` ## What actually happens A plan records a function's registered name and parameters, not its body: the body is compiled into your pipeline binary. So running a function elsewhere means putting that binary there and running it. ```mermaid sequenceDiagram participant C as Coordinator participant T as Target (ssh host, container or pod) C->>C: cross-compile if platforms differ (cached, once per release per arch) C->>T: stage the binary (ssh: upload once per host / container: bind / pod: tar per pod) C->>T: run senro-sha256-… __step --state-fd 0 C->>T: step state, one JSON document on stdin T-->>C: handshake frame, reporting its own digest T-->>C: stdout and stderr frames, kept apart T-->>C: final result frame ``` - **The child is re-entered as `senro-sha256- __step --state-fd 0`.** The whole step state (step id, function name, parameters, workspace directories and secret file paths over there, run id, timeout) arrives on stdin as one JSON document. That's because command lines are visible to `ps`, and every account on the target could otherwise read them. - **Frames come back on stdout, length-prefixed**: a handshake, the function's stdout and stderr kept apart, and a final result. These go through the same redactor and offset-recording writers a local step uses, so `senro logs`, the TUI, and `step.log.appended` look identical either way. The child's own stderr is not framed and is captured verbatim: it's the diagnostic channel for a child that dies before sending a frame. - **A `binary.staged` event** closes out the sequence, covered [below](#version-skew-is-fatal-and-staging-is-visible). Everything else is inherited rather than reimplemented, since the split happens deep in the engine, after the sandbox already exists: retries, `Timeout`, snapshots, the cache, secrets, handlers, redaction, traces. ## How the binary gets there **Over ssh: a transfer, once per host.** The binary lands at `/bin/senro-sha256-`, mode `0700`, owned by the connecting account. The path is the digest, so a second step, run, or coordinator all refer to the same file. senro just checks whether the host already has it at the right length before uploading. That directory sits alongside the per-attempt ones, and is beyond the reach of `Close` and the reaper: nothing removes it automatically, so run `rm -rf ~/.senro/work/bin` yourself to reclaim the space. **In a container: a read-only bind mount, and no transfer at all**, since the daemon runs on the coordinator's own machine ([Containers](/docs/executors/containers/)). Two caveats: the image must not swallow the command (the staged binary is the container's `Cmd`, and `ENTRYPOINT` is left alone, so an entrypoint that ignores or rewrites its arguments never actually runs it), and since a container image is Linux, a macOS coordinator cross-compiles for every func step in a container, no matter how local the daemon is. **In a pod: a `tar` over the apiserver's `exec` subresource, once per pod.** It lands on an `emptyDir` at `/senro/bin/senro-sha256-`, mode `0700`. The step's container starts and holds so the child can be exec'd into it, which is what keeps the child's stdout and stderr apart, which a pod's merged log couldn't do, and what gives the step an exit code of its own. The image needs `sh` and `tar`, exactly as carrying a workspace does ([Kubernetes](/docs/executors/kubernetes/)). A pod's filesystem doesn't outlive the pod, and senro keeps no cluster object to store a copy in, so this transfer happens on every attempt and `reused` is always `false`. A genuinely large pipeline binary is the one thing that makes this executor a worse fit for a func step than an ssh host. `k8s.DelegateSecrets()` is refused for a func step that declares a secret: delegation hands a source URI to a command to resolve, but a function reads `ctx.Secret(name)` instead. See [why the two can't mix](/docs/steps/functions/#why-delegated-secrets-and-func-steps-cannot-mix) for the two ways around it. ## What your function sees Your function gets exactly the same [`senro.Ctx`](/docs/steps/functions/) it would see on the coordinator, just with paths pointing at the target instead: - **`ctx.Workspace(name)`** is the directory over there (inside the attempt's root on an ssh host, or at the mount's declared path in a container or a pod). It's filled before the step runs and read back afterward. - **`ctx.Secret(name)`** is a file path over there: the host's tmpfs where it has one, or `/run/senro/secrets` in a container or pod. It's removed when the step ends, reaped if the coordinator dies first, and never travels as a value: not in the state document, an argument, or an environment variable. - **`ctx.RunID()`, `ctx.StepID()`, `ctx.Attempt()`** stay the coordinator's values, so an idempotency key built from them means the same thing on either side. - **`TRACEPARENT`** is in the child's environment, naming this attempt's own span. A func step on the coordinator is the one case that gets none, since there's no process to give one to. - **`os.Stdout`**, if you write to it instead of `ctx.Stdout()`, reaches the child's stderr rather than corrupting the frames. That's a courtesy, not a contract to rely on. **`Timeout` actually matters here.** On the coordinator it only bounds how a func step's outcome gets reported, since nothing can force a Go function to return. Off the coordinator, the function has a process of its own, which ends itself once the deadline passes. Declare a timeout on every remote func step. It stops a lost coordinator from leaving a function running on somebody else's build host. The deadline crosses as a duration rather than a wall-clock time, so the two machines' clocks don't need to agree. ## Version skew is fatal, and staging is visible In its first frame, the child reports the digest of the file it actually is, computed from its own executable. If that disagrees with what senro staged, the step aborts: *"the binary on build-07 reports sha256:9f2c…, and senro staged sha256:41ab… there. Something replaced the staged file, or two coordinators of different builds are sharing this host's staging directory."* This is deliberately not retried, since a retry would just re-run the same wrong binary forever. That digest is also part of the [cache key](/docs/data/cache-keys/) for every func step, so a new release invalidates func-step results instead of answering from a changed function. A 40 MiB transfer is affordable once per host per release, not once per step. `binary.staged`, emitted for every func step, records what you're paying for: digest, platform, strategy, destination, size, and whether anything actually moved. ```json {"type":"binary.staged","step":"deploy","payload":{"digest":"sha256:41ab…","platform":"linux/arm64", "strategy":"cross-build","target":"ssh://build-07.internal","bytes":41287168,"reused":true, "path":"/home/deploy/.senro/work/bin/senro-sha256-41ab…","duration_ns":163490958}} ``` `reused: true` means senro didn't have to move the binary. So an ssh host whose func steps always report `reused: false` is transferring on every step, which shouldn't happen, and `reused: false` should never appear at all in a container. In a pod it's always `false`, and honestly so, because every pod gets a fresh filesystem. ## Cross-compiling, and the cgo constraint When platforms match, senro ships `os.Executable()` unchanged and compiles nothing. Otherwise it runs `GOOS=… GOARCH=… CGO_ENABLED=0 go build -tags netgo,osusergo -ldflags '-extldflags=-static' -o `, keyed by the coordinator binary's digest, the package, and the target platform. The result is cached under your senro cache directory, once per architecture per release rather than per run. `netgo`, `osusergo`, and the static link mean glibc and musl skew simply can't happen. `CGO_ENABLED=0` isn't a preference. Cross-compiling with cgo needs a C cross-toolchain per target, which a pipeline engine can't require of you. So no package anywhere in your module's dependency graph can compile a cgo file, and the offenders aren't always obvious: `os/user` under some build configurations, `net` without the `netgo` tag, and anything wrapping a C library. senro checks this before the run emits a single event, and refuses with the import path and the dependency chain that pulled the cgo package in: ``` ./ci cannot be cross-compiled for another platform: 1 package(s) in its dependency graph compile a cgo file, and a cross-compile is built with CGO_ENABLED=0, which cannot link their C dependency for the target github.com/acme/ci/internal/db (sqlite3.go) via: github.com/acme/ci -> github.com/acme/ci/internal/db ``` This report comes from the same detector that [`senro func check`](/docs/cli/workspaces/) uses, so the two stay consistent. Run `senro func check ./...` in CI before you depend on this. What it costs in practice/some examples: - a pure-Go SQLite driver instead of a cgo one - `os/user` lookups that read `/etc/passwd` rather than calling NSS, which behaves differently under LDAP or SSSD - DNS through Go's own resolver rather than the host's `nsswitch.conf` If any of that is unacceptable, run the step on the coordinator instead, or make it an `exec` step calling a binary you built for the host yourself. > One warning for the case where senro compiles nothing: a pipeline binary built on a glibc machine > with cgo enabled, then run as a func step in a musl image of the same architecture, fails to start. > The daemon reports that the file doesn't exist, which is what a missing dynamic loader looks like. > Build with `CGO_ENABLED=0` if your pipeline drives containers. ## When it goes wrong **`the binary staged on … did not re-enter as a step child`** The staged binary ran, but never called back into senro to act as the step. Two likely causes: - Your `main` never reaches `senro.Run`, usually because a flag parser exits early when it sees the `__step` argument. Call `senro.StepChild` before you parse flags, as shown [above](#what-you-have-to-set-up). - In a container, the image's `ENTRYPOINT` doesn't `exec` its arguments, so the staged binary is never actually run. **The step failed and stdout is empty** Look at the step's stderr instead. It's captured unframed and verbatim, so it's where you'll see a binary that couldn't execute, a Go runtime that crashed before `main`, or a shell that couldn't find the file. **The daemon reports `/senro/bin/senro-sha256-…` does not exist** Your pipeline binary is dynamically linked and won't run in a musl-based image (like Alpine). See [Cross-compiling, and the cgo constraint](#cross-compiling-and-the-cgo-constraint) above. **In a pod: `sending the step binary into pod …failed (tar exited …)`** The image is missing `tar` or `sh`. Both are required, the same as for mounting a workspace ([Kubernetes](/docs/executors/kubernetes/)). **The step settled as `panicked`** Your function panicked, just as it would on the coordinator. The stack trace is in its stderr log. Panics are not retried ([States](/docs/steps/states/)). **A function needs to report an infrastructure failure** Wrap the error in the executor's infra sentinel before returning it. `retry.OnInfra()` matches it once it's back on the coordinator. ## Func handlers come along A [handler](/docs/steps/handlers/) declares no executor of its own, and runs wherever its parent ran. A `senro.Func` handler is staged and re-entered on that target exactly like a func step. ```go deploy.Step("apply", exec.Command("./deploy.sh")). OnFailure(senro.Handler("collect", senro.Func("ci/collect", CollectParams{}))) ``` - It reuses the binary the parent step already staged, so a handler costs no second transfer. `binary.staged` still fires for it, with `reused: true`, so a run that's paying for a transfer per step stays visible as one. - `ctx.Failure()` tells it what broke, carried over the wire with the rest of the step state. See [Failure handlers](/docs/steps/handlers/#a-handler-can-be-a-go-function). - The one refusal left: a func handler that declares a delegated secret, for the same reason a func step can't. Delegation hands the pod a source URI for the step's own command to resolve, and a function has no environment to read it from. ## What is not here - **Windows targets.** Refused by name, since a staged binary runs through a POSIX shell, gets `chmod`ed `0700`, and is reaped with `rm -rf`, none of which works on a Windows host. - **Garbage collection**, so the ssh staging directory grows by one binary per release. - **Embedded cross-compiled variants** (`-tags senro_embed`, the air-gapped answer), **pre-built OCI images**, and **wrapping `exec` steps** in the same staged binary. All designed, none built yet. --- > Source: https://xavidop.github.io/senro/docs/monorepo/generators # Generate a subgraph `Expand` fans out over a list senro can discover **before the run starts**, by globbing the tree. A generator fans out over a list that only exists **after something has run**: the resources `terraform plan` says changed, the clusters an API reports right now, the shards a timing tool just computed. Without one, that work goes in a loop inside a single step. In this example: ```sh for c in $(list-clusters); do ./preflight "$c" && ./apply "$c"; done ``` senro can't see inside that loop. It's one step, so it gets one cache entry, one log, one state, and one retry. All forty clusters deploy under one name. One failure ends the whole loop, and retrying means retrying every cluster. A generator turns the same work into real steps: ```go l.Step("discover", exec.Command("./bin/list-clusters")). Mount(ws.At("/src", senro.RO)). Generates(senro.Generate(func(ctx senro.GenCtx) (*senro.Fragment, error) { var cs []Cluster if err := ctx.OutputJSON("clusters.json", &cs); err != nil { return nil, err } f := senro.NewFragment() for _, c := range cs { pre := f.Step("preflight-"+c.Name, exec.Command("./preflight", c.Name)) f.Boundary(f.Step("apply-"+c.Name, exec.Command("./apply", c.Name)).Needs(pre.ID())) } return f, nil })) ``` Each generated step is an ordinary step. It gets its own state, log, cache entry, and retry, scheduled under the run's `MaxParallel`. Retrying one cluster retries just that cluster. ### What's actually in a `Fragment` A `Fragment` is nothing more than a list of steps to add, plus an optional boundary. It never talks to the engine, the closure just builds one and hands it back: - `senro.NewFragment()` starts empty. - `f.Step(id, action)` adds one step to the fragment and returns a `*StepBuilder`, the same builder `l.Step` gives you, so `.Needs(...)`, `.Mount(...)`, and the rest work exactly the way they do on an ordinary step. - `f.Boundary(...)` names which of the fragment's own steps count as "the generator is done" (see below). senro takes it from there: it serializes the fragment, checks it, and splices its steps into the run under the generator's id. ## Ids are hierarchical A fragment names its steps **relatively**, and senro prefixes each one with the generator's id. The fragment above produces `discover/preflight-west` and `discover/apply-west`. That's what lets a fragment be written once, without knowing where it will sit in the graph. It's also why two different generators can both produce an `apply` step without colliding: the prefix makes the full id unique even when the relative name repeats. ```mermaid flowchart TD subgraph discover["generator: discover"] direction TB a1["discover/preflight-west"] a2["discover/apply-west"] end subgraph rollout["generator: rollout"] direction TB b1["rollout/apply-west"] end ``` ## The boundary is what dependents wait for Say a `publish` step should run only after every cluster is deployed: ```go discover := l.Step("discover", exec.Command("./bin/list-clusters")). Mount(ws.At("/src", senro.RO)). Generates(senro.Generate(func(ctx senro.GenCtx) (*senro.Fragment, error) { // same generator as before: reads clusters.json, builds preflight/apply steps ... })) l.Step("publish", exec.Command("./notify-slack")).Needs(discover.ID()) ``` Here's the trap: `discover` is the step that *runs the generator function*, and generating a fragment is fast, it's just building a Go value. `discover` finishes the instant it hands that fragment back to senro, which is the moment `apply-west` and the other generated steps **start**, not the moment they're done. Left alone, `publish` (which only `Needs(discover.ID())`) fires while clusters are still mid-deploy. `Boundary` fixes this by redirecting `discover`'s existing dependents onto the fragment's real finish line instead of onto `discover` itself. Inside the generator function, pass it the `*StepBuilder` that `f.Step(...)` returned for each "actually done" step, `apply-west` in this case: ```go apply := f.Step("apply-"+c.Name, exec.Command("./apply", c.Name)).Needs(pre.ID()) f.Boundary(apply) ``` Now `publish` effectively waits on `discover/apply-west` (and every other step named in the boundary), not on `discover`. Pass `Boundary` several steps if dependents should wait for more than one to finish, which is exactly what the first example on this page does in one line: `f.Boundary(f.Step("apply-"+c.Name, ...).Needs(pre.ID()))`. ```mermaid flowchart LR subgraph noBoundary["no Boundary: publish fires too early"] direction TB d1["discover"] --> dep1["publish"] end subgraph withBoundary["Boundary(applyStep): publish fires on time"] direction TB d2["discover"] --> pre["preflight-west"] --> ap["apply-west"] ap --> dep2["publish"] end ``` Declaring no boundary is legal, and correct when nothing downstream consumes what the generator produced, e.g. a generator that fans out cleanup work nobody waits on. In that case dependents wait only on `discover` itself. An empty fragment (`return senro.NewFragment(), nil`) is legal too. It means "nothing to do here," and the generator's dependents run immediately rather than being skipped. ## Any language can write one The Go closure is one way to write a generator. The other doesn't involve Go at all: the step's command writes a JSON file, and `GenerateFromJSON` just points senro at it. **`fragment.json` is that file**, and nothing more exotic: the same `nodes` + `boundary` shape a `Fragment` builds in Go, written to disk by whatever tool the step ran. That's what makes it work from a shell script, a Python tool, or a wrapper around Terraform, none of which can return a Go value: ```go l.Step("plan-infra", exec.Command("./bin/plan-infra")). Mount(ws.At("/src", senro.RW)). Generates(senro.GenerateFromJSON("fragment.json")) ``` Three things have to line up for this to work: 1. `./bin/plan-infra` runs `terraform plan`, looks at what changed, and **writes `fragment.json`** into the step's own output directory as a side effect, the same directory `Mount` gave it. 2. The step succeeds (exit code 0). A failed step produces no fragment, Go or JSON, senro doesn't go looking. 3. senro reads `fragment.json` back, validates it, and splices it into the graph, exactly like it would a Go closure's return value. A minimal `fragment.json` your script could write: ```json { "version": 1, "nodes": [ {"id": "apply-vpc", "kind": "exec", "cmd": ["terraform", "apply", "-target=vpc"]}, {"id": "apply-db", "kind": "exec", "cmd": ["terraform", "apply", "-target=db"], "needs": ["apply-vpc"]} ], "boundary": ["apply-db"] } ``` Same rules as the Go form: `id`s are relative (senro prefixes them with `plan-infra/`), `needs` can only name other nodes in this same file, and `boundary` lists which of those nodes `plan-infra`'s dependents should actually wait for. Both forms go through exactly the same validation, because a Go fragment is serialized to this exact schema before the engine reads it. Nothing in senro's core reads Go value data directly, it always reads this JSON shape, so writing it by hand is a first-class option, not a fallback. The file path resolves against the step's own output root, the same one `Outputs` reads from. That's why the step needs `Mount`: with no workspace mounted, there's no directory for senro to look in for `fragment.json` once the step exits. ## Your generator doesn't have to be deterministic When a generator produces a fragment, senro **records it** in the content store (the same content-addressed store that holds cached step outputs) and saves its digest in the generator's cache entry. A cached generator doesn't run again at all. Its subgraph is restored from that recording. So a generator can call an API, read a clock, or iterate a map. It doesn't need to be reproducible, because a re-run doesn't depend on it producing the same answer twice. It depends on the recording. Many workflow engines forbid this, and require deterministic code so replay works. senro takes a different approach: it records the answer instead of recomputing it. Two consequences follow from that: - If a cache entry's recorded fragment has been garbage collected, that cache hit is no longer usable. senro re-runs the generator instead of serving a run with the work missing. - `run.rerun_from` on a generator replays what it produced. The generated nodes are already in the graph, so re-running the generator re-runs them instead of creating a second copy. ## Fragments only add A fragment can add nodes, add edges between its own nodes, and attach its boundary to the generator's existing dependents. It cannot modify, remove, or re-parent anything already in the run, and its steps can only depend on its own steps. That rule keeps the splice safe: every cache key already recorded, and every attached client's view of the run, stays valid across it. If a fragment breaks any of this, senro refuses it **entirely**, and the generator step fails. Nothing is ever half-applied, because a half-applied fragment would be a graph no re-run could reproduce. ## Generators are bounded A generated step can itself be a generator. Two limits stop that from turning into a fork bomb holding your deploy credentials: | Limit | Default | What it bounds | | --- | --- | --- | | `MaxDepth` | 3 | How deep generators may nest | | `MaxNodes` | 5000 | Nodes in the whole run, the plan's own included | Exceeding either limit fails the run, and names the whole generator **chain** responsible, not just the last step in it. ## When to reach for what 1. **`When`** if the work is already in the graph and you need to skip it. 2. **`Expand`** if the list can be discovered from the tree before the run. 3. **A generator** if the list only exists once a step has run. Generators are the most powerful option, and the hardest to reason about. Since the node count isn't known in advance, senro shows counts of the nodes it knows about rather than a completion percentage it would otherwise have to revise downward. ## When a loop is not a graph Some control flow can't be drawn as a DAG at all: "roll out one cluster at a time until quorum, then stop" depends on what earlier steps did. For that, a registered function can run a graph itself: ```go senro.RegisterFunc("deploy/rolling", func(ctx senro.Ctx, p RollParams) error { for _, c := range p.Clusters { f := senro.NewFragment() f.Step("apply-"+c.Name, exec.Command("./apply", c.Name)) if err := senro.RunSubgraph(ctx, f); err != nil { return err } if quorumReached(ctx) { return nil } } return errNoQuorum }) ``` `RunSubgraph` blocks until the nested graph finishes, and returns its failure. So a rollout that couldn't complete fails the step that was running it. This is deliberately weaker than a generator, and it's worth understanding the trade-off. A generator describes work and hands it to the scheduler, which is why its nodes are ordinary steps you can cache and retry individually. A subgraph is work the function itself is doing: it belongs to that one step, so the cache and re-run granularity is the whole subgraph, and re-running means re-running the step. You can't use `senro rerun --step` to jump into the middle of one. Use a generator whenever the work is a graph. Use this only when it genuinely isn't. A subgraph runs on the coordinator. A func step running on a remote host can't start one, because it's a staged binary on the far side of a transport, and the engine lives back on the coordinator. If you call `RunSubgraph` from a remote step, it tells you so by name. ## Where to go next - **[Fan out with `Expand`](/docs/monorepo/fan-out/)**: the plan-time fan-out to prefer when it fits. - **[Caching a step](/docs/data/caching/)**: what makes a generator cacheable, and what a recorded fragment costs. - **[`senro rerun`](/docs/cli/rerun/)**: replaying a recorded subgraph, and `--regenerate`. --- > Source: https://xavidop.github.io/senro/docs/triggers/gitea # Gitea triggers `provider: "gitea"`. Reads Gitea's `push`, `pull_request` and `create` webhook bodies. ## Wire it ```go ev, err := trigger.LoadEvent(*eventPath) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")), trigger.OnPullRequest(trigger.Actions("opened", "synchronized")), trigger.OnTag(trigger.Semver(">=1.0.0")), )) ``` ## Write the event file `event` is Gitea's `X-Gitea-Event` header, verbatim: ```json {"provider":"gitea","event":"push","payload":{ ...Gitea's body... }} ``` ## What it accepts | `event` | Becomes | |---|---| | `push` | `Push`, or `Tag` when the ref is `refs/tags/...` | | `pull_request` | `PullRequest` | | `create` | `Tag`, and only for a tag | ## Worth knowing **A tag arrives twice: forward one, not both.** A tag push is a `push` with a `refs/tags/...` ref, exactly as GitHub's, and Gitea *also* sends a `create` for the same tag. Forwarding both runs the pipeline twice for one tag. Pick whichever your receiver finds easier and drop the other: ```sh # keep the push, drop the create [ "$GITEA_EVENT" = "create" ] && exit 0 ``` **`create` is parsed for a tag only.** A branch `create` is refused, because the push Gitea sends for the same new branch carries its commits and their changed files. A `create`'s `ref` is the **short** name where every other Gitea payload carries the full one, so `ref_type` is the only thing saying what it names. **The actions are Gitea's own words**, carried through untranslated: `opened`, `closed`, `reopened`, `edited`, `synchronized`, and the label, assignee and review ones. ```go trigger.OnPullRequest(trigger.Actions("opened", "synchronized")) ``` Note `synchronized`, with a `d`, where GitHub says `synchronize`. **Deletions and creations are the all-zero SHA**, not a flag, as GitLab does it. A deleted ref never matches. **The commit list is truncated**, with the real count in `total_commits`. senro reports a truncated list as **no list at all**, so `Paths` errors instead of hiding a match. Same for `pull_request` payloads, which carry no file list to begin with. See [when there is no file list](/docs/triggers/events/#when-there-is-no-file-list). **A pull request's `Number` is `pull_request.number`**, the per-repository number. The sibling `id` is the instance-wide key and is deliberately not read. Older payloads carry the number only at the top level, and senro falls back to it. ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers, and what a match carries into the run. - **[The event file](/docs/triggers/events/)**: the envelope every source shares. - **[Write your own](/docs/triggers/custom/)**: layering a Gitea event this build skips on top of `trigger.Gitea()`. --- > Source: https://xavidop.github.io/senro/docs/notifications/github-checks # GitHub Checks Puts the run's result back on the commit, as a check run with a conclusion and per-step annotations. It closes the loop a [trigger](/docs/triggers/) opens: GitHub sends you an event, you build, GitHub gets an answer. ```go n := notify.New( notify.GitHubChecks("acme", "web", sha, os.Getenv("GITHUB_TOKEN"), ""), ) defer func() { _ = n.Close() }() err := senro.Run(ctx, p, senro.WithSink(n)) ``` ```go func GitHubChecks(owner, repo, sha, token, name string, opts ...DestinationOption) *Destination ``` | Argument | What it is | |---|---| | `owner`, `repo` | The repository, `acme` and `web` for `github.com/acme/web`. | | `sha` | The full commit SHA the check attaches to. | | `token` | A token with `checks:write`. In GitHub Actions, `${{ github.token }}` has it. | | `name` | The check's name in GitHub's UI. Empty means `senro`. | senro discovers none of these. It holds no GitHub credential of its own and does not read the environment looking for one, so the SHA and the token come from wherever your dispatcher got them: ```go sha := os.Getenv("GITHUB_SHA") ``` If a [trigger event](/docs/triggers/events/) started the run, the SHA is on the event too: ```go ev, err := trigger.LoadEvent(*eventPath) // ... notify.GitHubChecks("acme", "web", ev.Base.To, os.Getenv("GITHUB_TOKEN"), "senro ci") ``` ## What appears on the commit - **When the run starts**, the check is created and shows as in progress. - **When the run finishes**, it is completed with a conclusion, a summary counting the steps, and one annotation per failed step pointing at what broke. **Steps do not each cost a request.** A thousand-step run would be a thousand calls into GitHub's secondary rate limits, so failures accumulate locally and travel with the completion. Two requests per run, whatever its size. Past GitHub's cap of 50 annotations, the summary says how many were left out, rather than showing a short list that reads like a complete one. ## Run status becomes a conclusion A run status this build does not recognise becomes `neutral`, not `failure`. Blocking a merge over a status GitHub's own UI cannot explain would be worse than saying nothing. See [Step states](/docs/steps/states/#the-runs-own-rollup) for the five statuses a run can end with. ## GitHub Enterprise ```go notify.GitHubChecks(owner, repo, sha, token, "", notify.GitHubChecksAPI("https://github.example.com/api/v3")) ``` The default is `https://api.github.com`. ## Options Options are applied last, so [every one](/docs/notifications/#options) works, including `Named`, `Client`, `Retry` and `Timeout`. senro's own headers are still set after them. `On` is the exception worth knowing about: this destination decides for itself which events matter (the run starting, steps finishing, the run finishing) and ignores the rest, so narrowing it with `On` only takes things away. ## Where your token lives The token is sent as an `Authorization` header, which senro's redactor never sees: redaction covers **event payloads**, and a header is not one. It is held and sent by this process and nothing else. Keep it in an environment variable or a secret store, never in source. ## Where to go next - **[Triggers](/docs/triggers/)**: the other half of the loop, where the SHA comes from. - **[Notifications](/docs/notifications/)**: retries, drops, and how a failed delivery is reported. - **[Write your own](/docs/notifications/custom/)**: `notify.Requester`, the seam GitHub Checks is itself built on. --- > Source: https://xavidop.github.io/senro/docs/triggers/github # GitHub triggers `provider: "github"`. Reads GitHub's `push` and `pull_request` webhook bodies. ## Wire it Nothing to register: `github` is one of the five sources [`LoadEvent`](/docs/triggers/events/#load-one) already knows. ```go ev, err := trigger.LoadEvent(*eventPath) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")), trigger.OnPullRequest(trigger.Actions("opened", "synchronize")), trigger.OnTag(trigger.Semver(">=1.0.0")), )) ``` ## Write the event file The `event` field is GitHub's `X-GitHub-Event` header, verbatim. From a GitHub Actions workflow: ```yaml - name: Build the event file run: | jq -n --arg e "$GITHUB_EVENT_NAME" --slurpfile p "$GITHUB_EVENT_PATH" \ '{provider:"github", event:$e, payload:$p[0]}' > event.json - run: go run ./ci --trigger-event event.json ``` From a webhook receiver, `$e` is the `X-GitHub-Event` header and the payload is the request body. ## What it accepts | `event` | Becomes | Notes | |---|---|---| | `push` | `Push`, or `Tag` when the ref is `refs/tags/...` | Carries a changed-file list | | `pull_request` | `PullRequest` | No changed-file list | ## Worth knowing **There is no separate GitHub event for a tag.** A pushed tag arrives as a `push` whose `ref` is `refs/tags/v1.2.3`, so senro reads the kind from the ref and `OnTag` matches it. Forward the `push`; GitHub's `create` for the same tag carries strictly less, and senro does not read it. ```json {"provider":"github","event":"push", "payload":{"ref":"refs/tags/v1.2.3","before":"000...","after":"fd48986..."}} ``` ```go trigger.OnTag(trigger.Semver(">=1.0.0")) // matches the push above ``` **A push that deleted a ref never matches any trigger.** There is nothing to build at a ref that no longer exists. **A `pull_request` payload carries no changed-file list**, so `Paths` against one is an [error, not a no-match](/docs/triggers/events/#when-there-is-no-file-list): ```go trigger.OnPullRequest(trigger.Paths("services/**")) // errors on every GitHub PR event ``` Narrow a pull request run with an [affected set](/docs/monorepo/affected/) instead, which is more precise anyway: it catches a shared library change that breaks a service containing none of the changed files. **`Branches` on a pull request tests the base branch**, not the head, the same question GitHub Actions' `branches:` filter answers. ## Actions GitHub's own words, untranslated: ```go trigger.OnPullRequest(trigger.Actions("opened", "synchronize", "reopened")) ``` `synchronize` fires when an open pull request gets new commits, so you almost always want it alongside `opened`. ## GitHub Enterprise Same provider, same payloads. `trigger.GitHub()` is exported if you want to hold one, wrap it, or fill in something a body cannot say: ```go ev, err := trigger.ReadEvent(r, myWrapper{inner: trigger.GitHub()}) ``` ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers, the three outcomes, and what a match carries into the run. - **[The event file](/docs/triggers/events/)**: the envelope every source shares. - **[GitHub Checks](/docs/notifications/github-checks/)**: reporting the result back on the commit. --- > Source: https://xavidop.github.io/senro/docs/triggers/gitlab # GitLab triggers `provider: "gitlab"`. Reads GitLab's push, tag push and merge request hooks. ## Wire it ```go ev, err := trigger.LoadEvent(*eventPath) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")), trigger.OnPullRequest(trigger.Actions("open", "update")), trigger.OnTag(trigger.Semver(">=1.0.0")), )) ``` **A GitLab merge request is senro's `PullRequest` kind.** There is one vocabulary across every source, so there is no `OnMergeRequest`: you declare `trigger.OnPullRequest(...)` and it matches GitLab's merge request hooks. ## Write the event file `event` takes either spelling: GitLab's `object_kind` (`push`, `tag_push`, `merge_request`) or the `X-Gitlab-Event` header (`"Push Hook"`, `"Tag Push Hook"`, `"Merge Request Hook"`). ```json {"provider":"gitlab","event":"merge_request","payload":{ ...GitLab's body... }} ``` ## What it accepts | `event` | Becomes | |---|---| | `push` / `Push Hook` | `Push` | | `tag_push` / `Tag Push Hook` | `Tag` | | `merge_request` / `Merge Request Hook` | `PullRequest` | ## Worth knowing **The actions are GitLab's own words.** `open`, `close`, `reopen`, `update`, `merge`, `approved` and the rest, never GitHub's `opened` or `synchronize`: ```go trigger.OnPullRequest(trigger.Actions("open", "update")) // yes trigger.OnPullRequest(trigger.Actions("opened")) // never matches a GitLab event ``` **The body has to agree with the envelope.** A payload whose `object_kind` contradicts the `event` field is refused, rather than parsed as whichever was read first. **Deletions and creations are the all-zero SHA**, not a flag: `after` is all zeros for a deletion, `before` for a creation. A deleted ref never matches any trigger. **The commit list is truncated at 20.** The real count is in `total_commits_count`, and a push carrying more commits than that has an incomplete changed-file list. senro reports it as **no list at all**, so `Paths` errors rather than quietly matching against a partial list: ``` trigger: Paths was asked of a push whose provider supplied no changed-file list ``` Either narrow with `Branches` instead, or fetch the real list and pass it through [the neutral shape](/docs/triggers/manual/). **A merge request payload carries no changed-file list either**, so `Paths` against one is always an error. Use an [affected set](/docs/monorepo/affected/). **A merge request names no commit on the target branch**, so the base is a `to` with no `from`. `object_attributes.oldrev` is the previous head of the *source* branch, which answers a different question, so senro does not use it. ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers, and what a match carries into the run. - **[The event file](/docs/triggers/events/)**: the envelope every source shares. - **[Write your own](/docs/triggers/custom/)**: layering a GitLab event this build does not read, such as a `Note Hook`, on top of `trigger.GitLab()`. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/glob # `glob` One unit per directory matching a pattern. The simplest graph there is, and the one to reach for when your repository has a convention but no manifest describing it. ```go import "github.com/xavidop/senro/unit/glob" verify.Expand("lint", glob.Dirs("apps/*")). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("make", "lint")).WorkDir(u.Dir) }) ``` ## The two constructors | | | |---|---| | `glob.Dirs(pattern)` | One unit per matching **directory**. | | `glob.Files(pattern)` | One unit per directory that **contains** a matching file. | ```go glob.Dirs("apps/*") // apps/web, apps/api, apps/admin glob.Files("services/*/go.mod") // one unit per service that has a go.mod glob.Files("**/Dockerfile") // one unit per directory holding a Dockerfile ``` Two matching files in one directory still produce **one** unit. Patterns use senro's standard [path pattern syntax](/docs/data/workspaces/#pattern-syntax): `*` and `?` match within a path segment, `**` spans segments. ## What a unit looks like `ID`, `Name` and `Dir` are all the slash-separated path relative to the root: ```go senro.Unit{ID: "apps/web", Name: "apps/web", Dir: "apps/web"} ``` So `u.Base()` is `"web"`, which is usually what a deployment step wants to name. ## It cannot narrow to affected units A path does not say what it imports, so `glob` knows nothing about which unit depends on which. `Affected` over it is **refused at build time**: ``` senro: expansion "lint": unit: glob dirs apps/* cannot compute an affected set: it discovers units but knows nothing about which unit depends on which. ``` That is the honest answer, not a limitation being worked around. If you want a narrowed run, fan out over a graph that reads your ecosystem's manifests: [`gowork`](/docs/monorepo/unit-graphs/gowork/), [`cargo`](/docs/monorepo/unit-graphs/cargo/), [`jswork`](/docs/monorepo/unit-graphs/jswork/), [`maven`](/docs/monorepo/unit-graphs/maven/) or [`gradle`](/docs/monorepo/unit-graphs/gradle/). Fanning out over everything is still worth having: twenty apps linting in parallel with a `MaxParallel` cap beats one `for` loop in a shell script. ## When it is the right choice - Your repository's layout is a convention (`apps/*`, `services/*`) with no manifest naming the members. - Every unit is genuinely independent, so there is nothing to narrow anyway. - You are mixing ecosystems and just want "one step per directory that has a `Makefile`". ## Where to go next - **[Fan-out](/docs/monorepo/fan-out/)**: `Template`, `MaxParallel`, `MaxNodes` and per-unit edges. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. - **[Write your own](/docs/monorepo/unit-graphs/custom/)**: when your layout is described by a file only you have. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/gowork # `gowork`: Go One unit per Go module, or per Go package. **Computes an affected set**, so a change to one package runs only the packages that import it, transitively. ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/gowork" ) verify.Expand("test", gowork.Modules()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) ``` ## Modules or packages | | A unit is | `Name` is | |---|---|---| | `gowork.Modules()` | One per `go.mod` | The module path, `github.com/acme/api` | | `gowork.Packages()` | One per Go package, finer grained | The import path, `github.com/acme/api/internal/store` | `Modules()` is usually right for separately released services. `Packages()` pays off in a large single module, where a change to one package should not retest the other four hundred. Both come out of **one** listing, so they can never disagree: `Modules` is `Packages` collapsed onto module directories, edges included. ```go // One step per module: go test ./... inside each. verify.Expand("test", gowork.Modules()). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) // One step per package: go test on the package by import path. verify.Expand("test", gowork.Packages()). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", u.Name)) }) ``` ## It runs `go list` This is the one shipped graph that uses the ecosystem's own tool, so it needs a `go` binary on `PATH`. In exchange, its answer is the toolchain's answer rather than a manifest parse. Under the hood: `go list -deps -test -e -json`, once per module found under the root, in that module's own directory. - **`-deps`** reaches a package in another workspace module, so cross-module edges are real edges. - **`-test`** because a test-only import is still an import. A graph missing them would disconnect packages that only a test connects. `TestImports` and `XTestImports` are both read. A large listing takes seconds and is exactly the call somebody presses Ctrl-C on, so the graph honours its context. ## Worth knowing **`go.work` is not required.** Module discovery is a walk for `go.mod` files, so a repository with several modules and no workspace file works fine. **A tree with no `go.mod` anywhere is an error**, not an empty graph. An expansion that silently produced no steps looks exactly like one that passed. **A `go.mod` change affects every package in its module.** A dependency bump changes what all of them compile against, so `Owns` attributes the file to all of them. **A file no unit owns runs everything.** A `Makefile` above every module genuinely can change what all of them build, so senro over-approximates rather than guessing. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows, and where `change` comes from. - **[Per-unit edges](/docs/monorepo/needs-each/)**: `test[unit=api]` waiting on `build[unit=api]`. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/gradle # `gradle`: Java One unit per Gradle project. **Computes an affected set.** Nothing is run: no Gradle, no daemon, no JDK. ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/gradle" ) verify.Expand("test", gradle.Projects()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("./gradlew", u.Name+":test")) }) ``` | Field | For `:libs:core` | |---|---| | `Name` | `:libs:core`, the project path, which is what `./gradlew :libs:core:test` takes | | `ID`, `Dir` | The project directory, which a `settings.gradle` can put anywhere | ## It reads the declarative subset of your build `settings.gradle(.kts)` and `build.gradle(.kts)` are programs, not data. Most of them are a list of literal `include` calls and literal `project(":libs:core")` dependencies, and the graph reads exactly that, in both Groovy and the Kotlin DSL. **When a build computes something instead of naming it, the graph refuses rather than guesses.** The error wraps `gradle.ErrNotDeclarative` and says which file it stopped at, so `errors.Is` can tell that case apart from a broken build: - A build script that **computes the project a dependency points at** makes the affected set refuse. There is no one project to attribute the edge to, and the only remaining reading is "every project depends on every project", which is this feature switched off while still looking switched on. - A `settings.gradle` with more includes than the bound is treated as **generated**, and refused for the same reason. - An `include` that names no project, a `projectDir` reassignment for a project no `include` creates, a project outside the root, or two projects sharing one directory are each an error naming what it found. Reading only the literal parts of a generated build would produce a plausible **short** project list, and a short list is an affected set that skips the project a change broke. Refusing is recoverable; a plausible wrong answer is not. The message tells you the way out: **fan out over these units without `Affected` and run every one.** Discovery still works; only the narrowing is refused. ## Worth knowing **Type-safe project accessors are resolved**, so `implementation(projects.libs.core)` draws the same edge `project(":libs:core")` does. **A project dir override moves the unit.** `project(':app').projectDir = file('src/app')` puts the unit at `src/app`, and the graph follows it. **A container project runs the projects under it.** A change to `:libs` runs `:libs:core` and its siblings. **A root file affects everything**: `build.gradle`, `gradle.properties`, the version catalog. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows. - **[Sharding](/docs/monorepo/partition/)**: splitting a large project list across runners. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/steps/handlers # Handlers A handler is what runs after a step, to clean up or to collect evidence. `OnFailure` runs its handlers once retries are exhausted and the step still failed; `Always` runs its handlers whatever the outcome. ```go cleanup := senro.Handler("release-lock", exec.Command("./scripts/unlock.sh")) diagnostics := senro.Handler("dump-log", exec.Command("cat", "test-results.log")) deploy.Step("deploy", exec.Command("./deploy.sh")). Retry(2, retry.OnInfra()). OnFailure(diagnostics). Always(cleanup) ``` - Handlers run **in order**, as listed. - **A handler failing never masks the step's own failure.** The run's recorded cause stays the original step, and the handler's failure is reported alongside it. ## Handlers are not steps `senro.Handler(id, action)` returns a `*senro.StepBuilder` deliberately **not attached to any workflow**. Passing a `Workflow.Step(...)` return value to `OnFailure` or `Always` is rejected at `Build()`, because that step would then run twice: once on its own, once as the handler. The error names exactly this and suggests `senro.Handler`, so the mistake is caught at build time rather than as a double-executed cleanup in production. ## What a handler may declare A handler is a `*StepBuilder`, so every step method is within reach. The ones with no meaning for a handler are refused by `Build()` rather than quietly dropped. | On a handler | Status | |---|---| | `Env`, `SecretEnv`, `WorkDir`, `Timeout` | Work as on any step | | `Needs` | Refused: `plan: handler "notify" of step "s" must not declare Needs` | | `Mount` | Refused: `plan: handler "collect" of step "s" declares its own mounts; a handler already has its parent's workspaces, mounted read-only at the same paths` | | `When`, `Retry`, an executor of its own, cache settings, handlers of its own | Refused | ## A handler can be a Go function A handler's action is a [step action](/docs/steps/#the-two-step-kinds), so it can be a `senro.Func` as easily as a command: ```go type CollectParams struct { Bucket string `json:"bucket"` } func init() { senro.RegisterFunc("ci/collect", Collect) } func Collect(ctx senro.Ctx, p CollectParams) error { f, ok := ctx.Failure() if !ok { return errors.New("ci/collect only makes sense as a handler") } fmt.Fprintf(ctx.Stdout(), "%s ended %s (exit %d) on attempt %d\n", f.Step, f.State, f.ExitCode, f.Attempt) // Classify without opening the log file: the tail came with the failure. if strings.Contains(f.LogTail, "no space left on device") { return upload(ctx, p.Bucket, f.Step, f.LogTail) } return nil } // in the pipeline: deploy.Step("apply", exec.Command("./deploy.sh")). OnFailure(senro.Handler("collect", senro.Func("ci/collect", CollectParams{Bucket: "ci-evidence"}))) ``` ### `ctx.Failure()` This is the func equivalent of the `SENRO_FAILURE_*` variables an `exec` handler reads. It returns a `senro.StepFailure` and an `ok` that is **false for an ordinary step**, so a function used both ways can tell which it is. | Field | What it holds | |---|---| | `Run` | The run's id | | `Step` | The id of the step this handler belongs to. The **parent's**, not the handler's | | `State` | Its terminal state, one of [the ten](/docs/steps/states/) | | `ExitCode` | The exit code it ended on | | `Attempt` | The attempt the step actually reached. `0` for a node that never ran one | | `Error` | The substrate's own message when the attempt failed to run at all, and empty when the step ran and returned a verdict | | `LogTail` | The tail of the failed attempt's combined output | `Error` and `LogTail` are the two an environment has no room for, which is why an `exec` handler has to go and open the log file and a function does not. > An **`Always` handler** gets a `Failure` too, describing whatever happened, so `State` is how it > tells a passing step from a failing one. `ok` says "I am a handler", never "something broke". ### It runs on the parent's executor A `senro.Func` handler runs wherever its parent ran: the coordinator, an ssh host, a container, a pod. senro stages the binary on the parent's target and re-enters it there, exactly as it does for a func step, **reusing the copy the parent step already staged**. `ctx.Workspace(...)` and `ctx.Secret(...)` report paths on that machine. The one refusal left is a func handler that declares a **delegated** secret, for the same reason a func step cannot: delegation hands the pod a source URI for the step's own *command* to resolve, and a function reads `ctx.Secret(name)`, which is a file senro wrote. See [Func steps off the coordinator](/docs/executors/func-remote/). ## What a handler inherits A handler runs on the **same executor** as its step, with the **same workspaces**, mounted read-only at the same paths. That is what makes a diagnostic handler worth attaching: it collects evidence from the environment that actually broke. ```go src := senro.Workspace("src", senro.Scope(senro.ScopeRun)) deploy.Step("build", exec.Command("./build.sh")). Mount(src.At("/repo", senro.RW)). WorkDir("/repo"). OnFailure(senro.Handler("dump-log", exec.Command("cat", "build.log"))) ``` `build.sh` writes `build.log` into `/repo`; the handler reads it back from the same `/repo`, in a sandbox of its own on the same executor. For a container step that is a fresh container from the parent's image with the same bind mount. **A handler with no `WorkDir` starts in the directory the step started in**, which is what lets the unqualified `cat build.log` find the file. - **The view is read-only.** The step's `ws.snapshot` digest was recorded while its sandbox was still open, and nothing snapshots again afterwards, so a handler write would move bytes the ledger already describes. The container executor enforces this as a read-only bind mount; the local executor cannot, exactly as for a step's own `senro.RO` mount. Write to the handler's own working directory. - **Only declared workspaces are inherited.** A file the step wrote outside them went into the executor's private sandbox, which for a container no longer exists when the handler starts. Evidence a handler must read belongs in a workspace. Scratch caches are not inherited: they are caches, not evidence. - **A handler cannot declare mounts of its own.** It already has its parent's, so `Build()` refuses a declaration that could only restate or contradict them. ## The failure arrives in the environment An `exec` handler reads what broke out of four variables: | Variable | What it holds | |---|---| | `SENRO_FAILURE_STEP` | The id of the step that failed | | `SENRO_FAILURE_STATE` | Its terminal state, from [the ten](/docs/steps/states/) | | `SENRO_FAILURE_EXIT_CODE` | The exit code it ended on | | `SENRO_FAILURE_ATTEMPT` | The attempt the step actually reached, so a handler can find that attempt's log. `0` for a node that never ran an attempt, skipped or cancelled before it started | ```sh #!/bin/sh echo "$SENRO_FAILURE_STEP ended $SENRO_FAILURE_STATE" \ "(exit $SENRO_FAILURE_EXIT_CODE) on attempt $SENRO_FAILURE_ATTEMPT" ``` A [func handler](#a-handler-can-be-a-go-function) reads the same evidence through `ctx.Failure()`, plus the error text and the log tail. These four variables are unchanged either way: a func handler does not take them away from anything. ## Where to go next - **[Retries](/docs/steps/retries/)**: what has to be exhausted before `OnFailure` fires. - **[Step states](/docs/steps/states/)**: what `SENRO_FAILURE_STATE` can say. - **[Go functions as steps](/docs/steps/functions/)**: `senro.Func`, `senro.Ctx` and `RegisterFunc`. - **[The event stream](/docs/run/event-stream/)**: the `handler.started`, `handler.succeeded` and `handler.failed` events a handler run emits. - **[Workspaces](/docs/data/workspaces/)**: what a handler gets to read back. --- > Source: https://xavidop.github.io/senro/docs/install # Install The CLI is one binary. The library is a Go module. You need the library to write a pipeline, and the CLI only to run, attach to or inspect one. ## The library ```bash go get github.com/xavidop/senro ``` That is all a pipeline needs. A pipeline is an ordinary `main` package that imports `senro`, so `go run ./ci` executes it with no daemon and nothing else installed. See [Quickstart](/docs/quickstart/). ## The CLI Pick whichever fits. All four give you the same binary. ### Homebrew ```bash brew install xavidop/tap/senro ``` ### go install ```bash go install github.com/xavidop/senro/cmd/senro@latest ``` This builds from source, so it needs the Go toolchain and puts the binary in `$(go env GOPATH)/bin`. ### A released binary Every release publishes a tarball per platform, named `senro___.tar.gz`, holding the binary, `LICENSE` and `README.md`: | | `amd64` | `arm64` | |---|---|---| | linux | yes | yes | | darwin | yes | yes | ```bash VERSION=$(gh release view --repo xavidop/senro --json tagName -q '.tagName' | tr -d v) OS=$(uname -s | tr '[:upper:]' '[:lower:]') ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/') curl -fsSLO "https://github.com/xavidop/senro/releases/download/v${VERSION}/senro_${VERSION}_${OS}_${ARCH}.tar.gz" tar -xzf "senro_${VERSION}_${OS}_${ARCH}.tar.gz" senro sudo install senro /usr/local/bin/senro ``` ### From source ```bash git clone https://github.com/xavidop/senro cd senro go build -o senro ./cmd/senro ``` `senro help` prints the synopsis of every command. Full reference: [CLI](/docs/cli/). ## Verifying a download Each release carries `checksums.txt` (SHA-256), an SBOM per archive (`.sbom.json`, CycloneDX), and SLSA build provenance (`senro.intoto.jsonl`). Check the archive against the published checksums: ```bash curl -fsSLO "https://github.com/xavidop/senro/releases/download/v${VERSION}/checksums.txt" sha256sum --ignore-missing -c checksums.txt # shasum -a 256 -c on macOS ``` Verify the provenance with the GitHub CLI, which confirms the artifact was built by this repository's release workflow rather than uploaded by hand: ```bash gh attestation verify "senro_${VERSION}_${OS}_${ARCH}.tar.gz" --repo xavidop/senro ``` ## Requirements | | | |---|---| | Go | 1.26 or newer, for the library, `go install` and building from source | | Operating system | Linux or macOS, on `amd64` or `arm64` | | Windows | Not supported | A released binary needs no Go toolchain. Homebrew and `go install` need one only in the sense that `go install` compiles. Windows is not a supported target, deliberately: attach's security boundary is a kernel peer-credential check with no Windows equivalent implemented, and senro fails to build for it. See [Attach security](/docs/attach/security/#platform-support) for the detail. ## Optional extras The [AI failure analyzer](/docs/analyzers/genkit/) lives in its own module, so senro itself never pulls in an AI SDK. Install it only if you want it: ```bash go get github.com/xavidop/senro/contrib/genkitanalyzer ``` --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/jswork # `jswork`: JavaScript One unit per workspace package, across npm, pnpm, Yarn (classic and Berry) and Bun. **Computes an affected set.** ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/jswork" ) verify.Expand("test", jswork.Packages()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("pnpm", "--filter", u.Name, "test")) }) ``` `Name` is the package's `"name"` from its `package.json`, which is what `--filter` takes. ## One graph, four package managers Only **discovery** differs by manager, and senro reads both places at once: - the root `package.json`'s `"workspaces"` (an array, or Yarn v1's object with `"packages"`); - `pnpm-workspace.yaml`'s `packages` list. The two are unioned, because a pnpm root usually has no `"workspaces"` key at all. A `!` pattern excludes, from either. **The dependency graph does not differ by manager.** An edge is one `package.json` naming another package's `"name"` in `dependencies`, `devDependencies`, `peerDependencies` or `optionalDependencies`. Any version range draws the edge: `workspace:*`, `^1.2.3` and `*` alike. > The edge is on the package **name**, not the directory name. A package at `libs/core` named > `@acme/core` is depended on as `@acme/core`. **Lockfiles are not read.** Four formats, one binary, and none of them says anything about the workspace graph the manifests do not. ## The honest limit This is the **declared** graph. Nothing parses JavaScript, so an undeclared import gets no edge. That is the same hole turbo, nx, lerna and `pnpm --filter` have, so it is not a senro-specific risk. But it is a real one: if your repository hoists with npm or Yarn and relies on undeclared imports, the affected set will be wrong, and you should fan out over everything instead. TypeScript project references are a second route this graph does not read. ## Worth knowing **The root manifest and lockfile affect everything.** **A change to an excluded package affects everything**, because senro cannot tell what depends on something it does not track. **Deleted files are answered from their path alone**, never by stat'ing disk. A deletion is exactly the change whose dependents most need rebuilding, and the file is already gone by plan time. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows. - **[Fan-out](/docs/monorepo/fan-out/)**: `MaxParallel` for a workspace with a hundred packages. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/executors/kubernetes # Kubernetes `k8s.Pod(ref)` targets a workflow at a Kubernetes cluster. Every step runs as its own pod: one step, one pod, one container. The step's command becomes the container's command, its output comes back through the pod's log, and its exit code comes from the container's terminated status. ```go import "github.com/xavidop/senro/executor/k8s" runner := k8s.Pod("ghcr.io/acme/runner@sha256:9f2c1e8b…", k8s.Namespace("ci")) deploy := p.Workflow("deploy", senro.Needs("verify"), senro.On(runner)) deploy.Step("apply", exec.Command("helm", "upgrade", "--install", "web", "./chart")) ``` The digest and the namespace are both required. `Build()` refuses the pipeline if either is missing. ## Configure the cluster The cluster comes from environment variables read at run start. senro never reads `$KUBECONFIG` or `~/.kube/config`, and never uses your current context. If none of the variables below are set, the run fails at the start and tells you what's missing. | Variable | Required | What it is | |---|---|---| | `SENRO_K8S_SERVER` | yes | The apiserver's base URL, `https://10.0.0.1:6443` | | `SENRO_K8S_CA_FILE` | yes | PEM file holding the cluster's CA. senro will not skip verification | | `SENRO_K8S_TOKEN_FILE` | one of | File holding a bearer token | | `SENRO_K8S_TOKEN` | one of | The bearer token inline | | `SENRO_K8S_CLIENT_CERT_FILE` + `SENRO_K8S_CLIENT_KEY_FILE` | one of | A client certificate and its key | Prefer the file over the inline token. An environment variable is visible in `/proc//environ`, gets inherited by every child process, and can land in a crash dump. Inside the cluster, point at the projected files: ```sh export SENRO_K8S_SERVER=https://kubernetes.default.svc export SENRO_K8S_CA_FILE=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt export SENRO_K8S_TOKEN_FILE=/var/run/secrets/kubernetes.io/serviceaccount/token ``` In your namespace, the identity needs `create`, `get`, and `delete` on pods; `get` on `pods/log`; and `create` on `pods/exec` (this is how workspaces cross over, how a session enters a pod, and how a `Func` step's binary gets in). It also needs `create`, `patch`, and `delete` on secrets, plus `list` and `get` on nodes at cluster scope (used to read the execution platform). ## What runs where, at a glance | Behavior | On this executor | |---|---| | Image reference | Must be pinned to a digest, or `Build` refuses it | | Namespace | Must be stated with `k8s.Namespace`; no fallback to `default`. Two namespaces are two executors | | Workspaces | Carried into the pod and back, both directions | | `senro.RO` mounts | **Genuinely enforced**, read-only in the pod | | Secrets | Files under `/run/senro/secrets`, never fields of the pod | | Scratch caches | Carried into the pod and back, like a workspace. Two full transfers per step; see below | | `Func` steps | Yes: the pipeline binary is sent into the pod and re-entered there | | `senro shell` | Yes, `--tty` included, in a pod of its own | | stdout and stderr | Merged into one stream, since Kubernetes keeps one log per container | | The step's command | **Replaces** the image's `ENTRYPOINT`, where the container executor passes it as arguments to one. An image with a wrapper script behaves differently on the two | | Platform | Read from the cluster's nodes, not the image manifest. Nodes that disagree are refused until you say which you meant with `k8s.Platform("linux", "amd64")` | | Restarts | senro's alone. `restartPolicy` is `Never`, so the kubelet never restarts a command behind the engine's back | | Cache class | The image digest and the platform. Never the namespace, never the cluster | ## Workspaces cross into the pod, and come back A mounted workspace becomes an `emptyDir` volume at the path you declared. It's filled from the coordinator's copy before your step runs, and read back afterwards, so a Kubernetes step can receive a workspace a local step filled, and hand its output to a later step. Declaring the mount works the same way everywhere ([Workspaces](/docs/data/workspaces/)). ```mermaid flowchart LR Co["Coordinator"] -->|"tar in"| Init["init container
fills the emptyDir"] Init --> Step["step container
your command runs"] Step --> Idle["idle container
keeps the pod alive"] Idle -->|"tar out"| Co ``` Under the hood, this is a `tar` stream over the apiserver's `exec` subresource, in both directions, the same way `kubectl cp` works. There's no PVC, no reachable content store, and no second credential involved. A few things follow from that: - **Every byte crosses the apiserver twice per attempt.** A hundred-megabyte workspace means two hundred megabytes on a shared apiserver. For large inputs, it's better to fetch them inside the pod under `NoSnapshot()` instead. - **The step's image needs `sh` and `tar`**, so a distroless image can't carry a workspace. The [SSH executor](/docs/executors/ssh/) has the same requirement for a host. - **The pod gains two extra containers whenever a step mounts a workspace**, and none otherwise: an init container fills the volume before your step's container can start, and an idle container keeps the pod alive so the result can be read out. Both die with the pod either way. - **A mount carries exactly what a snapshot carries.** `.git` and `node_modules` are excluded, so they're not sent and don't come back. Your copy is replaced rather than merged. - **An interrupted transfer fails loudly.** The far side's `tar` exit status arrives on its own stream, and a stream that ends without one is treated as an error rather than a truncated success with the wrong digest cached. The recorded digest is always computed from the copy that actually came back. A [scratch cache](/docs/data/scratch/) crosses the same way and is read back before the run saves it, so what lands under the key is whatever the pod left behind. Two things differ: nothing is excluded from a scratch cache (`node_modules` is usually the point of it), and there's no digest, since a scratch cache never enters a cache key. Think before putting one on a pod: a dependency tree big enough to be worth caching is often big enough that carrying it across the shared apiserver twice per step costs more than just downloading it again. If the copy doesn't come back, the run saves nothing rather than storing the coordinator's stale copy under a key nothing can rewrite. Two ways to get more out of one. **Across runs and machines**, set `SENRO_REMOTE_SCRATCH` and the cache is kept in the bucket, so a coordinator with a cold disk fills the pod from the shared copy instead of from nothing ([Sharing scratch caches](/docs/data/scratch-sharing/)). Note this does not reduce what crosses the apiserver: the pod is still filled from the coordinator, which now merely has something to fill it with. **Within one run**, a pod can share a cache with a local or container step as long as a `Needs` orders the two ([handing one over](/docs/data/scratch/#handing-one-between-a-remote-step-and-a-local-one)). ## Persistent workspaces, with or without a claim A [persistent workspace](/docs/data/persistent/) works here with no PVC at all by default. `ScopePersistent` and `PersistentVolumeClaim` share a word and nothing else. The coordinator holds the canonical copy, fills the pod from it, and reads it back, with the same bounds, lease, and cache-key rule as a local run. The cost is a transfer twice per attempt, which hurts most on a large tree. `k8s.Claim` removes that cost, by naming a claim you've already created: ```go k8s.Pod(img, k8s.Namespace("ci"), k8s.Claim("build-cache", "senro-build-cache")) ``` The pod mounts the claim directly: no staging container, no reader, nothing carried. senro creates, binds, resizes, and deletes nothing: storage class, size, and access mode are cluster decisions with real money attached. You give up three things in exchange: - **A step mounting a claim can't be `Pure()`.** `Build` refuses it, naming the workspace and the claim, because the coordinator can't walk a tree that lives only in the cluster. No `ws.snapshot` is emitted either. - **Exclusion moves into the cluster**, as a `coordination.k8s.io` `Lease` in the namespace. Takeover is conditional on the lease's `resourceVersion`, so two racing coordinators still produce one winner. But it's unfenced: a holder partitioned from the apiserver can keep writing even after its lease is taken. - **Scheduling gets narrower.** `ReadWriteOnce` ties every mounting pod to one node, so a fan-out serializes. `ReadWriteMany` means a networked filesystem and real money, and senro checks neither. For an ordinary workspace, a claim isn't the fix for transfer cost. ## Secrets are files, and never fields of the pod A secret reaches a Kubernetes step the same way it reaches every step: as a file under `/run/senro/secrets`, with its path in the step's environment ([Secrets](/docs/secrets/)). It's delivered as a namespaced `Secret`, created for one attempt, projected at mode `0400`, and mounted read-only. - **The value is never in the pod spec.** Not in `env`, `envFrom`, a command argument, or an annotation. A pod spec is readable by anyone with `get pods`, is what `kubectl describe` prints, and is what support bundles and admission webhooks collect, so an `envFrom` value would survive in every dump of it forever. - **The Secret is short-lived**: created for one attempt, marked immutable, and deleted when the sandbox closes on every path. It has an `ownerReference` to its pod, so the apiserver's garbage collector removes it even if the coordinator is killed first. - **It's still weaker than a local file.** The value passes through the apiserver and lands in etcd, readable by anyone with `get secrets` there, and encrypted at rest only if your cluster has configured a provider for that. The step's declared environment is an ordinary pod field, and so is the [trace context](/docs/extend/exporter/) (`TRACEPARENT`, and `TRACESTATE` when the run has one). A traceparent doesn't name a principal or grant any access, so it's fine there. ### Delegating secrets `k8s.DelegateSecrets()` inverts the default. senro resolves nothing and creates no `Secret`. No value ever crosses the boundary: each secret's source URI arrives as `SENRO_SECRET__SOURCE`, and the pod fetches its own secret using its ServiceAccount's identity. ```go k8s.Pod(img, k8s.Namespace("ci"), k8s.ServiceAccount("senro-ci"), // annotated for IRSA, or bound via Pod Identity k8s.DelegateSecrets()) ``` (IRSA, IAM Roles for Service Accounts, and Pod Identity are AWS's and Azure's own mechanisms for letting a ServiceAccount assume a cloud identity without a static credential in the pod; senro doesn't set either one up for you.) This is refused without a ServiceAccount, since the namespace's default account is one every other pod already has. Whatever resolves the URI is up to the step itself. There are three costs: - **The pod gets a ServiceAccount token**, which senro otherwise refuses it (`automountServiceAccountToken: false`), and the step's command can read it. Naming a ServiceAccount alone does not turn delegation on by itself. - **senro can no longer tell you the secret arrived.** A push either delivers a value or fails the step. A delegation succeeds as far as senro knows, and can still fail inside your command. - **The redactor never sees the value**, so it can't scrub it from the step's log. ## What a failure means If the container actually ran, you get its exit code back, with no error and no retry from `retry.OnInfra()`. A container that ran reports its code no matter what happened to it: - the command exited non-zero, whatever the code - the container was killed for memory (`OOMKilled`, exit 137) - the command doesn't exist in the image (`StartError`) These count as infrastructure failures instead, which [`retry.OnInfra()`](/docs/steps/retries/) retries: - the image could not be pulled, or the reference is not a valid name - the container's configuration could not be built (usually a missing mounted secret) - the pod was evicted, its node stopped reporting, or it was deleted out from under the run - the apiserver could not be reached, or the run was cancelled - the pod did not start within five minutes, with the scheduler's own account of why, such as `0/12 nodes are available: 12 Insufficient cpu` ## `senro shell` is a pod of its own A session on a Kubernetes step is a second pod, created when you ask for one: the step's image, the step's workspaces staged into it read-only at the step's paths, the step's environment and working directory, and your command exec'd into a container held open for it. `--tty` works too, and the pty belongs to the container runtime, so your window size travels the same connection your input does. This is deliberately not the step's own pod. The step's pod projects the step's `Secret` and carries its `SENRO_SECRET_*` paths, so a session there would hand you a credential the engine is supposed to withhold. It also mounts workspaces the way the step asked for them, so a session there could rewrite bytes the run's digests already describe. What it costs: capacity for a second pod, one more workspace transfer across the apiserver, and `sh` in the image (plus `tar` when the step mounts anything): the same requirements as carrying a workspace. A claim-backed workspace is mounted by the session's pod too, so a `ReadWriteOnce` claim held on another node leaves the session pending until the pod can be scheduled beside it. No new RBAC is needed: `create` on `pods/exec` is already required to move a workspace. Once the run is over, there's nothing left to open a session in, on this executor or any other: `senro shell` needs the live engine that owns the workspaces. Use `senro ws pull` to write the files out instead ([The shell](/docs/attach/shell/)). ## `Func` steps run in the pod A [`senro.Func` step](/docs/steps/functions/) targeted at a pod runs in the pod. A function's body is compiled into your pipeline binary, so senro sends that binary in and re-enters it as a step child, exactly as it does over ssh and in a container ([Func off the coordinator](/docs/executors/func-remote/) covers the whole mechanism). What's specific to this executor: - **The binary crosses as a `tar` over `pods/exec`**, the same transport a workspace already uses, landing in an `emptyDir` at `/senro/bin`. No new RBAC, no registry, no reachable content store, and senro doesn't build your image. - **The step's container starts and holds, and the child is exec'd into it.** That's what keeps stdout and stderr apart, which a pod's log can't do on its own (the child's protocol is framed on stdout, with unframed diagnostics on stderr), and it's what gives the step an exit code of its own. - **The image needs `sh` and `tar`**, exactly as carrying a workspace does, and it must be able to run a static Linux binary for the node's architecture. senro cross-compiles with `CGO_ENABLED=0`, `-tags netgo,osusergo`, and a static link, so no dynamic loader is involved and a musl image works fine. A macOS coordinator therefore cross-compiles for every func step in a pod, and needs `senro.WithFuncBuild("./ci")` and a Go toolchain. - **One transfer per pod, which means per attempt.** A pod's filesystem doesn't outlive it, and senro keeps no cluster object to store a copy in, so `binary.staged` reports `reused: false` every time, unlike ssh, where the binary is staged once per host. The cross-compile itself is still cached, once per architecture per release. - **`k8s.DelegateSecrets()` is refused for a func step that declares a secret.** Delegation delivers a source URI in the pod's environment for the step's own command to resolve, but a function reads `ctx.Secret(name)`, the path of a file senro wrote. `Build()` catches this rather than handing the function an empty string. See [why the two can't mix](/docs/steps/functions/#why-delegated-secrets-and-func-steps-cannot-mix) for the two ways around it. ## Pod tuning Resource requests and limits, a node selector, tolerations, and image pull secrets are all unset by default: the pod gets whatever the namespace and the scheduler would otherwise give it. Four options on `k8s.Pod` declare them. ```go k8s.Pod(img, k8s.Namespace("ci"), k8s.Resources( map[string]string{"cpu": "500m", "memory": "256Mi"}, // requests map[string]string{"cpu": "1", "memory": "512Mi"}, // limits ), k8s.NodeSelector(map[string]string{"disktype": "ssd"}), k8s.Toleration("dedicated", "Equal", "ci", "NoSchedule"), k8s.ImagePullSecrets("regcred"), ) ``` `k8s.Resources` takes Kubernetes quantity strings exactly as the apiserver does (`"500m"`, `"256Mi"`); senro parses neither map, so a malformed one is reported by the apiserver rejecting the pod, not by `Build()`. It applies to the step's own container only: senro's staging and reader containers are its own plumbing, not part of what the pipeline asked to run, and a limit sized for the step would starve them. `k8s.Toleration` is called once per taint the target nodes carry; each call appends one. `Operator` is `"Equal"` (`Value` must match) or `"Exists"` (`Value` is ignored). `k8s.ImagePullSecrets` names a Secret that must already exist in the target namespace; senro creates, resizes, and deletes nothing, the same restraint as `k8s.Claim`. It is distinct from `container.RegistryAuth`, which this executor refuses at `Build()`: that credential is resolved by senro and pushed for the container executor to pull with directly, while this is a reference the node's own kubelet resolves, the ordinary way a pod pulls a private image. All four are part of the executor's instance key: two targets naming one image but disagreeing about any of them are two executors, not one sharing a resolve. ## What is not here - **A [scratch cache](/docs/data/scratch/) shared with a step on the coordinator's own filesystem when nothing orders the two.** Refused at `Build()`: a local or container step writes that directory while it runs, and a pod tarring the same directory at that same moment would send a half-written tree and then save it under a key nothing can rewrite. A `Needs` between them removes the same moment and the share is allowed, handing the cache from whichever runs first to whichever runs second. Two pods can still share one freely, ordered or not. If a coordinator is killed before cleaning up, every object senro creates is still findable. Each one carries the step's full id as the annotation `senro.dev/step-id`: ```sh kubectl -n ci get pods,secrets -l senro.dev/managed=true kubectl -n ci get pods -l senro.dev/run= ``` > senro ignores kubeconfig because it typically holds dozens of contexts, mostly pointed at > production, with whichever one you last used selected. An executor that defaulted to it could > deploy to the wrong cluster. The namespace isn't part of the cache class because the same bytes > come out of `ci` as out of `ci-staging`, and a class built from where you happened to run would mean > a fleet never shares a cache entry. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/maven # `maven`: Java One unit per Maven reactor project. **Computes an affected set.** ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/maven" ) verify.Expand("test", maven.Modules()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("mvn", "-pl", u.Name, "test")) }) ``` `Name` is `groupId:artifactId`, which is what `mvn -pl ` takes. ## What counts as a unit The root `pom.xml` plus the transitive closure of its ``, **profiles included**, since the graph cannot know which profiles a build will activate. **Aggregators are units too.** Leaving them out would lose the edge that makes a change to a parent pom run its children. ## Where the edges come from - `` at every scope, `test` included. - `` and ``. - A `import` BOM. - Plugins built inside the reactor. Coordinates are interpolated against the pom's own properties, its parents', and the `${project.*}` built-ins, so a version held in a property still resolves. **A `` entry is deliberately not a dependency** unless it is an imported BOM. Treating managed versions as dependencies would make every change run everything: the feature switched off while looking switched on. ## No Maven needed The graph reads the poms. No `mvn`, no JDK, no repository download on the machine planning the run. ## Worth knowing **The root pom affects everything**, which is usually what you want: it holds the versions everything else inherits. **A file under no module belongs to the root project**, so a change to a top-level script or config runs the reactor. ## Where to go next - **[Affected units](/docs/monorepo/affected/)**: what `Affected` narrows. - **[Sharding](/docs/monorepo/partition/)**: splitting a large reactor across runners. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/monorepo # Monorepos One repo, many units: packages, modules, crates, services. senro has four tools built for that shape, and each one solves a different problem. This page helps you pick the right one. ```go import ( "github.com/xavidop/senro" "github.com/xavidop/senro/change" "github.com/xavidop/senro/exec" "github.com/xavidop/senro/unit/gowork" ) verify := p.Workflow("verify") verify.Expand("test", gowork.Modules()). Affected(change.FromTrigger(ev)). MaxParallel(4). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) ``` One step per Go module. Only the modules this run's change actually reaches. Four at a time. ## The four tools | Tool | The problem it solves | |---|---| | [`Expand`](/docs/monorepo/fan-out/) | Writing one `Step` per package stopped scaling. One step per unit, discovered from the tree. | | [`Affected`](/docs/monorepo/affected/) | Every change runs every unit. Run only the units a change reaches, plus everything depending on them. | | [`NeedsEach`](/docs/monorepo/needs-each/) | One slow module holds up every other module's tests. One edge per unit instead of a barrier. | | [`Partition`](/docs/monorepo/partition/) | Fifty units and eight machines. Fewer steps than units, balanced by how long each one took last time. | They work together. A single expansion can be narrowed with `Affected`, ordered per unit against another expansion with `NeedsEach`, and split into shards with `Partition`, all at once. ```mermaid flowchart LR G["unit graph"] --> E["Expand
one step per unit"] E --> Aff["+ Affected
narrow to what changed"] Aff --> P["+ Partition
group into shards"] O["a second Expand"] -.->|NeedsEach| E ``` ## Which one is your problem? - **"I keep adding the same step for each new package."** Start with [`Expand`](/docs/monorepo/fan-out/) and stop there. It's useful on its own. - **"CI takes twenty minutes to tell me a docs typo is fine."** Use [`Affected`](/docs/monorepo/affected/), over a graph that reads your ecosystem's manifests. - **"The fan-out is fast, but the step after it waits for the slowest child."** Use [`NeedsEach`](/docs/monorepo/needs-each/). - **"Fifty tiny steps, each paying for its own container pull."** Use [`Partition`](/docs/monorepo/partition/). ## What a unit is A **unit graph** discovers units and, optionally, tells you who depends on whom. senro ships eight of them. Five of those can compute an affected set; the other three won't guess, so they refuse if you try. The full list, and the catch for each ecosystem, is in [The shipped unit graphs](/docs/monorepo/unit-graphs/). If none of the shipped graphs fit your layout, you can write your own. See [Implement a unit graph](/docs/monorepo/unit-graphs/custom/). ## Where to go next - **[Fan out with `Expand`](/docs/monorepo/fan-out/)**: `Template`, child ids, `MaxParallel`, `MaxNodes`. - **[Per-unit edges](/docs/monorepo/needs-each/)**: `NeedsEach` and how it differs from a barrier. - **[Partition](/docs/monorepo/partition/)**: shards and the duration history. - **[Running only what changed](/docs/monorepo/affected/)**: `Affected` and the `change` package. - **[Triggers](/docs/triggers/)**: where the change a run consumes comes from. --- > Source: https://xavidop.github.io/senro/docs/notifications # Notifications Telling somebody how a run went, when nobody is watching the terminal. senro ships three destinations, and one small interface for writing a fourth. ```go import "github.com/xavidop/senro/notify" n := notify.New( notify.Slack(os.Getenv("SLACK_WEBHOOK_URL")), ) defer func() { _ = n.Close() }() err := senro.Run(ctx, p, senro.WithSink(n)) ``` That is the whole setup: build a `*notify.Notifier` with the destinations you want, and hand it to `senro.Run` as a sink. `Run` flushes it before returning, so `run.finished` is on its way out before your process is. A runnable version with a receiving endpoint lives at `examples/notify` (`go run ./examples/notify`). ## The destinations | Destination | Sends | Gets, by default | |---|---|---| | **[Slack](/docs/notifications/slack/)** | A short line a person reads in a channel | `run.finished` only | | **[Webhook](/docs/notifications/webhook/)** | The raw `api.Event` as JSON, to any HTTP endpoint | Every event | | **[GitHub Checks](/docs/notifications/github-checks/)** | A check run on the commit, with annotations | The whole run | | **[Your own](/docs/notifications/custom/)** | Whatever bytes you render | What you declare | The opposite defaults for Slack and Webhook are deliberate: a webhook receiver is a program and can drop what it does not want; a person in a channel cannot. A Slack destination widened to every event on a two hundred step fan-out is two hundred messages. ## Options Every option below works on every destination, built-in or your own. | Option | What it does | Default | | --- | --- | --- | | `On(types...)` | Deliver only these event types. `On()` with no arguments delivers nothing, which is a tidy way to switch a destination off from configuration. | per destination | | `Named(name)` | The name used in `notify.*` events and the shutdown report. Set it when two webhooks would both be called "webhook". | the constructor's | | `Sign(secret)` | HMAC-SHA256 sign every request. See [Webhook](/docs/notifications/webhook/#verify-a-signed-request). | off | | `Retry(attempts, base)` | Total requests one event gets (not extras after the first), and the first backoff interval. | 3, 250ms | | `Timeout(d)` | Bounds one request. | 5s | | `Client(c)` | Your own `*http.Client`, for a proxy or a pinned CA. | - | | `ContentType(ct)` | The `Content-Type` header. | `application/json` | | `Header(name, value)` | An extra request header, for an API key or a routing key. Repeatable; repeating a name replaces its value. | - | Two more live on the notifier rather than a destination: `WithGrace(d)` sets how long the end-of-run flush waits for queued notifications before giving up (10s by default), and `WithReportWriter(w)` redirects the shutdown report away from standard error. ```go n := notify.New( notify.Slack(slackURL, notify.On(api.RunFinished, api.StepFinished)), notify.Webhook(hookURL, notify.Named("ci-bus"), notify.Sign(os.Getenv("SENRO_HOOK_SECRET")), notify.Retry(5, time.Second)), notify.WithGrace(30*time.Second), ) ``` ## What every request carries Whatever the destination, senro sets these headers: | Header | Meaning | | --- | --- | | `X-Senro-Event` | The event's type, so a receiver can route without parsing the body. | | `X-Senro-Run` | The run ID. | | `X-Senro-Seq` | The event's sequence number within that run. | | `X-Senro-Delivery` | `/`: the deduplication key. | | `X-Senro-Timestamp` | Unix seconds, on a signed request only. | | `X-Senro-Signature` | The signature, on a signed request only. | **Delivery is at-least-once.** A request that times out may well have been processed, and there is no way to tell from this side, so senro retries it. A receiver that must act exactly once should deduplicate on `X-Senro-Delivery`, which identifies an event permanently. No answer at all, a 429 or any 5xx is retried. Any other 4xx is not, because a 400 will be a 400 next time too. Waits double and are jittered. ## A failed notification is visible Delivery never blocks the run. Each destination has a bounded queue and its own goroutine, and a full queue drops the event rather than waiting for room, so a wedged endpoint costs a build nothing. ```mermaid flowchart LR E[api.Event] --> F{On filter} F -->|wanted| Q[bounded queue] Q --> W[worker: render,
sign, send] Q -->|full| DR[notify.dropped] W -->|2xx| OK[notify.delivered] W -->|no answer, 429, 5xx| W W -->|gave up| FA[notify.failed] ``` Nothing is lost quietly. Every outcome becomes an event in the run's own stream carrying an `api.NotifyBody`: which destination, which event and its sequence number, the attempts, the HTTP status, the duration, the error, or a running total of drops. So `senro attach --run `, your own sink, or `events.jsonl` after the fact all show that the Slack message did not go out. ### The one outcome that is not an event `run.finished` is the run's last event by construction, so the outcome of delivering it cannot itself be an event: by the time it is known, the stream has closed. It is also the outcome you most want, so it is printed on standard error as the run shuts down: ``` senro notify: 1 delivery outcome arrived after this run's event stream closed, so it is reported here instead of in the ledger: slack: run.finished NOT delivered after 3 attempts in 6.2s: Post: context deadline exceeded ``` ## Secrets never reach a destination The engine redacts every event payload **before the event exists**, upstream of every sink, so a notifier receives an already-redacted event and has nothing left to do about it. See [Secrets](/docs/secrets/). The one thing the redactor cannot know is the destination URL, which for a Slack incoming webhook is the whole credential. Go's HTTP client puts the URL into every error it returns, so `notify` strips it out of every error it records or prints: it never appears in an event, in the shutdown report, or in a log line. ## Watching a run from your own code If you want the events themselves rather than an HTTP request, skip `notify` and use a sink directly: ```go err := senro.Run(ctx, p, senro.WithSink(senro.SinkFunc(func(e api.Event) { log.Printf("%d %s %s", e.Seq, e.Type, e.Step) }))) ``` `senro.WithSink` gives you every event the run appends to its ledger, in ledger order. It is repeatable and composes with `senro.WithAttach`. > **`Emit` must not block.** The engine calls it inline, holding the lock that makes an append and > its delivery a single atomic step, so a slow `Emit` slows the whole run and a wedged one stops > it. If your sink talks to anything, hand the event to a goroutine of your own and return. ## Where to go next - **[Slack](/docs/notifications/slack/)**, **[Webhook](/docs/notifications/webhook/)**, **[GitHub Checks](/docs/notifications/github-checks/)**: each destination in full. - **[Write your own](/docs/notifications/custom/)**: one method, with a complete PagerDuty example. - **[The event stream](/docs/run/event-stream/)**: the events a destination renders. - **[Writing a trace exporter](/docs/extend/exporter/)**: the other thing built on `WithSink`. --- > Source: https://xavidop.github.io/senro/docs/steps/ordering # Ordering senro has two things called `Needs`, at two levels. They are different, and mixing them up is the mistake this page exists to prevent. | Call | Names | What it means | |---|---|---| | `(*senro.StepBuilder).Needs(ids ...string)` | **steps** | One edge: this step waits for those steps | | `senro.Needs(names ...string)`, passed to `Workflow` | **workflows** | A barrier: every step of this workflow waits for every step of those workflows | **Passing a step id to the workflow-level `Needs` is refused at `Build()`**, naming both the workflow that asked and the name it asked for. A dangling step-level `Needs` is refused the same way. Neither one is silently ignored. ```go setup := p.Workflow("setup") setup.Step("install", exec.Command("pnpm", "install", "--frozen-lockfile")) verify := p.Workflow("verify", senro.Needs("setup")) // a workflow name verify.Step("lint", exec.Command("pnpm", "lint")) verify.Step("test", exec.Command("pnpm", "test")) // runs alongside lint verify.Step("build", exec.Command("pnpm", "build")).Needs("test") // a step id deploy := p.Workflow("deploy", senro.Needs("verify")) deploy.Step("ship", exec.Command("./deploy.sh")) ``` ```mermaid flowchart LR subgraph setup install end subgraph verify lint test build end subgraph deploy ship end install --> lint & test & build test --> build lint & test & build --> ship ``` ## Workflows disappear when you build Workflows are for **you**, not for the engine. `Build()` turns every workflow-level barrier into ordinary step edges and throws the grouping away. What actually runs is one flat graph of steps. So this: ```go setup := p.Workflow("setup") setup.Step("install", ...) verify := p.Workflow("verify", senro.Needs("setup")) verify.Step("lint", ...) verify.Step("test", ...) ``` builds into exactly the same plan as this: ```go one := p.Workflow("everything") one.Step("install", ...) one.Step("lint", ...).Needs("install") one.Step("test", ...).Needs("install") ``` Two things follow from that: - **Reorganizing your workflows is free.** Moving a step from one workflow to another, splitting one workflow into three, or renaming them changes nothing about the plan, so nothing about its digest, so nothing about your [cache](/docs/data/caching/). Group them however reads best. - **Adding a workflow-level `Needs` is not free.** It adds real edges between real steps, so it does change the plan and does invalidate cache entries downstream of it. It is also why **step ids must be unique across the whole pipeline**: once the grouping is gone, `verify`'s `test` and `build`'s `test` would be the same node. ## `Needs` on a step - **Multiple calls accumulate.** `Needs("a").Needs("b")` is `Needs("a", "b")`. - **`Build()` rejects an id that doesn't exist**, before a single step runs. - **Unrelated branches are not cancelled when one fails.** A failing step settles its dependents as `skipped_upstream_failed`; everything else runs to completion, so a failure produces one clear report instead of a half-explored graph. See [Step states](/docs/steps/states/). - **`ContinueOnError()` on a step lets its dependents run anyway**, receiving whatever outputs it produced. See [Step settings](/docs/steps/settings/). ## What `Build()` refuses All five are caught before a single step runs. **A cycle.** Two steps waiting for each other can never both start. ```go verify.Step("a", ...).Needs("b") verify.Step("b", ...).Needs("a") // plan: dependency cycle: a -> b -> a ``` **A duplicate step id.** Because [workflows disappear](#workflows-disappear-when-you-build), two steps called `test` are one node with two definitions. ``` senro: step id "test" is declared in both workflow "build" and workflow "verify"; step ids are unique across the whole pipeline ``` **A `Needs` naming a step that does not exist.** Usually a typo, and the error names the dangling id. Silently ignoring it would mean the step runs immediately instead of waiting. **`senro.Needs` naming something that is not a workflow.** The workflow-level `Needs` takes **workflow names**; passing it a *step* id is the single most common mistake with this API: ```go setup := p.Workflow("setup") setup.Step("install", ...) verify := p.Workflow("verify", senro.Needs("install")) // "install" is a STEP, not a workflow ``` ``` senro: workflow "verify" needs workflow "install", which pipeline "ci" does not declare. senro.Needs names workflows, not steps; use (*senro.StepBuilder).Needs for a dependency on a step ``` Write `senro.Needs("setup")` to wait for the whole workflow, or move the dependency down to the step: `verify.Step("lint", ...).Needs("install")`. **A `Needs` on a handler.** A [handler](/docs/steps/handlers/) is the cleanup or diagnostic step you attach with `OnFailure` or `Always`. It is not a node in the graph: it runs when its parent step settles, and nothing else can wait for it or be waited on by it. So there is nothing for a `Needs` on one to mean: ```go senro.Handler("notify", ...).Needs("build") // plan: handler "notify" of step "s" must not declare Needs ``` ## Ordering a fan-out A **fan-out** (or expansion) is one `Expand` call that generates many steps at once, one per app, module or package in your repository. It is the [monorepo](/docs/monorepo/) feature; skip this section if you are not using it. An expansion is one declaration but many steps, so it gets two ways to be ordered: | | | |---|---| | `(*ExpandBuilder).Needs(ids...)` | The whole fan-out waits for those steps. Every generated child waits. See [Fan-out](/docs/monorepo/fan-out/). | | `(*ExpandBuilder).NeedsEach(expansions...)` | One edge **per unit**: `test[unit=api]` waits on `build[unit=api]` and nothing else, so `api` can finish testing while `web` is still building. See [Per-unit edges](/docs/monorepo/needs-each/). | ## Where to go next - **[Step settings](/docs/steps/settings/)**: `ContinueOnError` and the rest of the builder. - **[Step states](/docs/steps/states/)**: how a failure or a skip travels down the graph. - **[Conditions](/docs/steps/conditions/)**: pruning branches of the graph at run start. --- > Source: https://xavidop.github.io/senro/docs/monorepo/partition # `Partition`: fewer steps than units Some fan-outs want fewer steps than units: fifty modules but only eight machines, say, or a per-unit step whose startup cost dwarfs the actual work. `.Partition(n, history)` groups the units into at most `n` buckets and makes **one step per bucket**. ```go import "github.com/xavidop/senro/duration" verify.Expand("test", gowork.Modules()). Partition(8, duration.FromFile(".senro/durations.json")). TemplateShard(func(sh senro.Shard) *senro.StepBuilder { return senro.NewStep(exec.Command(append([]string{"go", "test"}, sh.Dirs()...)...)). Pure().Inputs(sh.Sources()...) }) ``` A partitioned expansion takes `TemplateShard` rather than [`Template`](/docs/monorepo/fan-out/#what-a-template-receives), since a bucket holds several units. A `senro.Shard` carries `Index`, `Total` and `Units`, plus `IDs()`, `Names()`, `Dirs()` and `Sources()` for handing the whole bucket to a command or to `.Inputs(...)`. > **Why balance by duration.** A round-robin or alphabetical split often puts the three slowest > modules in the same shard, and then the whole fan-out takes as long as that one shard. Weighing > each unit by how long its step took last time keeps that bounded, where a naive split doesn't. ```mermaid flowchart LR subgraph units["units, weighted by last run's duration"] direction TB u1["auth (2m)"] u2["search (2m)"] u3["billing (30s)"] u4["docs (10s)"] end u1 --> s0["shard 0"] u3 --> s0 u2 --> s1["shard 1"] u4 --> s1 ``` The two slow units land in different shards instead of stacking up in one, so no single shard is stuck carrying both. ## Where the history comes from `duration.Record(runDir, path)` folds a previous run's event stream into a small JSON file. `duration.FromFile(path)` reads it back. Record it deliberately, after a run, and **commit the file like a lockfile**. - Per-machine histories would give two machines two different plan digests and two sets of cache keys, so your shared cache would stop being shared. A committed file also makes the effect reviewable in a diff. - **`Record` merges** rather than replacing, so a run narrowed by [`Affected`](/docs/monorepo/affected/) doesn't discard the modules it never touched. - **Only steps that ran to completion are recorded.** A cached step finishes in milliseconds, and recording that time would make the slowest module look free. ## The first run, and unmeasured units - **No history** (first run, or a missing file) is fine. Every unit weighs the same, and the split falls back to round robin over the sorted unit set: the same split you'd have had anyway. - **A unit missing from a non-empty history** is estimated at the **median** of the units that are there. Zero would make it weightless, and using the maximum would treat every new module as the slowest one. - **A file that can't be read or parsed, or whose format version is unknown,** is an **error**, and fails the build. Treating it as empty would quietly revert the whole fleet to balancing by count. - **`duration.None()`** is the explicit "there is no history". ## Shard ids don't move when the history does A child is named `test[shard=0]`: numbered, never named after its contents. The number of shards is `min(n, number of units)`. Two machines with different histories still build the same step ids. If an id moved along with the timing, it would drag every cache key hanging off it along too. What the history *does* move is which unit lands in which bucket, and with it each shard's command, inputs, cache key and the plan digest. That is correct: a step that runs three modules is not the step that ran two. ## Limits of a partitioned run - A shard's ten minutes can't be split back out among its five modules, so `duration.Record` **ignores** shard steps. When the numbers go stale, re-record from a run of the same expansion without partitioning: a nightly build, say, or a run with partitioning turned off. - **`MaxNodes` is still checked against the whole graph**, before partitioning. It's not a way around the guard. - [`NeedsEach`](/docs/monorepo/needs-each/) pairs a partitioned expansion by unit **set**: a shard waits on every upstream child covering any of its own units. ## Where to go next - **[Fan out with `Expand`](/docs/monorepo/fan-out/)**: the unpartitioned surface, `MaxNodes` included. - **[Caching](/docs/data/caching/)**: `Pure()`, `Inputs` and what a shard's cache key covers. - **[Running only what changed](/docs/monorepo/affected/)**: narrowing the unit set before it is bucketed. --- > Source: https://xavidop.github.io/senro/docs/monorepo/needs-each # Per-unit edges: `NeedsEach` `.Needs(...)` on an expansion, and the workflow-level `senro.Needs`, are both **barriers**: nothing downstream starts until *every* child of the fan-out has finished. `.NeedsEach(expansions ...string)` works differently. It adds one edge per unit. ```go verify := p.Workflow("verify") verify.Expand("build", gowork.Modules()). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "build", "./...")).WorkDir(u.Dir) }) verify.Expand("test", gowork.Modules()). NeedsEach("build"). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) ``` `test[unit=api]` waits only on `build[unit=api]`. It starts the moment that one finishes, even while `build[unit=web]` is still running. The fan-out pipelines instead of stalling. ## The two shapes ```mermaid flowchart LR subgraph barrier["Needs: the barrier"] direction TB b1["build[unit=api]"] --> t1["test[unit=api]"] b1 --> t2["test[unit=web]"] b2["build[unit=web]"] --> t1 b2 --> t2 end subgraph each["NeedsEach: one edge per unit"] direction TB b3["build[unit=api]"] --> t3["test[unit=api]"] b4["build[unit=web]"] --> t4["test[unit=web]"] end ``` A barrier is right when the downstream step consumes the whole fan-out, for example a step that publishes a manifest of every built image. It's wrong when the downstream work is itself per unit: one slow module then holds up every other module's tests. ## Rules - It takes **expansion ids**: the id you gave `Expand`, not step ids (`Needs` takes those). A name that matches no expansion is rejected at build time. Otherwise a `NeedsEach` could silently add no edges at all, leaving the fan-out with no ordering. - It's an **addition**, not a replacement. The barrier is still available, and the two work together: a child can have both `Needs("install")` and `NeedsEach("build")`. A pipeline that uses neither behaves exactly as before. - **Naming an empty expansion** (a glob that matched nothing) adds no edges. > **Put both expansions in the same workflow.** If they're in two separate workflows joined by > the workflow-level `senro.Needs`, you get entry-to-exit edges on top of the per-unit ones. > Those edges *are* a barrier, so nothing pipelines. ## When the two unit sets differ A module with no tests, or two expansions built over different graphs, can leave some units without a counterpart. Dropping the edge would let a step run before its input exists. Dropping the step would silently drop work. So senro does neither: - **A unit here with no counterpart there** keeps its step, and falls back to the whole-expansion barrier: it waits for *every* child of the named expansion. This can only add ordering, never remove it. If the two expansions are fully disjoint, this degenerates back to a plain barrier. - **A unit there with no counterpart here** is ordinary, not an error. It has no per-unit dependent. ## With a partitioned expansion `NeedsEach` pairs a [partitioned](/docs/monorepo/partition/) expansion by unit **set**: a shard waits on every upstream child covering any of its own units. ## Where to go next - **[Fan out with `Expand`](/docs/monorepo/fan-out/)**: `Needs` and the rest of the expansion surface. - **[Ordering steps](/docs/steps/ordering/)**: `Needs` on a step versus `senro.Needs` on a workflow. - **[Partition](/docs/monorepo/partition/)**: fewer steps than units. --- > Source: https://xavidop.github.io/senro/docs/data/persistent # Persistent workspaces `Scope(senro.ScopePersistent)` makes a [workspace](/docs/data/workspaces/) survive between runs. It's one directory on the machine, named after the workspace, and every later run that mounts that name starts from it. Use it for an expensive tree you don't want to rebuild every run. ```go mods := senro.Workspace("go-mod-cache", senro.Scope(senro.ScopePersistent), senro.MaxAge(7*24*time.Hour), senro.MaxSize(4<<30)) verify.Step("test", exec.Command("go", "test", "./...")). Mount(src.At("/src", senro.RW), mods.At("/root/go/pkg/mod", senro.RW)) ``` It works on every executor, including Kubernetes and SSH. The coordinator (the machine running senro itself, as opposed to a remote executor target) holds the canonical copy and stages it out to each target. ## The four rules ### 1. `MaxAge` and `MaxSize` are mandatory, with no default `Build()` rejects a persistent workspace that's missing either one, and names which one is missing. An unbounded workspace that outlives a run will fill up a disk silently. The right bound depends on your pipeline, so senro won't guess one for you. | Declaration | `Build()` | | --- | --- | | `Scope(senro.ScopePersistent)` with both `MaxAge` and `MaxSize` | **Accepted** | | `Scope(senro.ScopePersistent)` missing either bound | **Refused**, naming the missing one | | `MaxAge` or `MaxSize` on any other scope | **Refused**: nothing would ever apply it | ### 2. Eviction happens outside every step, never during one senro checks `MaxAge` when a run leases the workspace (claims it for that run's exclusive use, as rule 4 below covers), and checks `MaxSize` when the run releases it. It checks `MaxSize` again at the next lease, against what the previous run recorded. That second check still happens even if a run gets killed. ```mermaid flowchart LR A["Run A leases
(checks MaxAge)"] --> B["Run A releases
(checks MaxSize)"] B --> C["Run B leases
(rechecks MaxSize)"] ``` An eviction clears the whole workspace. Half a dependency tree isn't a useful smaller one. The eviction emits a `ws.evicted` event with the bound and the measurement, so if a workspace keeps starting cold, you can see which number to raise. ### 3. Its content is part of the cache key senro measures the workspace once, before the first step runs, and that digest enters `workspace_digests` for every step that mounts it. An unchanged workspace shares a key across runs, so the action cache hits. A changed one misses, and [`senro cache explain`](/docs/data/cache-keys/) will name `workspace_digests` as what changed. ### 4. One run at a time holds one If a second run wants the same workspace, it's refused immediately, before any of its steps run. The error names the run that holds it. senro doesn't make the second run wait, because the lease covers the whole run, and waiting could mean waiting for someone else's entire pipeline. It also doesn't give each run its own private copy: that's what a `ScopeRun` workspace is for. ## Name it for what is in it A persistent workspace is **machine-global and keyed by name alone**. If two pipelines both declare a workspace called `"cache"`, they share one directory. Name it for what it holds, like `"go-mod-cache"`, not for the role it plays. ## On Kubernetes it can live in the cluster instead By default, the tree stays on the coordinator and is staged into each pod, a full transfer on every attempt. [`k8s.Claim`](/docs/executors/kubernetes/) backs the workspace with a `PersistentVolumeClaim` you already created instead, so the pod mounts it directly and nothing needs to be transferred. The trade-off: - A step that mounts a claimed workspace **can't be `Pure()`**, because the coordinator can't measure a tree it can't reach. - The lease becomes a `coordination.k8s.io` `Lease`, so two coordinators still exclude each other. `ScopePersistent` and `PersistentVolumeClaim` share a word and nothing else. A persistent workspace needs no claim, and no cluster at all. ## Where to go next - **[Workspaces](/docs/data/workspaces/)**: mounting, scopes, snapshots and excludes. - **[Cache keys](/docs/data/cache-keys/)**: what `workspace_digests` covers. - **[Kubernetes](/docs/executors/kubernetes/)**: the claim form in full. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/pyproject # `pyproject`: Python One unit per Python distribution. **Does not compute an affected set**, deliberately: fan out over everything. ```go import "github.com/xavidop/senro/unit/pyproject" verify.Expand("test", pyproject.Packages()). MaxParallel(8). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("uv", "run", "pytest")).WorkDir(u.Dir) }) ``` ## What counts as a unit A directory whose `pyproject.toml` names a distribution (PEP 621's `[project]` or Poetry's `[tool.poetry]`), or which holds a `setup.py` or `setup.cfg`. `Name` is the distribution name. - A **nameless** `pyproject.toml` (a virtual uv root, a ruff-only config) is not a unit. - `[tool.uv.workspace] exclude` is honoured. - A virtual environment directory is not searched. - A `setup.py`-only distribution is named after its directory. - No distribution anywhere is an **error**, not an empty graph. ## Why `Affected` is refused Adding `.Affected(...)` over this graph fails at build time, the same way it does over [`glob`](/docs/monorepo/unit-graphs/glob/). The manifests do carry dependency declarations. In Python, a declaration is just not what makes an import work: - `uv sync` and `pip install -e` make every workspace member importable, so a package can `import acme_core` with nothing in its own `pyproject.toml`, and it works for years undetected. - A src layout on `PYTHONPATH`, a `conftest.py`, an entry-point pytest plugin, and an import inside a function are four more real dependencies with nothing static to read. - `dynamic = ["dependencies"]` says outright that the dependencies are not in the manifest. - uv, Poetry, Hatch and setuptools disagree about where an intra-repo dependency is even written. Any one of those is a missing edge, and a missing edge is a green build for a broken tree. **Fanning out over everything is still worth having.** Twenty services testing in parallel under a `MaxParallel` cap, each cached independently, is most of the win. If your repository genuinely declares and enforces every dependency, [write a graph that knows it](/docs/monorepo/unit-graphs/custom/). ## Where to go next - **[Fan-out](/docs/monorepo/fan-out/)**: `MaxParallel`, `MaxNodes` and per-unit edges. - **[Caching a step](/docs/data/caching/)**: making each unit's step skippable on a re-run. - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the other seven. --- > Source: https://xavidop.github.io/senro/docs/quickstart # Quickstart From nothing to a running, attachable pipeline. [Install](/docs/install/) first. ## 1. Write the pipeline A pipeline is a Go package with a `main` that builds a `senro.Pipeline`, adds a `Workflow` and its `Step`s, and calls `senro.Run`. Put this in `ci/main.go`: ```go // Command ci is a two-step pipeline: test, then build. package main import ( "context" "log" "os" "github.com/xavidop/senro" "github.com/xavidop/senro/attach" "github.com/xavidop/senro/exec" "github.com/xavidop/senro/retry" ) func main() { ctx := context.Background() p := senro.New("ci") verify := p.Workflow("verify") verify.Step("test", exec.Command("go", "test", "./...")) verify.Step("build", exec.Command("go", "build", "./...")). Needs("test"). Retry(3, retry.OnInfra()) // retries a dropped connection, never a failing test // Opens a unix socket a second terminal can attach to while this runs. att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket}) if err != nil { log.Fatal(err) } defer att.Close() // Run builds p first, so a dangling Needs, a duplicate id, or an empty // command surfaces here, before anything executes. if err := senro.Run(ctx, p, senro.WithAttach(att)); err != nil { os.Exit(1) } } ``` Four things are doing the work: - `exec.Command` wraps any command as a step. [Steps](/docs/steps/) has the rest. - `.Needs("test")` makes `build` wait for `test`. [Ordering](/docs/steps/ordering/). - `.Retry(3, retry.OnInfra())` allows up to three attempts, but only when the environment failed the step (a dropped SSH connection, a killed process), never when the command simply exited non-zero. [Retries](/docs/steps/retries/). - `attach.Listen` plus `senro.WithAttach` is the one addition that makes the run observable from another process. Without it, `senro.Run` costs exactly what the engine costs: no attach server, no extra goroutine. [Attach](/docs/attach/). ## 2. Run it ```bash senro run ./ci ``` The CLI builds the package, execs it, and attaches automatically: a terminal UI on a TTY, plain streaming lines otherwise. Or run the binary yourself and attach from a second terminal: ```bash go run ./ci & senro attach ``` ## 3. Watch and steer it Both routes end in the same place: an interactive view of `test` and `build` running. | Key | What it does | |---|---| | `enter` | Focus a step and follow its log | | `r` | Retry the focused step | | `c` | Cancel the run | [The TUI](/docs/attach/tui/) has the full key list and the other renderers; [Control operations](/docs/attach/control-ops/) covers what each key asks the engine to do. ## 4. Reopen the run after it finished Every run's events land on disk, so the process does not need to be alive: ```bash senro attach --run ``` This is the same client over the same protocol, reading recorded events and step logs instead of a live socket. When a run fails, that directory is where the answer is: [Reading a failed run](/docs/run/debugging/) covers every file in `runs//` and where a step's stdout and stderr live. ## Where to go next - [Concepts](/docs/concepts/): pipeline, plan, execution, and the event stream underneath all of this. - [Steps](/docs/steps/): `Env`, `WorkDir`, `Timeout`, `ContinueOnError`, and the two step kinds. - [Handlers](/docs/steps/handlers/): `OnFailure` and `Always` when a step fails anyway. - [Executors](/docs/executors/): the same steps in a container, on a pod, or over SSH. - [Secrets](/docs/secrets/): a credential without ever putting it in argv, an env value, or a log. - [CLI](/docs/cli/): every flag `senro run` and `senro attach` accept. --- > Source: https://xavidop.github.io/senro/docs/run/debugging # Reading a failed run A run that fails ends with one line and a pointer: ``` senro: run failed: step "boom" failed (exit 7); see runs/20260812T152953-84221d6cf4/events.jsonl ``` This page covers what to do next. The fastest fix is to reopen the finished run in the same UI: ```bash senro attach --run 20260812T152953-84221d6cf4 --follow ``` ## The run directory Every run writes one directory: `runs/`, **relative to the directory the pipeline process started in**. `` is a UTC timestamp plus a short random suffix. `senro.WithDir` and `attach.Options.Dir` override the location; `senro.WithRunID` and `attach.Options.RunID` override the ID. ``` runs/20260812T152953-84221d6cf4/ ├── run.json the run's manifest: id, pipeline name, start time, trigger. │ Written before the first event, readable mid-run via │ senro.ReadRunManifest. No parameter values, deliberately ├── events.jsonl the timeline: one JSON object per line, append-only. Safe to │ grep, jq or tail -f. Everything below is derived from it ├── plan.json every node's id, kind, command, workdir, needs and mounts as │ Build() resolved them: the command here is the command that ran ├── logs/ │ ├── greet/1/stdout per step, then per attempt: the step's own output, byte for │ ├── greet/1/stderr byte, after redaction. Compiler errors and stack traces │ ├── boom/1/stdout live here │ └── boom/1/stderr ├── work/ each step's sandbox working directory, kept as the step left │ ├── greet/1/ it. Local executor only. A handler gets its own under its │ └── boom/1/ composite id (work/deploy%2Fon_failure%2Fcollect/1/); it │ shares its parent's workspaces read-only, never this directory ├── ws/ workspaces the run materialized. Only for a run that declared one └── cache/ ├── scratch.json scratch cache restore and save records; [] when none mounted └── .json one Pure() step's full cache key, plus whether it hit. senro cache explain reads this for you ``` **`senro` never deletes a run directory.** It stays until you remove it yourself, so the evidence is still there tomorrow. `senro cache gc` cleans up the *cache root* instead, a separate place, and its `--keep-failed` flag protects a failed run's cached content there for a week. Three things to know about `logs///`: - **`` starts at 1.** A retried step keeps every attempt (`logs/flaky/1/`, `2/`, `3/`), so you can compare the first failure against the last. - **An expanded step's id is percent-encoded** to fit in one path segment: `lint[unit=apps/web]` writes to `logs/lint%5Bunit=apps%2Fweb%5D/1/`. Quote it in the shell. - **Secrets are already redacted.** Every value `senro.WithSecrets` resolved gets replaced before it reaches the file, so these logs are safe to attach to a bug report. See [Secret channels](/docs/secrets/channels/). ## Finding the failing step Read `events.jsonl` from the end. The last line is the verdict, whenever the run got far enough to record one: ```bash tail -1 runs/20260812T152953-84221d6cf4/events.jsonl ``` ```json {"v":1,"seq":15,"ts":"2026-08-12T15:29:53.636086Z","type":"run.finished","run":"20260812T152953-84221d6cf4","payload":{"status":"failed","steps":{"failed":1,"skipped_upstream_failed":1,"succeeded":1},"duration_ns":24948000}} ``` `status` is one of five values, worst first: `cancelled`, `failed`, `partial`, `succeeded_with_recovery`, `succeeded`. See [Step states](/docs/steps/states/) for how steps roll up into it. In this example, one step failed, one succeeded, and one was skipped because its dependency failed. To name the failing step, filter for `step.finished` events whose state is a real failure: ```bash jq -r 'select(.type=="step.finished" and (.payload.state=="failed" or .payload.state=="timed_out" or .payload.state=="panicked")) | "\(.step)\t\(.payload.state)\texit \(.payload.exit_code)"' \ runs/20260812T152953-84221d6cf4/events.jsonl ``` ``` boom failed exit 7 ``` Without `jq`, `grep '"type":"step.finished"' | grep '"state":"failed"'` gets you the same line. For the whole timeline at a glance, use `jq -r '"\(.seq)\t\(.type)\t\(.step // "-")"'`. Three things to know when reading this stream: - **A `step.finished` with no `step.started` never ran.** Its state is `skipped_upstream_failed`, which rolls the run up to `partial` on its own, or `failed` if a step genuinely failed too. - **A `step.finished` payload carries the whole verdict**, including a one-line `error` when there is one. A `Func` step that panics settles as `panicked`, message included: `{"state":"panicked","exit_code":1,"duration_ns":1623708,"error":"panic: assignment to entry in nil map"}`. - **`step.log.appended` only records the offset and length** of bytes written, never the bytes themselves. Read the actual output from `logs/`: `cat runs//logs/boom/1/stderr`. See [The event stream](/docs/run/event-stream/) for the envelope and every event type. ## Which tool, when Reading files by hand is the fallback. Which tool to reach for depends on one question: ```mermaid flowchart TD Q{Is the run still alive?} -- yes --> S["senro shell --step:
stand in the step's workspaces"] Q -- no --> D{Run directory still on this machine?} D -- yes --> A["senro attach --run ID --follow:
replay it in the full UI"] D -- no --> F["senro logs fetch ID:
pull the archived run back, then attach"] A --> W["senro ws pull / ws diff:
the files a step left behind"] ``` ### Replaying it: `senro attach --follow` `--follow` reads the run **straight from disk**, no socket or live process needed, so it works long after the pipeline exited. You get the full UI, showing exactly what a live run would have shown. - **`--follow` requires `--run`**, and **`--run ` resolves `runs/` relative to your current directory.** Run it from where the pipeline ran. - Without `--follow`, `senro attach --run ` looks for a live run with that ID first, and falls back to the recorded directory. Use plain `--run` while a run might still be alive, `--follow` once you know it's over. See [The TUI](/docs/attach/tui/) and [Run and watch](/docs/cli/run/). ### Standing in it: `senro shell` While a run is **still alive**, `senro shell --step build` opens a session inside that step's own workspaces: read-only, at the same paths the step saw, on the step's own executor. Pair it with a breakpoint (`b` in the TUI) to freeze the workspace while you look around. See [The shell](/docs/attach/shell/). Once the process has exited, there's no engine left to create a sandbox for you. Instead, use [`senro ws pull`](/docs/cli/workspaces/) to write those same files out of the content store. `senro ws diff RUN-A RUN-B src` shows you what changed in a workspace between two runs, file by file. ### Reading it after the machine is gone A CI runner is destroyed when its job ends, taking `runs//` with it. [`senro logs fetch `](/docs/cli/workspaces/) brings it back as an ordinary run directory. ## Errors you will actually hit Every message is what `senro` prints, verbatim, with where the fix is documented. | Message | What it means | |---|---| | `plan: step "flaky" retry policy allows 1 attempt(s), want at least 2` | `Retry`'s first argument is the *total* number of attempts, not extra ones. "Run it again once" is `Retry(2, ...)`. See [Retries](/docs/steps/retries/) | | `plan: dependency cycle: a -> b -> a` | Two steps depend on each other, directly or through a chain. The message prints the whole cycle. Nothing ran. See [Ordering](/docs/steps/ordering/) | | `senro: no live senro runs found. Start one with 'senro run ', ...` | Bare `senro attach` found no live run. Either it already finished (use `--run --follow`), or the pipeline never called `attach.Listen`. Exit code `2`. If several runs are listed instead, pick one with `--pid` or `--run` | | `engine: step "publish" puts the value of secret "RegistryToken" in command argument 3` | A credential would have reached a place senro can't redact, so the run was refused before it started. There's no run directory to inspect. The same refusal applies to an env value, `WorkDir`, `Inputs`, `Outputs`, or a mount's name. See [Secrets](/docs/secrets/) | | `engine: step "install" needs secret "NPMToken", which the struct passed to senro.WithSecrets does not provide (resolved: RegistryToken)` | `SecretEnv`'s **second** argument is a field name on your config struct, not the `source` tag. The message lists which fields did resolve | | `1 cgo-dependent package(s) in .` | From `senro func check`. Whether this actually breaks anything depends on where your `Func` steps run: a step on the coordinator, or on a target with the same platform, is unaffected. See [A Func step off the coordinator](/docs/executors/func-remote/) | | `senro: --ui=tui requires a terminal, but stdout is not a TTY.` | You asked for the TUI where there's no terminal. senro never falls back to something else silently, because escape sequences in a CI log would look like a run that worked when it didn't. See [CLI](/docs/cli/) | | `senro run: no Go toolchain found on PATH` | `senro run` compiles your package first. Install Go, or build the binary yourself and run `./pipeline --tui`. A package that fails to compile stops the same way. Both exit `2`, not `1`, because nothing ran, so there's no run to have failed | | `senro run: unknown flag "--help"` | Subcommands don't take the top-level `--help`. `senro help`, `-h`, and `--help` print the synopsis and exit `0`; anything else is exit `2`. There's no `senro version` | | `dockerd: no container runtime socket found` | The container executor needs a compatible daemon running. The message lists every socket it tried, and how to point at one with `DOCKER_HOST`. See [Containers](/docs/executors/containers/) | ### A cache miss you did not expect `senro cache explain --run measure` compares the step's current cache key against the last recorded one and names what changed. The first `MISS` for a step is expected, and it says so. `workspace_digests` moving tells you a workspace changed, but not what changed in it; `senro ws diff` answers that. See [Cache and verify](/docs/cli/cache/) and [Cache keys](/docs/data/cache-keys/). ## Getting the same detail from Go If you call `senro.Run` directly instead of using the CLI, the error it returns carries all of this: ```go var runErr *senro.RunError if errors.As(err, &runErr) { log.Printf("%s: %d step(s) to blame, evidence in %s", runErr.Status, len(runErr.Steps), runErr.Dir) } ``` `runErr.Steps` names up to three of them, each with `ID`, `State` and `ExitCode`. See [Run options and outcomes](/docs/run/options/) for the full field list. ## Where to go next - **[Step states](/docs/steps/states/)**: the ten step states and how a run's status rolls up. - **[The event stream](/docs/run/event-stream/)**: the envelope, and folding a stream in your own code. - **[CLI](/docs/cli/)**: every command and flag, and the exit-code contract. --- > Source: https://xavidop.github.io/senro/docs/cli/rerun # `senro rerun` Re-execute the plan a previous run recorded. ``` senro rerun [--run RUN] [--step STEP] [--regenerate] [--dir DIR] [--cache-dir DIR] [--local-class CLASS] ``` Reach for this when you need to reproduce a past run exactly: replaying a failure to debug it, or re-executing just one step and everything that depends on it, rather than starting a fresh run from source. `senro rerun` reads `/plan.json`, not your pipeline package. That's intentional: rebuilding the pipeline would re-resolve its definition, which might have changed since the original run, giving you a different plan and a different run. Reading the recorded plan instead means a re-run repeats the original run rather than re-discovering a new one. It also means no Go toolchain or pipeline source is needed. Steps whose inputs haven't changed are served from the action cache, so an unchanged re-run is mostly cache hits. ## Flags | Flag | Meaning | | --- | --- | | `--run RUN` | The run to re-execute. Defaults to the most recent one here. | | `--step STEP` | Re-execute this step, what it needs, and everything below it. | | `--regenerate` | Ask generators for a fresh subgraph instead of replaying the recorded one. | | `--dir DIR` | Where to write this run. Defaults to a new run directory. | | `--cache-dir DIR` | The storage root. Defaults to `$SENRO_CACHE_DIR`. | | `--local-class CLASS` | Mirrors `senro.WithLocalClass`. | Exit codes: `0` if the re-run succeeded, `1` if it failed, `2` for a usage error. ## `--step` includes what the step needs ``` senro rerun --step deploy/apply-west ``` This runs `deploy/apply-west`, along with everything it needs (directly or transitively) and everything that depends on it. Any branch unrelated to it is skipped. Including its dependencies is deliberate. A step can't run without its inputs, so if you only selected the step and its dependents, the dependencies would be marked skipped, which would skip the dependents too. The one step you actually asked for would end up being the only thing that didn't run. Including the dependencies costs little, since the unchanged ones are just cache hits. The dependency graph comes from the recorded plan, so "everything below it" means exactly what it meant in the run you're repeating. ## `--regenerate` is a separate verb, deliberately By default, a [generator](/docs/monorepo/generators/) replays the subgraph it recorded, rather than being asked to generate again. This is what lets a re-run actually reproduce the original run: the generator might query an API that now answers differently, and replaying the recording guarantees you get the graph that actually ran the first time. `--regenerate` asks for a fresh subgraph instead: ``` senro rerun --regenerate ``` Use `--regenerate` when things have genuinely changed and the recorded graph no longer describes reality, like a fleet that no longer exists. It's a separate flag, not the default, because silently re-deriving the graph during what looks like a retry would be confusing: the run would quietly do different work than the one it claims to repeat. A regenerated fragment gets recorded too, so the next plain `senro rerun` replays that new fragment. ### Go generators cannot be regenerated from a recorded plan A generator written with `senro.Generate` is a Go closure. It lives in your pipeline package and never makes it into `plan.json`, because a plan has to be serializable. So: - **Without `--regenerate`**, a Go generator replays from its cache entry, and everything works as normal. - **With `--regenerate`**, there's no closure to call. The run fails and names the step, rather than quietly replaying it and pretending it regenerated. A generator declared with `GenerateFromJSON` has no such limitation. Its fragment is just a file the step writes, so re-running the step produces a fresh one. ## Where to go next - **[Generated subgraphs](/docs/monorepo/generators/)**: what is being replayed, and why recording it is what lets a generator be nondeterministic. - **[cache & verify](/docs/cli/cache/)**: why a step hit or missed, and re-checking a `Pure()` claim. --- > Source: https://xavidop.github.io/senro/docs/steps/retries # Retries A command exiting non-zero is not the same as the infrastructure that ran it breaking. senro makes you say which one you want to retry. ```go verify.Step("test", exec.Command("go", "test", "./...")). Retry(3, retry.OnInfra()) // only a dropped connection retries; a failing test never does ``` `Retry(maxAttempts int, p retry.Predicate)` retries while `p` matches. ## `maxAttempts` is total attempts **`Retry(1, ...)` is refused at `Build()`**, because one attempt is no retry at all: ``` plan: step "flaky" retry policy allows 1 attempt(s), want at least 2 ``` "Retry once on failure" is `Retry(2, ...)`. This is the refusal that surprises people most. ## The predicates ```go retry.OnInfra() // the step's process failed to start or run at all retry.OnExitCode(75) // specific exit codes only retry.OnLogMatch(`connection refused`) // last resort, matching on log text retry.Any(retry.OnInfra(), retry.OnExitCode(75)) retry.Named("http-status", HTTPStatus{...}) // your own Go function; see below ``` - **`retry.OnInfra()`** matches infrastructure failures only: an SSH reset, an image that will not pull, a pod evicted out from under the step. The substrate failed the step, not the other way round. A non-zero exit is never matched here. - **`retry.OnExitCode(codes ...int)`** filters `0` out of its list rather than refusing it. Exit `0` is success, so there is nothing to retry: `OnExitCode(0)` builds cleanly and never matches, and `OnExitCode(0, 75)` behaves exactly like `OnExitCode(75)`. - **`retry.OnLogMatch(pattern)`** returns `(Predicate, error)`; the pattern compiles at construction, so a broken regexp fails when the pipeline is built, not on a failing host. Treat it as a last resort: the retry silently stops firing the day somebody rewords the log message. - **`retry.Any(preds ...Predicate)`** matches if any of its predicates does. - **`retry.Named(name, params)`** is a Go function of your own, registered under a name so a plan can carry it. See [Deciding in Go](#deciding-in-go-retryregisterpredicate). ## Deciding in Go: `retry.RegisterPredicate` When the four predicates above cannot express the rule, write the decision as a Go function and give it a name. Registering it is what lets a plan carry it: what gets recorded is the **name and the arguments**, not the closure. It is the same shape [`senro.RegisterFunc`](/docs/steps/functions/) has. Register once, from an `init`; name it at the call site. ```go type HTTPStatus struct { Codes []int `json:"codes"` } func init() { retry.RegisterPredicate("http-status", func(p HTTPStatus, a retry.Attempt) bool { for _, c := range p.Codes { if strings.Contains(a.LogTail, strconv.Itoa(c)) { return true } } return false }) } // in the pipeline verify.Step("publish", exec.Command("./publish.sh")). Retry(3, retry.Named("http-status", HTTPStatus{Codes: []int{502, 503}})) ``` The plan records `func:http-status:{"codes":[502,503]}`, and the engine looks the name up to reconstruct the predicate before the first attempt. ### What your function is handed ```go type Attempt struct { Number int // 1 on the first try ExitCode int // the workload's verdict Err error // set when the attempt failed to run at all LogTail string // the tail of this attempt's output } ``` So a rule that needs the attempt number, which none of the built-in predicates can express: ```go func init() { retry.RegisterPredicate("infra-then-give-up", func(p struct{}, a retry.Attempt) bool { // Two goes at a broken substrate, then stop: past that it is not // a hiccup, and a third attempt only delays the report. return a.Number <= 2 && errors.Is(a.Err, executor.ErrInfra) }) } verify.Step("publish", exec.Command("./publish.sh")). Retry(5, retry.Named("infra-then-give-up", nil)) ``` Pass `nil` for a predicate that takes no parameters. It composes with everything else, because it is recordable like everything else: ```go retry.Any(retry.OnInfra(), retry.Named("http-status", HTTPStatus{Codes: []int{503}})) ``` ### The rules | | | |---|---| | **The name is API** | It is what `plan.json` records and what the engine looks up. Renaming it breaks any recorded plan that still names it, exactly as renaming a command would. | | **Register from `init`** | Registering the same name twice panics, as does an empty name or one containing `:`. | | **Parameters must be JSON-serializable** | And are decoded strictly. A recorded field your struct does not have means **no match**, never a blind retry. | | **A name nothing registered is a build error** | `Build()` names it, and lists what this binary does have. A typo never becomes a policy that silently never fires. | | **Do not block, do not have side effects** | It is asked once per failed attempt, inside the retry loop. | ### `retry.Func`, and why it cannot be built `retry.Func(f func(retry.Attempt) bool)` wraps a bare function with no name attached. A step using one is refused at `Build()`: ``` senro: step "publish" retry predicate has no serialized form and cannot be built into a plan, which is what retry.Func produces; use retry.OnInfra, retry.OnExitCode, retry.OnLogMatch or retry.Any instead ``` A `Plan` is JSON: every other predicate carries a written-down form (`"infra"`, `"exit_code:75,111"`, `"func:http-status:{...}"`) that survives being saved to disk and reconstructed by the engine in another process. A bare closure has none, so senro would have to either drop it silently and retry on every failure, or refuse. **`retry.RegisterPredicate` is the answer to that**: the same Go code, with a name a plan can record. `retry.Func` remains for a `retry.Policy` you hold and apply directly, in code that never builds a plan. ## Backoff with `RetryPolicy` `RetryPolicy(policy retry.Policy)` is the same thing with explicit backoff: ```go verify.Step("publish", exec.Command("./publish.sh")). RetryPolicy(retry.Policy{ MaxAttempts: 4, On: retry.OnInfra(), Backoff: retry.Backoff{Base: time.Second, Max: 30 * time.Second, Factor: 2}, }) ``` `retry.Backoff{Base, Max, Factor}` is exponential with jitter: 1s, 2s, 4s, 8s, capped at 30s, each wait nudged by a random amount. **Jitter is what keeps every retrying step from hammering the same failure point at the same moment.** ## What a retry does to the rest of the step - **A step that failed and then passed settles as `recovered`, not `succeeded`.** Collapsing the two is how flaky infrastructure stays invisible for months. See [Step states](/docs/steps/states/). - **`Timeout` bounds one attempt**, so a step with three attempts can take three timeouts' worth of wall clock. See [Env, dir & timeout](/docs/steps/settings/). - **`OnFailure` handlers run once retries are exhausted**, not per attempt. See [Failure handlers](/docs/steps/handlers/). - **A panicking `senro.Func` step is not retried.** It settles as `panicked`. - **A handler cannot declare a `Retry` of its own**; `Build()` refuses it. ## Knowing which attempt you are on - A `senro.Func` step reads `ctx.Attempt()`, which is `1` on the first try. That is what an idempotency key needs to know before retrying against a remote API. See [Func steps](/docs/steps/functions/). - A handler reads `SENRO_FAILURE_ATTEMPT`, the attempt the step actually reached, so it can find that attempt's log. ## Retrying by hand, while the run is going `Retry` is a **policy written into the plan**: senro decides on its own, using the predicate, with nobody watching. Separately, if you are [attached](/docs/attach/) to a run, you can retry a step **yourself**: focus it in the TUI and press `r`. That works on any settled step, whether or not it declared a `Retry`, and it is how you rerun something after fixing the machine under it. The two never interact. A step with `Retry(3, retry.OnInfra())` that failed a test exhausted nothing (the predicate never matched), and pressing `r` still runs it again. Refusals are per-mechanism too: `r` is refused while a step is still running, or if it was never reached. See [Control operations](/docs/attach/control-ops/) for `step.retry` on the wire. ## Where to go next - **[Failure handlers](/docs/steps/handlers/)**: what runs after the last attempt fails. - **[Step states](/docs/steps/states/)**: `recovered`, `failed`, `timed_out`, `panicked`. - **[Reading a failed run](/docs/run/debugging/)**: finding a given attempt's logs. --- > Source: https://xavidop.github.io/senro/docs/triggers/server # Run it as a server Your pipeline binary can be the webhook endpoint. One line changes: ```go ev, err := trigger.FromRequest(r, trigger.Secret(hookSecret)) // instead of LoadEvent(path) ``` No event file, no dispatcher, no second binary. Everything after that line is the same pipeline and the same triggers. Runnable: [`examples/server`](https://github.com/xavidop/senro/tree/main/examples/server). ## The handler ```go func (s *server) webhook(w http.ResponseWriter, r *http.Request) { ev, err := trigger.FromRequest(r, trigger.Secret(s.secret)) if err != nil { httpError(w, err) return } w.WriteHeader(http.StatusAccepted) // Not r.Context(): it is cancelled when the response is written. go senro.Run(context.Background(), pipeline(), senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")))) } ``` `FromRequest` does three things: works out which source sent the delivery, verifies its signature, and parses the body into the same [`trigger.Event`](/docs/triggers/) a file would have produced. Two things it does not do: manage [concurrency](#one-run-at-a-time), and wait for the run. > **Reply before the run finishes.** A webhook sender times out in seconds; a pipeline takes > minutes. Reply `202` and run in the background. > **Never pass `r.Context()` to `senro.Run`.** It is cancelled the moment the response is written, > which kills the pipeline the instant it starts. ## Verifying each provider The four sources authenticate differently. `trigger.Secret` covers the difference; you pass the same secret you configured on the webhook. > **Verification is not optional.** `FromRequest` with neither `Secret` nor `Unverified` is an > error, because an endpoint that verifies nothing runs your pipeline for anybody who can reach > it. Passing both is an error too, not a precedence rule. ### GitHub ```go trigger.FromRequest(r, trigger.Secret(os.Getenv("SENRO_HOOK_SECRET"))) ``` HMAC-SHA256 over the raw body, in `X-Hub-Signature-256`. Set the same secret in the repository's webhook settings. ### GitLab ```go trigger.FromRequest(r, trigger.Secret(os.Getenv("SENRO_HOOK_SECRET"))) ``` GitLab sends the secret **itself** in `X-Gitlab-Token`, unsigned. That is only as good as the transport, so serve this over HTTPS. ### Gitea ```go trigger.FromRequest(r, trigger.Secret(os.Getenv("SENRO_HOOK_SECRET"))) ``` Same HMAC as GitHub, in `X-Gitea-Signature`. Newer builds also send `X-Hub-Signature-256`; either is accepted. ### Bitbucket ```go trigger.FromRequest(r, trigger.Unverified()) ``` **Bitbucket Cloud signs nothing and sends no token.** There is nothing for a secret to check, so senro says so instead of pretending. Restrict the endpoint by network: Bitbucket publishes its egress ranges. `Unverified()` is also the answer when a proxy in front of you already verified the delivery. ## Status codes ```go func httpError(w http.ResponseWriter, err error) { switch { case errors.Is(err, trigger.ErrUnsigned): http.Error(w, "unauthorized\n", http.StatusUnauthorized) case errors.Is(err, trigger.ErrUnknownSource): http.Error(w, "unrecognised delivery\n", http.StatusBadRequest) default: http.Error(w, "could not read the delivery\n", http.StatusBadRequest) } } ``` | Error | Means | Answer | |---|---|---| | `trigger.ErrUnsigned` | Signature absent, malformed **or** wrong | `401` | | `trigger.ErrUnknownSource` | No header naming a source senro knows | `400` | | anything else | Body unreadable, too large, unparseable | `400` | `ErrUnsigned` covers all three failure modes on purpose: which one it was is not something to confirm to whoever sent it. ## One run at a time senro has no opinion here. Pick one and write it: ```go func (s *server) take() bool { s.mu.Lock() defer s.mu.Unlock() if s.running { return false } s.running = true return true } ``` ```go if !s.take() { http.Error(w, "a run is already in progress\n", http.StatusConflict) return } ``` `409` says "busy, nothing queued, retrying immediately will not help". That is reject-don't-queue, the same choice [contrib/dispatcher](#two-processes-instead) makes. A queue has a backlog, an eviction policy and a memory: worth having on purpose, not by accident. ## A custom provider A [source of your own](/docs/triggers/custom/) has no header `FromRequest` could recognise, so say what to parse it as: ```go ev, err := trigger.FromRequest(r, trigger.As("deploy-bus", r.Header.Get("X-Bus-Event")), trigger.WithProviders(deploybus.Provider{}), trigger.Unverified()) // senro knows no signature scheme for your bus ``` Verify the delivery yourself before calling, since `Unverified()` means senro checks nothing. `As` also pins a built-in, for a proxy that rewrites headers or an endpoint that only ever receives one source. ## Without `FromRequest` If you already hold the pieces separately: ```go func Parse(provider, event string, payload []byte, providers ...Provider) (*Event, error) ``` ```go ev, err := trigger.Parse("github", r.Header.Get("X-GitHub-Event"), body) ``` No signature check, no header sniffing, no body read: those are yours. It is the primitive `FromRequest` is built on. ## Two processes instead Keep the pipeline a one-shot binary and put a receiver in front of it. senro ships one at [`contrib/dispatcher`](https://github.com/xavidop/senro/tree/main/contrib/dispatcher), which adds a per-group lock (a file lock, or a Kubernetes `Lease` with `-namespace`): ```sh dispatcher -addr :8080 -secret-file /etc/senro/webhook-secret \ -pipeline ./ci -group ci-main ``` **A raw webhook body is not an event file.** No GitHub, GitLab, Bitbucket or Gitea body says which event it is, which is why [the file format](/docs/triggers/events/#the-format) is an envelope. Writing the body verbatim gets `the event names no provider` from the pipeline, at the far end of an exec where nobody is looking. Two helpers build one, and are what the dispatcher uses: ```go provider, event, ok := trigger.SourceOf(r.Header) // from the delivery's headers file, err := trigger.Envelope(provider, event, body) ``` ### Which shape to pick | | A server | A dispatcher | |---|---|---| | Processes | One | Two | | Concurrency | Yours to write | A lock, `-cancel-in-progress` | | A crash takes down | Endpoint **and** run | The run only | | Replicas exclude each other | You arrange it | `-namespace`, over a `Lease` | | Pipelines per endpoint | One | One dispatcher each | Start with a server for one pipeline and one thing to deploy. Move to a dispatcher when a crashed run should not take the endpoint with it. ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers and the three outcomes. - **[The event file](/docs/triggers/events/)**: the envelope, for the two-process shape. - **[Write your own](/docs/triggers/custom/)**: a source that is not one of the four. - **[GitHub Checks](/docs/notifications/github-checks/)**: reporting the result back on the commit. --- > Source: https://xavidop.github.io/senro/docs/run/options # `senro.Run`: options and outcomes Every senro pipeline is a Go program that calls `senro.Run` (see [Quickstart](/docs/quickstart/) for the basic pattern). This page is the reference for that call: every option `Run` accepts, and how to read what it returns when a run fails. ## `Run` and `RunPlan` ```go func Run(ctx context.Context, p *Pipeline, opts ...Option) error func RunPlan(ctx context.Context, p *Plan, opts ...Option) error ``` `Run` takes your `Pipeline`, builds it, and runs it. If the pipeline doesn't validate, you get the build error back. Use `Run` for the ordinary case. `RunPlan` runs a `Plan` you already built, without building it again. Use it when you need to inspect the plan first and guarantee that exact plan is what runs, not a fresh build of whatever the pipeline looks like by the time it executes. > **A plan is a snapshot.** Once you `Build()` a pipeline, the resulting `Plan` can't be changed by > anything you do to the pipeline afterward. That's why `RunPlan` is useful: it guarantees the plan > that runs is the one you inspected. ## Run options Both `Run` and `RunPlan` take the same options: | Option | What it does | | --- | --- | | `WithAttach(att)` | Sends every event to `att`'s socket, and uses its run directory and run ID (unless you override them), so the attach server and the engine agree on one run. | | `WithSink(s)` | Adds your own observer, which gets every event the run produces, in order. You can pass this more than once, and it works alongside `WithAttach`. This is how a program watches a run without a terminal; it's also what the notifiers use. See [Notifications](/docs/notifications/). | | `WithDir(dir)` | Pins the run's directory on disk. If you don't set it, the run uses `att.Dir()` under `WithAttach`, or generates a new `runs/` directory. | | `WithRunID(id)` | Same idea, for the run's ID. | | `WithSecrets(cfg)` | Passes the resolved secrets config from [mamori](https://github.com/xavidop/mamori)'s `Load`, so steps can read credentials with `SecretEnv`. See [Secrets](/docs/secrets/) for what this protects and what it doesn't. | | `WithParams(p)` | The run's parameters: a flat `map[string]string`, like `{"branch": "main"}`. Never recorded in an event or a cache key. `senro.When`'s `Branch` and `ParamIs` conditions read these. See [Conditions](/docs/steps/conditions/). | | `WithCacheDir(dir)` | Where the content store, action cache, and scratch cache live. Defaults to `$SENRO_CACHE_DIR`, or your platform's standard cache directory. Unlike `WithDir`, this is shared across every run on the machine, not per run. | | `WithLocalClass(class)` | Overrides the cache key senro uses to identify "this machine" (normally something like `"local/darwin/arm64"`). Most pipelines don't need this. | | `WithTrigger(ev, ts...)` | Gates the run on the event that started it. If nothing matches, `Run` returns `trigger.ErrNoMatch` instead of running. See [Triggers](/docs/triggers/). | | `WithRemoteCache(rc)` | Backs the content store and action cache with an S3-compatible bucket or an OCI registry, so runs on different machines can share a cache. `senro.RemoteCacheFromEnv()` reads the same settings from `SENRO_REMOTE_CACHE`. See [Shared cache](/docs/data/shared-cache/). | | `WithFuncBuild(pkg)` | Names the Go package this program was built from, so a `Func` step can be cross-compiled to run on a different platform. `senro run` sets this for you automatically. See [A Func step off the coordinator](/docs/executors/func-remote/). | | `WithTraceContext(tp, ts)` | Continues an inbound W3C trace, so the run's events show up under the CI job or webhook delivery that started it. See [Writing a trace exporter](/docs/extend/exporter/). | | `WithAnalyzer(a, opts...)` | Adds your own analyzer, which gets a chance to look at every step that fails. It can only propose a fix; nothing happens without a human approving it. See [Writing an analyzer](/docs/analyzers/custom/). | Calling `Run` with no options costs nothing extra: no attach server, no background work. A run directory and ID still get created either way, so even an unattached run leaves a real, inspectable record on disk. ## Handling the outcome ```go if err := senro.Run(ctx, p, senro.WithAttach(att)); err != nil { var runErr *senro.RunError if errors.As(err, &runErr) { // A real run outcome: failed, partial, or cancelled. log.Printf("run ended: %s", runErr.Status) for _, s := range runErr.Steps { log.Printf(" %s %s (exit %d)", s.ID, s.State, s.ExitCode) } if runErr.StepsOmitted > 0 { log.Printf(" and %d more", runErr.StepsOmitted) } log.Printf(" evidence: %s/events.jsonl", runErr.Dir) } else { // An engine-level failure: an invalid plan, a disk write failure. log.Fatal(err) } os.Exit(1) } ``` `Run` returns a single `error`. To find out *why* a run failed, use `errors.As` to recover a `*senro.RunError`. It carries everything a failure report needs, without re-reading the event log: - **`Status`**: the run's overall outcome, one of `succeeded`, `succeeded_with_recovery`, `partial`, `failed`, or `cancelled`. See [Step states](/docs/steps/states/) for how steps roll up into this. - **`Dir`**: the run directory, containing `events.jsonl`, `plan.json`, and every step's logs. See [Reading a failed run](/docs/run/debugging/). - **`Steps`**: up to three steps behind the failure, each with `ID`, `State`, and `ExitCode`. `ExitCode` only means something when `State` is `failed`; otherwise it's `0`. - **`StepsOmitted`**: how many more failing steps exist beyond those three. **`succeeded` and `succeeded_with_recovery` never produce a `RunError`.** Both return `nil`. A run that failed a step but passed on retry counts as a success, not an error. ## Where to go next - **[Quickstart](/docs/quickstart/)**: the same `senro.Run` call, in context. - **[Attach](/docs/attach/)**: what `attach.Listen`'s options configure. - **[CLI](/docs/cli/)**: the wrapper `senro run` puts around this exact pattern. - **[Steps](/docs/steps/)**: building the `Pipeline` this page assumes exists. - **[Secrets](/docs/secrets/)**: `senro.WithSecrets`, the other option most pipelines reach for. - **[Notifications](/docs/notifications/)**: the webhook and Slack sinks built on `WithSink`. - **[Triggers](/docs/triggers/)**: `senro.WithTrigger`, for a pipeline that decides for itself whether an event is its business. --- > Source: https://xavidop.github.io/senro/docs/monorepo/affected # Running only what changed: `Affected` `.Affected(src)` on an `ExpandBuilder` narrows the fan-out to the units a change reaches: the ones owning a changed file, plus everything depending on them at any depth. ```go import ( "github.com/xavidop/senro/change" "github.com/xavidop/senro/unit/gowork" ) verify.Expand("test", gowork.Modules()). Affected(change.FromTrigger(ev)). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) ``` It needs a graph that knows about dependencies. Six do: `gowork`, `cargo`, `jswork`, `maven`, `gradle`, and `bazel.Query()`. Three don't: `glob`, `pyproject`, and `bazel.Packages()`. Using `Affected` with one of those three is **rejected at build time**, rather than silently running everything. See [The shipped unit graphs](/docs/monorepo/unit-graphs/). ## The rule A unit runs if the change touched a file it owns, **or** if it depends, at any depth, on a unit that did. In the [example workspace](https://github.com/xavidop/senro/tree/main/examples/monorepo): ```mermaid flowchart LR api["services/api"] -->|imports| config["libs/config"] -->|imports| log["libs/log"] worker["services/worker"] -->|imports| log ``` `services/api` never imports `libs/log` directly, but a change to `libs/log` still runs it, through `libs/config`. That transitive hop is the whole point of the feature, and exactly where a one-level implementation would quietly get it wrong. ```sh go run ./examples/monorepo --trigger-event examples/monorepo/events/push-api.json # 1 step: services/api. Nothing imports it. go run ./examples/monorepo --trigger-event examples/monorepo/events/push-config.json # 2 steps: libs/config and services/api. worker does not import config. go run ./examples/monorepo --trigger-event examples/monorepo/events/push-log.json # 4 steps: everything. ``` ## What changed: the `change` package `github.com/xavidop/senro/change` answers "what did this run change". Four sources: | Source | What it reports | |---|---| | `change.FromTrigger(ev)` | What the event that started the run recorded. See below. | | `change.Paths("a/x.go", ...)` | A literal list, for callers with their own idea of what changed, and for tests. | | `change.Everything()` | Every unit runs. | | `change.Ignoring(src, "docs/**")` | `src`, with matching paths dropped. | `change.Paths()` with no arguments means **"nothing changed,"** not "everything." Use `change.Everything()` for that. They're different answers. ### `FromTrigger` consumes the base, it does not invent one A [trigger](/docs/triggers/) has already decided what changed. `FromTrigger` just reads what it recorded, in this order: 1. **No event at all** means everything runs. This is the local loop: running `./pipeline` with no `--trigger-event` builds everything. If a dispatcher forgets the flag, it over-runs visibly instead of silently skipping work. 2. **Mode `all`** means everything runs: a default-branch push, a tag, or a scheduled run covers the whole repository by definition. 3. **A base with both ends set** is `git diff `. 4. Otherwise **the event's own changed-file list**, if it carried one. GitHub sends one on a push to an existing ref and none on a pull request. 5. Otherwise **everything**. senro never computes a merge base, resolves a ref, or picks a "since" on its own. A guessed base would be a base nobody actually declared. A few details worth knowing: - **The base wins over the event's file list.** A GitHub push payload truncates its `commits` array at twenty, so its file list under-counts a large push. Two commit ids, by contrast, are exact. - **The diff is two-dot**, not three-dot. Three-dot needs a merge base, and a shallow CI clone often can't compute one. For a pull request whose base branch has moved on, two-dot reports that drift as changed too, which can mean more units than strictly necessary, but never fewer. - **Renames are turned off**, so a moved file is reported at both its old and new path. - **A base commit not in the clone is a loud error**, not a fallback: ``` change: git diff --name-only --no-renames -z -- in .: exit status 128: fatal: bad object ; if the base commit is not in the clone, deepen the checkout (fetch-depth: 0) so the diff has something to compare against ``` This is the expected result of a shallow checkout. An unchecked git failure would eventually be read as "nothing changed," reporting a green run without anything having compiled. ## Where it deliberately runs too much A wrong affected set is worse than no affected set at all. An extra unit just costs CI minutes, but skipping a unit that a change actually broke reports a green build for a broken tree. So whenever it's unclear, senro runs more, not less: - **A file no unit owns** affects every unit: a `Makefile`, a CI workflow or a linter config above every module can change what all of them build. - **An uncompiled file at a module's root** affects every unit of that module: `go.mod`, `go.sum`, a `.golangci.yml`, a `Dockerfile`. A `.go` file at the same level is compiled into the root package and attributed to it alone. - **`go.work` and `go.work.sum`** affect every unit; they decide which module path resolves where. - **Anything else** belongs to the nearest unit at or above its directory, without crossing into another module. If a file is inside a module that has no unit above it, it belongs to every unit in that module. - **A deleted file** is resolved from its path alone: nothing checks the working tree. If a whole package is deleted, it's owned by nothing, so the answer is everything. That's the only answer that can't accidentally skip the importer whose build just broke. - **A change source that cannot tell what changed** says everything, never nothing. ### `Ignoring`, and the one way to get this wrong `change.Ignoring` is the one thing here that deliberately runs *less*, and that's your call, not senro's: ```go Affected(change.Ignoring(change.FromTrigger(ev), "docs/**", "*.md")) ``` A typo fix in `docs/` can't break a Go build. But a pattern that also matches something that *can* change a build, since `*.yml` catches a linter config as well as docs front matter, can turn a broken build green. Write the narrowest pattern that does the job. An `Everything()` set passes through `Ignoring` untouched, since filtering "build everything" would quietly turn into "build nothing". ## A plan-time filter, not a run-time skip Unaffected children are **not in the plan** at all. They're not skipped steps, they don't show up in the UI, nothing settles for them. That's different from [`When`](/docs/steps/conditions/), which prunes a node that the plan does contain. - Two runs of the same commit against the same base produce the same plan and digest, and a re-run reconstitutes exactly the same children. - An empty affected set materializes no children: the group is declared, `plan.expansion_skipped` is emitted, and the run is `succeeded`, same as a `glob` that matched nothing. - **`MaxNodes` is still checked against the whole graph**, not the narrowed set. ## What is not here - **Anything that guesses.** senro won't resolve a ref, compute a merge base, or shell out to `git` to figure out what a run is probably about. It uses the base the event recorded, or it builds everything. ## Where to go next - **[The shipped unit graphs](/docs/monorepo/unit-graphs/)**: which graphs answer this, and why the others decline. - **[Triggers](/docs/triggers/)**: where the mode and the base come from. - **[Trigger events](/docs/triggers/events/)**: the event file and the per-provider traps. - **[Reading a failed run](/docs/run/debugging/)**: `run.json` records the mode and base consumed. - **[Implement a unit graph](/docs/monorepo/unit-graphs/custom/)**: teach senro a layout no shipped graph fits. --- > Source: https://xavidop.github.io/senro/docs/triggers/manual # Schedule & manual triggers `provider: "senro"` is the source-neutral shape, for the two invocations with no webhook behind them: a nightly run, and somebody pressing a button. It is also the shape to reach for when you have the facts but not a webhook body: a cron job, an internal tool, a script that already knows the branch and the changed files. ## A scheduled run ```json {"provider": "senro", "event": "schedule", "payload": {"schedule": "0 3 * * *", "params": {"suite": "full"}}} ``` ```go senro.WithTrigger(ev, trigger.OnSchedule("0 3 * * *", trigger.Params{"suite": "full"}), ) ``` ``` 0 3 * * * cd /srv/app && ./ci --trigger-event /etc/senro/nightly.json ``` **senro grows no scheduler.** Something outside it still starts the binary at 03:00: cron, systemd timers, a Kubernetes `CronJob`, GitHub Actions' own `schedule:`. `OnSchedule` only matches the event that says "this is the 03:00 run". The cron string is compared to the event's own **as text**, with whitespace normalised. That is what lets two crontab lines pointing at one binary select different work: ``` 0 3 * * * ./ci --trigger-event nightly.json # matches OnSchedule("0 3 * * *") 0 * * * * ./ci --trigger-event hourly.json # matches OnSchedule("0 * * * *") ``` senro does not parse cron, so `0 3 * * *` and `0 3 * * 0-6` are **not** equal even though a scheduler would fire them alike. Write the same string in both places. ## A manual run ```json {"provider": "senro", "event": "manual", "payload": {"ref": "refs/heads/main", "params": {"reason": "rebuild"}}} ``` ```go senro.WithTrigger(ev, trigger.OnManual(), ) ``` Anything in `params` becomes a run parameter, so a condition can read it: ```go deploy := p.Workflow("deploy", senro.When(senro.ParamIs("reason", "rebuild"))) ``` See [Conditions](/docs/steps/conditions/). ## The payload fields | Field | What it is | |---|---| | `ref` | `refs/heads/main`, `refs/tags/v1.2.3`. The kind is read from it. | | `branch` | Derived from `ref` when you leave it out. | | `tag` | Same, for a tag ref. | | `repo` | `acme/app`. | | `default_branch` | What decides whether a push is mode `all` or `affected`. | | `schedule` | The cron string `OnSchedule` compares against. | | `files` | The changed-file list `Paths` filters on. | | `params` | Run parameters this event contributes. | **A field the shape does not have is an error.** `branches` where you meant `branch` is a message, not a filter that silently matched nothing. ## Supplying a file list yourself Every other source either carries a changed-file list or does not, and you cannot change that. The neutral shape is where you supply one you worked out yourself: ```sh FILES=$(git diff --name-only "$BASE".."$HEAD" | jq -R . | jq -sc .) jq -n --arg ref "refs/heads/$BRANCH" --argjson files "$FILES" \ '{provider:"senro", event:"manual", payload:{ref:$ref, files:$files}}' > event.json ``` `trigger.Paths("services/**")` now works against it, and so does an [affected set](/docs/monorepo/affected/). > An empty `files` list means "nothing changed", which is a real answer. Leaving `files` out > entirely means "this event does not say", and `Paths` against it is an error. The two are not > the same. ## Where to go next - **[Triggers](/docs/triggers/)**: the matchers, and what a match carries into the run. - **[The event file](/docs/triggers/events/)**: the envelope every source shares. - **[Conditions](/docs/steps/conditions/)**: reading the `params` a trigger contributes. --- > Source: https://xavidop.github.io/senro/docs/data/scratch # Scratch caches A `ScratchCache` is a mutable directory that senro restores best-effort by key. Think module caches, a `~/.cargo` directory, anything where a miss just costs time and nothing else. It's related to a [workspace](/docs/data/workspaces/) but deliberately different: a scratch cache is **never an input to an action cache key**. ```go gomod := senro.ScratchCache("gomod", senro.Key(`gomod-{{ hashFiles "go.sum" }}`), senro.RestoreKeys("gomod-")) verify.Step("test", exec.Command("go", "test", "./...")). Mount(src.At("/src", senro.RO), gomod.At("/root/go/pkg/mod")) ``` A scratch cache mount takes **no mode**: `gomod.At(path)` is the whole call. A workspace mount would need `senro.RW` or `senro.RO` as well. ## `Key` and `RestoreKeys` - **`senro.Key(template)`** is the lookup key, and it's required. `Build()` refuses a scratch cache that doesn't have one: `plan: scratch cache "gomod" has no key, so there is nothing to look it up by`. - The key is a **template evaluated once per run**, before the first step runs. It has one function available, `hashFiles`, which takes globs relative to the pipeline process's working directory. It's not a general templating environment, so a key can't pick up machine state or the current date. - **`senro.RestoreKeys(prefixes ...string)`** are prefixes senro tries, in order, when the exact key misses. The newest entry under the first matching prefix wins. In the example above, `gomod-` means a lockfile change still starts from the last module cache instead of from nothing. ## What it is not - **Not an input to a cache key.** A step's [cache key](/docs/data/cache-keys/) never covers a scratch cache's content, its key, or its mounts. That's what makes a stale hit harmless: it costs time, never correctness. - **Not shared between machines, unless you ask.** A scratch cache stays local by default, and **`SENRO_REMOTE_CACHE` alone does nothing for it**: that variable says where the [action cache](/docs/data/caching/) lives, and scratch entries don't go there. Turning on `SENRO_REMOTE_SCRATCH` shares them too, which is worth it on cold CI runners and not worth it on a large tree whose key churns. See [Sharing scratch caches](/docs/data/scratch-sharing/). - **Not namespaced on one machine.** A scratch key carries no repository in it: it renders from lockfile content alone. On a machine whose scratch directory is reused across projects (a persistent build agent, say), one project's `RestoreKeys("go-")` can match another project's entries, since the local fallback is decided by mtime alone. Shared through a bucket, entries are namespaced by your pipeline's name, which is what keeps two projects apart there. - **Not mutable once written.** An entry is written under its key and never rewritten. If a cache saved the wrong bytes under a key, it would keep serving those wrong bytes forever. That's the rule behind the section below. ## On a remote executor A scratch cache works on all four executors. On [`k8s.Pod`](/docs/executors/kubernetes/) and [`ssh.Host`](/docs/executors/ssh/), it crosses to the target and comes back the same way a [workspace](/docs/data/workspaces/) does. The run saves **what comes back**, never the copy the coordinator sent out. | Executor | Scratch cache | | --- | --- | | `senro.Local()` | The coordinator's own directory | | `container.Image(...)` | That directory, bind-mounted | | [`k8s.Pod(...)`](/docs/executors/kubernetes/) | Carried in and back, `tar` over the apiserver | | [`ssh.Host(...)`](/docs/executors/ssh/) | Carried in and back, `tar` over the connection | Three things follow from this. The first is the expensive one: - **It costs two full transfers per step.** The whole cache goes out before the step runs and comes back after, with no incremental transfer in either direction. On Kubernetes, every byte crosses the **shared apiserver** both times. A dependency tree big enough to be worth caching is often big enough that transferring it twice costs more than the download it saves. Measure this yourself: if your `npm ci` takes 40 seconds and the tree is two gigabytes, don't put it on a pod. - **If the copy doesn't come back, nothing is saved.** A pod deleted mid-read, an apiserver that went away, or a tarball senro rejected all leave the entry unwritten. senro won't save the coordinator's stale copy under a key it could never fix later. The step itself is unaffected, and `senro cache explain` reports `not saved, the step's own copy never came back`. - **A later step in the same run usually starts from the same restored copy**, not from whatever an earlier remote step added to it. What comes back is set aside for the save, rather than written over a directory another step might still be sending out. The exception is a cache handed between a remote step and a local one, which the section below covers: those are ordered, so nothing can be mid-transfer and the newer tree replaces the directory. Unlike a workspace, nothing is excluded from a scratch cache, in either direction. `node_modules` and `.git` both cross, because `node_modules` is usually the whole point of caching one. ### Handing one between a remote step and a local one A remote step and a local or container step **can** share one scratch cache, as long as your graph orders them. A local step warms a module cache and a pod reuses it, or a pod fills one and a coordinator step reads what it produced: ```go gomod := senro.ScratchCache("gomod", senro.Key(`gomod-{{ hashFiles "go.sum" }}`)) warm := main.Step("warm", exec.Command("go", "mod", "download")).Mount(gomod.At("/go/pkg/mod")) test := remote.Step("test", exec.Command("go", "test", "./...")). Needs("warm"). // this is what makes it legal Mount(gomod.At("/go/pkg/mod")) ``` The cache has **one lineage**. Whatever the first step leaves is what the second one mounts, whichever kind of target each is, and the run saves the tree at the end of the chain rather than either side's copy of it. **Unordered, it is still refused**, and that is the whole rule: ``` plan: scratch cache "gomod" is mounted by step "build", which runs on a machine of its own, and by step "lint", which runs on the coordinator's filesystem, and nothing orders the two. [...] ``` A local step writes that directory live for as long as it runs. A remote step tarring it at that same moment would send a half-written tree and save it under a key nothing can ever rewrite. An ordering is what removes the "same moment", so `Needs` on either step fixes it; a second scratch cache also does. Because the rule reads your graph, **removing a `Needs` edge can turn a working pipeline into a build failure**. That is the safe direction to fail: the alternative is the same edit quietly corrupting an immutable entry. Two remote steps still share one freely, and may still run concurrently: neither writes the coordinator's directory, they only read it. > If a read-back fails, nothing is saved, hand-off or not. The entry is written once, so an > incomplete tree stored now is what every later run would be served. ## Inspecting one - `senro cache explain` reports every scratch cache the latest run touched, alongside its `Pure()` steps. A run with neither says so and still exits `0`. - A run directory records restores and saves in `cache/scratch.json`, which is `[]` when none was mounted. See [Reading a failed run](/docs/run/debugging/). - `senro verify --recheck-pure` treats a scratch cache as **cold**, an empty directory, since it's never an input to a cache key. See [Cache commands](/docs/cli/cache/). ## Where to go next - **[Caching a step](/docs/data/caching/)**: the action cache, which is the opposite trade. - **[Workspaces](/docs/data/workspaces/)**: when a directory has to survive correctly, not just cheaply. --- > Source: https://xavidop.github.io/senro/docs/secrets/channels # Secret channels A **channel** is a route a value can travel: any place a secret can end up once it leaves the struct `mamori` resolved. The file `SecretEnv` writes is one. So are a command's argument list, an environment variable, a step's stdout, the recorded plan, the cache entry and the event stream. `senro` gives every channel one of three grades: | Grade | What it means | |---|---| | **Safe** | Only the step's own account can read it, and `senro` never prints it | | **Redacted** | The value would land in bytes `senro` writes, so a placeholder goes there instead | | **Refused** | The value would land where `senro` cannot follow it. The run does not start | The how-to is [Secrets](/docs/secrets/). This page is the reference for which channel gets which grade, and how much each grade is actually worth. ## Every channel, graded | Channel | Grade | Notes | |---|---|---| | The file `SecretEnv` points at | Safe | Written once, readable only by the step's own account. Where it lands differs per executor, see [below](#where-the-file-lands-per-executor) | | `SecretEnv("VAR", "Field")`'s variable, and `SENRO_SECRET_` | Safe | Holds the file's **path**, never the value | | A step's or a handler's stdout and stderr | Redacted | Replaced before the bytes reach a log file, the event stream, or an attached client | | The event stream | Redacted | The same redactor, over every event payload | | The cache key | Identity only | The secret's name, its source and a digest of its value. Never the value | | A command argument | Refused | See [why](#why-those-three-are-refused-not-redacted) | | An environment variable's **value** | Refused | Same | | A step's `WorkDir`, a declared `Inputs`/`Outputs` pattern, or a mount's workspace name, scratch name or path | Refused | Same | | A file the step writes itself | Yours to handle | `senro` never reads a step's own output files. One that lands in a workspace snapshot or a declared output is stored exactly as written | ## Why those three are refused, not redacted Redaction only works on bytes `senro` itself writes. A command argument or an environment variable's value is readable by anything else on the machine the instant the process starts: ```sh $ ps -o args= -p 4123 npm publish --token npm_9f2c1e8bd4a7… # visible to every account on the box $ tr '\0' '\n' < /proc/4123/environ | grep TOKEN NPM_TOKEN=npm_9f2c1e8bd4a7… # and inherited by every child process ``` Nothing cleans that up afterwards, so `senro` refuses to start the run instead, before the first step executes and before anything is written to the run directory. The [error](/docs/secrets/#channels-senro-refuses) names the step and the channel, never the value. `WorkDir`, `Inputs`/`Outputs` patterns and a mount's own names are refused for a related reason: those strings are copied verbatim into the run's plan and into the cache, both of which outlive the run and sit behind no redactor at all. ## What redaction catches Say a step has run `TOKEN="$(cat "$NPM_TOKEN")"`, so the value is now in its own shell: ```sh echo "$TOKEN" # redacted in the log echo "$TOKEN" | base64 # redacted: base64 is a registered form printf '{"token":"%s"}' "$TOKEN" # redacted: so is JSON escaping echo "$TOKEN" | xxd # NOT redacted: hex is not registered echo "$TOKEN" | gzip > out.gz # NOT redacted: the bytes to match are gone echo "${TOKEN:0:8}"; echo "${TOKEN:8}" # NOT redacted: split by other content ``` The registered forms, matched across separate writes so a value split by the step's own output buffering is still caught: - the raw value - base64: the standard and URL alphabets, padded and unpadded - URL escaping: query form (`+` for a space) and path form (`%20`) - JSON string escaping, with and without HTML escaping of `<`, `>` and `&` - shell quoting: the body of a single-quoted word and of a double-quoted word ## What it does not Stated plainly, because a redactor believed to cover more than it does is worse than none: - **Hashing, compression or encryption.** A step that gzips its own log, or encrypts an artifact before writing it, defeats redaction outright: the bytes it would have matched are gone. - **Hex, base32, or any encoding not listed above.** - **A value split by unrelated content**, as in the last line above. A value split across two separate *writes* is still caught; one split by other content is not. - **Values shorter than six bytes.** `senro` refuses to start a run whose configuration holds one, rather than deliver a credential it cannot protect: redacting a four-byte value would redact unrelated output right along with it. - **Two secrets that overlap.** If one resolved value is a substring of another, replacing the first can leave a fragment of the second behind. The exact guarantee is that no *complete* occurrence of a registered value survives, and no stronger one is claimed. - **Anything outside the `senro` process**: `ps(1)`, `/proc//environ`, shell history, audit logs. That is what the refusals above are for, not the redactor. ## Where the file lands, per executor The variable always holds a path. What differs is where that path is and who else can read it. | Executor | Where the file is | Mode | Removed by | |---|---|---|---| | Local | A `0700` directory under `$XDG_RUNTIME_DIR`, `/dev/shm` on Linux, or `$TMPDIR` | `0600` | The step's attempt ending | | Container | That same directory, bind-mounted read-only into the step's container at `/run/senro/secrets`. Never `-e`, never `--env-file`, never a build arg | `0600` | The step's attempt ending | | Kubernetes | A namespaced `Secret`, projected read-only into the pod as a volume | `0400` | Owner-referenced to its pod, so deleting the pod deletes it | | SSH | A `0700` directory under the **host's own** runtime dir (`$XDG_RUNTIME_DIR`, `/dev/shm`, `$TMPDIR`, in that order). The value crosses as stdin bytes, never as an argument or a variable | `0600` | The attempt ending, or a detached reaper on the host with a six-hour TTL if the coordinator dies first | Two of those rows promise less than they look like they do: - **Kubernetes**: the value transits the apiserver and is stored in etcd, so anyone with `get secrets` in that namespace can read it for as long as the pod lives. The guarantee is "not in a pod field, not in a log", not "the cluster cannot see it". - **SSH**: the value becomes a plaintext file on a machine `senro` does not own, and the reaper is all that bounds how long it stays there. ## Isolation between steps in one run Strongest to weakest: | Executor | Can another step in the run read it? | |---|---| | Kubernetes | No. One `Secret` per attempt, projected into one pod, deleted with it | | Container | No, not through the filesystem: each step's secret directory is bind-mounted only into its own container. That is the container boundary doing the work, not a feature layered on top; the coordinator still writes and reads every secret file directly | | Local | **Yes.** Every step runs as the same user under one run directory. Treat every local step as equally trusted with every secret the run resolved | | SSH | **Yes, and worse than local.** Steps on one host share an account, so anything else that account runs can read it too, including a process that was already on the host before the run started | ## Two more caveats **Read-only mounts are only enforced on two executors.** The container executor makes `ws.At(path, senro.RO)` a real read-only bind mount, and Kubernetes does the same through `readOnly` on the pod's volume mount, where the kubelet is what refuses the write. Local and SSH cannot stop a live write: a workspace there is a plain directory with no per-step mode, so a step writing through an `RO` mount succeeds while it runs and fails only afterwards, once the workspace's content is found to have changed. Keep credentials and other sensitive input out of any workspace a step could overwrite by mistake on those two. See [Workspaces](/docs/data/workspaces/). **On macOS the secret file is not on tmpfs.** On Linux it goes under `$XDG_RUNTIME_DIR` or `/dev/shm`, both memory-backed, so unlinking it frees the pages outright. macOS has no equivalent for the local executor, so the file lands under `$TMPDIR`: still deleted when the step's attempt ends, but the bytes are not shredded and may persist in free disk space after the unlink. ## Where to go next - **[Secrets](/docs/secrets/)**: declaring one and reading it in a step. - **[Attach security](/docs/attach/security/)**: how redaction and attach's access control compose. - **[Cache keys](/docs/data/cache-keys/)**: exactly what a secret contributes to a key. --- > Source: https://xavidop.github.io/senro/docs/secrets # Secrets Give a step a credential without leaking it. You declare secrets as a typed struct, hand it to `senro.Run`, and each step names the field it needs. The rule to hold on to: a step receives a **file path**, never the value. Resolution is [mamori](https://github.com/xavidop/mamori)'s job, a separate library. `senro` never talks to a secret store. It takes the struct mamori resolved and decides how its values may reach a step. ## Declare credentials ```go import ( "github.com/xavidop/mamori" "github.com/xavidop/mamori/secret" "github.com/xavidop/senro" ) type Config struct { RegistryToken secret.String `source:"env:NPM_TOKEN"` Registry string `source:"env:REGISTRY" default:"ghcr.io/acme"` } cfg, err := mamori.Load[Config](ctx) if err != nil { return err } senro.Run(ctx, pipeline(cfg), senro.WithSecrets(cfg)) ``` - `mamori` is a separate module: `go get github.com/xavidop/mamori`. - `secret.String` (from `github.com/xavidop/mamori/secret`) marks a field as sensitive. A plain `string` with a `source` tag, like `Registry` above, is ordinary configuration, never a credential. - The `env:` scheme needs no extra setup. mamori also resolves from files and, through a separately installed provider package, from a cloud secrets manager. `senro` doesn't care which. - `senro.WithSecrets(cfg)` takes the resolved struct as a run option. Passing anything that isn't a struct, or a pointer to one, is an error `Run` returns, not a silently empty secret set. Each value is resolved once, before the run starts. ## Deliver one to a step ```go setup.Step("install", exec.Command("pnpm", "install")). SecretEnv("NPM_TOKEN", "RegistryToken") ``` `SecretEnv(envVar, field)` delivers the named field to this step as a **file**, and puts that file's path in the environment variable `envVar`: ```sh # inside the step: the variable holds a PATH, never the token itself npm config set //registry.npmjs.org/:_authToken="$(cat "$NPM_TOKEN")" ``` - The second argument is the **field name on the struct** you handed to `WithSecrets` (`RegistryToken` above), not the `source` tag (`NPM_TOKEN`). - Naming a field the struct does not have is refused when the run starts, with an error listing the fields that were resolved. - Every declared secret also arrives under a second, uniform name, `SENRO_SECRET_`: the field name uppercased, every character outside `A-Z`, `0-9` and `_` replaced by `_`. A step can read that without the pipeline having chosen an alias. - A field inside a nested struct is referenced with a dot (`"Registry.Token"`). A field promoted from an embedded struct keeps its bare name. ## What a secret does to a step's cache key The key in question is the **step's own** [cache key](/docs/data/caching/), the one that decides whether this step is skipped and its recorded outputs replayed. Nothing here is a key for the run, the workflow, or any other step. You write nothing for this. On any step the action cache considers, every secret it declares puts its *identity* into that step's key: the secret's name, its source, and a digest of its value salted with that source. A rotated credential invalidates a hit on the steps that declare it, and only those. The value itself never enters. `CacheEnv` is separate and has no bearing on this. Naming the same variable in both `SecretEnv` and `CacheEnv` is refused at build time, since a `SecretEnv` variable holds a path that changes every run and the secret's identity is already in the key. See [Caching a step](/docs/data/caching/) for what else enters a key. ## Channels senro refuses A **channel** is a route a value can travel: any place a secret can end up once it leaves the struct `mamori` resolved. The file `SecretEnv` writes is a channel. So are a command's argument list, an environment variable, the step's stdout, the recorded plan, the cache entry, the event stream. senro grades every one of them, and the grade turns on a single question: once the value is there, can senro still control who reads it? - **Safe**: the value goes somewhere only the step's own account can read, like the file `SecretEnv` writes. - **Redacted**: the value would land in bytes senro itself writes, like a log line or an event, so senro replaces it with a placeholder on the way out. - **Refused**: the value would land somewhere senro cannot follow it, like `argv`. There is no cleaning that up afterwards, so senro does not start the run at all. Three channels are refused, and a plan that routes a value into one never runs: - a command argument - an environment variable's **value** - a step's `WorkDir`, a declared `Inputs`/`Outputs` pattern, or a mount's workspace name, scratch name, or path In practice you reach one of them by pulling the value out of the resolved struct yourself: ```go // Refused: the value becomes argv[3], which ps(1) shows to every account on the machine publish.Step("publish", exec.Command("npm", "publish", "--token", cfg.RegistryToken.Reveal())) // Refused: the value becomes the variable's value, readable through /proc//environ for the // life of the process and inherited by every child it spawns publish.Step("publish", exec.Command("npm", "publish")). Env("NPM_TOKEN", cfg.RegistryToken.Reveal()) // Refused: the value ends up inside a path, which is written verbatim into the recorded plan and // into the cache, and both outlive the run publish.Step("publish", exec.Command("npm", "publish")). WorkDir("/build/" + cfg.RegistryToken.Reveal()) ``` All three have the same fix, which is the safe channel: let senro write the value to a file and hand the step the path. ```go publish.Step("publish", exec.Command("sh", "-c", `npm publish --token "$(cat "$NPM_TOKEN")"`)). SecretEnv("NPM_TOKEN", "RegistryToken") ``` The error names the step and the channel, never the value: ``` senro: engine: step "publish" puts the value of secret "RegistryToken" in command argument 3; a command argument is visible in ps(1), in shell history and in auditd execve records, where senro cannot redact it, so senro refuses to run rather than leak it. Deliver it as a file instead: SecretEnv("VAR", "RegistryToken"), then read "$VAR" as a path in the step ``` > The check runs inside `Run`, not `Build`. `Build()` never sees the resolved struct, so a test > that calls `p.Build()` to assert a pipeline is safe from this always passes. Call `senro.Run` to > exercise it. Everything `senro` itself writes (logs, events, the cache) is redacted rather than refused. [Channels](/docs/secrets/channels/) has the full table, why these three are refused instead of redacted, and what redaction cannot cover. ## Where to go next - **[Channels](/docs/secrets/channels/)**: safe, redacted and refused, plus per-executor delivery. - **[Steps](/docs/steps/)**: the rest of what a step can be configured to do. - **[Attach security](/docs/attach/security/)**: how redaction and attach's access control compose. --- > Source: https://xavidop.github.io/senro/docs/attach/security # Security Who can attach is a real security boundary. The control channel can retry a step, skip a step, cancel a run, and open an interactive shell inside a step's workspace. There are two boundaries because the two transports differ. The unix socket is the default, and it's the stronger of the two. ## The two transports, and what each one buys | | Unix socket (default) | TCP, loopback | TCP, not loopback | |---|---|---|---| | Reachable from off the machine | No, ever | No | Yes | | Reachable by another local user | **No** | Yes, the port opens for anyone | Yes | | What refuses them | Peer credentials (`SO_PEERCRED`/`LOCAL_PEERCRED`), fails closed | The bearer token, and only the token | The bearer token, and only the token | | Credential on the wire | None; there is no credential | Yes, in cleartext | Yes, encrypted | | TLS | Meaningless, and refused | Optional | **Required, no opt-out** | The second row matters most. Another local user cannot open the unix socket at all; over loopback TCP they can, and only the token stands between them and the run. The token is a remote shell. > That trade is worth making for a browser on the same machine or a port-forward. It is not worth > making because a socket path was inconvenient. ## The unix socket `attach.Options{}` with no `Bind`, or `attach.AutoUnixSocket`, binds a unix socket, mode `0600`, under a `senro/` directory inside the platform's runtime directory (see [Discovery](/docs/attach/#discovery)). That gives you two layers: - The file mode means only the owning user can open the socket, and the `senro/` directory is kept at `0700` on every platform, however permissive the runtime directory above it is. - The server also checks the connecting process's peer credentials (`SO_PEERCRED`/`LOCAL_PEERCRED`) and refuses a uid that doesn't match its own. File permissions alone aren't sufficient on every platform, so this check isn't redundant. Over a unix socket the boundary is **"whoever can already run code as you"**, not "whoever can reach a port". A unix listener takes no token, and passing one is refused rather than ignored. ## The TCP transport ```go att, err := attach.Listen(ctx, attach.Options{Bind: "127.0.0.1:0"}) // att.Addr() -> "127.0.0.1:53211", the real port // att.Token() -> the run's bearer credential ``` Every request must present the token as `Authorization: Bearer `, and only in that header. There's no `?token=` query parameter, which would put the credential into shell history, proxy logs, and the address bar. There's no cookie either, which would make the endpoint reachable by a cross-site request. The check sits in front of the **whole** server, not per route. `GET /api/logs` is a file read, `GET /api/stream` is the run's entire event history, and `POST /api/shell` is a command prompt: there's no endpoint an unauthenticated caller should reach, including any added later. ### The token - **32 bytes from `crypto/rand`**, 43 base64url characters. Guessing is not a strategy. - **Compared in constant time**, via `subtle.ConstantTimeCompare` over SHA-256 digests of both sides. Hashing first matters: `ConstantTimeCompare` is only constant-time when both inputs are the same length, so hashing makes both sides exactly 32 bytes regardless of what arrived, instead of leaking the token's length through timing. - **A wrong token and a missing token get identical answers**: the same `401`, headers, and body, saying only `unauthorized`. It never names the run, the pipeline, or senro, because even confirming "there's a senro engine on this port" is worth withholding from someone scanning for one. - **Never printed, logged, emitted or written into the run directory.** The run directory is the artifact you attach to a bug report, and the event stream reaches every attached client. ### Getting the token On the machine running the pipeline, **there's nothing to do**: `senro attach` finds the run, reads its token, and attaches. `Listen` writes the token into the run's registry entry, `/senro/.json`, mode `0600` inside that `0700` directory. That's the same boundary the unix socket has: anyone who could read it could already have attached as you. From somewhere else, over a port-forward, there is no registry entry to read, so you supply it: ```sh export SENRO_ATTACH_TOKEN='...' senro attach --addr 127.0.0.1:8443 --tls senro shell --addr 127.0.0.1:8443 --tls --step build ``` **There's no `--token` flag, and there won't be one.** A flag value lands in `argv`, where `ps` shows it to every other user, and in shell history. An environment variable doesn't. An embedder reads the token with `att.Token()` and stores it wherever that deployment keeps secrets. ### Bounded attempts Failed authentication attempts draw from a token bucket of 20, refilling at one per second. Past that limit, a failed request gets a `429` without even being evaluated. Each refusal also closes the connection, so a program looping on the port pays for a fresh TCP connection (and, over TLS, a fresh handshake) on every attempt. This is **not** a defense against guessing. 32 random bytes already makes guessing infeasible. It just bounds cost, so nobody can make the process hash, allocate, and respond indefinitely for free. A **correct** token never touches the bucket. ## TLS is required off loopback, with no flag to turn it off `attach.Listen` refuses a bind that is not loopback unless you give it `TLSCertFile` and `TLSKeyFile`. A wildcard bind (`":8443"`, `"0.0.0.0:8443"`, `"[::]:8443"`) is not loopback. > There's no opt-in plaintext flag, on purpose. A flag like that tends to go into a CI config once, > added by someone working around an error message, and then gets copied by people who never read > what it does. A hard refusal can't be copy-pasted past. The cost of refusing is low, because the alternative is already universal: ```sh attach.Listen(ctx, attach.Options{Bind: "127.0.0.1:8443"}) # in the pipeline kubectl port-forward pod/ci-runner 8443:8443 # or ssh -L 8443:127.0.0.1:8443 host export SENRO_ATTACH_TOKEN="$(kubectl exec pod/ci-runner -- cat /run/senro-token)" senro attach --addr 127.0.0.1:8443 # from wherever you are ``` A port-forward or an SSH tunnel supplies exactly that transport security, authenticated, with no certificate to manage. **Loopback without TLS is allowed**, deliberately. Loopback traffic never reaches a network, and capturing it needs privileges (root, or `CAP_NET_ADMIN`) that would already let you read the token out of the process's memory anyway. What it doesn't give you is the unix socket's stronger local guarantee. ### Where the certificate comes from: you senro doesn't mint a certificate for you. A self-signed certificate that the client trusts blindly would encrypt the connection without actually authenticating it: an attacker could present their own certificate and receive the token, which looks like protection but is worse than no TLS at all. Doing this safely means pinning the key and delivering that pin out of band, which this build leaves up to you. The client verifies certificates against the system's root store. There's no `--insecure` flag and no environment variable to disable verification: a client that skips verification would hand the token to whoever answers the port. For a private CA, point `$SSL_CERT_FILE` or `$SSL_CERT_DIR` at it. ## The shell is a remote code execution surface [`senro shell`](/docs/attach/shell/) runs commands interactively inside a sandbox, and **it works over TCP**: anyone holding the token gets a command prompt inside a step's workspace. Blocking just this one route wouldn't help much, since `step.retry` and `run.rerun_from` on the same listener already re-run a step's own command. Three things still hold true on both transports: - **A `ReadOnly` server refuses a session outright**, before the connection is upgraded. - **A session is delivered no secrets.** senro removes a step's secret files once its sandbox closes, and a session, which can stay open as long as somebody leaves the window open, never gets them back. See [The shell](/docs/attach/shell/#no-secrets-ever). - **The session's connection is separate from the control channel** and gets identical treatment: the same listener, the same peer check or token, the same refusal when the server is read-only. ## `ReadOnly` ```go att, err := attach.Listen(ctx, attach.Options{ReadOnly: true}) ``` A `ReadOnly` attach server serves the event stream and logs normally, but rejects every control operation and refuses a session: a shared, look-but-don't-touch view of a run. It's a field a pipeline author sets when embedding senro; there's no CLI flag for it, since the CLI just attaches to whatever the pipeline process chose. Note what `ReadOnly` is not: it's not authorization. It's one setting for the whole listener, not a per-client capability, so every client of a given attach server has the same powers. ## Platform support senro targets Linux and macOS. Windows isn't supported, deliberately: the peer-credential check above is `SO_PEERCRED` on Linux and `LOCAL_PEERCRED` on macOS, and neither exists on Windows. On any other platform, the connection is refused outright rather than allowed through unchecked. > An attach endpoint that quietly allows everyone in the moment its credential check can't run > would be worse than having no check at all: it would claim a protection it doesn't actually > deliver. The TCP transport doesn't change this: it doesn't need a peer check, but senro still won't build on Windows, since reaping a dead registry entry relies on a Unix-only syscall. Use Linux, macOS, or WSL2. ## Redaction is not authorization A pipeline that resolves credentials with `senro.WithSecrets` gets them redacted out of every stream sink (stdout, stderr, the event log) before a byte reaches an attached client. That is a backstop against a value leaking through logs, not a substitute for the checks above. The two work together: even a client that passes the peer check or token still never receives a secret's raw bytes, because the redactor sits in front of the stream no matter who's on the other end. See [Secret channels](/docs/secrets/channels/) for what neither can cover. ## Where to go next - **[The protocol](/docs/attach/)**: transport, discovery, and what `Listen` opens. - **[Control operations](/docs/attach/control-ops/)**: what a connected client can actually do. - **[The shell](/docs/attach/shell/)**: the session protocol, and what a prompt can reach. - **[Secrets](/docs/secrets/)**: resolution, delivery, and redaction in full. --- > Source: https://xavidop.github.io/senro/docs/data/shared-cache # Shared cache The [local cache](/docs/data/caching/) is nearly useless in CI, since every job starts on a fresh runner with an empty disk. A shared cache points every machine at one store instead, so a branch build can reuse what the trunk build computed, even on a runner that no longer exists. ## Turn it on Point `SENRO_REMOTE_CACHE` at an S3-compatible bucket: ```sh export SENRO_REMOTE_CACHE="s3://acme-senro-cache" export SENRO_REMOTE_CACHE_ENDPOINT="https://s3.eu-west-1.amazonaws.com" export SENRO_REMOTE_CACHE_REGION="eu-west-1" export AWS_ACCESS_KEY_ID=... # in CI, normally from an assumed role export AWS_SECRET_ACCESS_KEY=... ``` or at an OCI registry repository: ```sh export SENRO_REMOTE_CACHE="oci://ghcr.io/acme/senro-cache" export SENRO_REMOTE_CACHE_USERNAME="x-access-token" export SENRO_REMOTE_CACHE_PASSWORD="$GITHUB_TOKEN" ``` That's the whole setup, either way. No code change needed: a run that wasn't given a cache in code reads these variables, the same way it already reads `SENRO_CACHE_DIR`. The URL scheme (`s3://` or `oci://`) picks the backend. Everything on this page applies to both unless noted otherwise. [Cache stores](/docs/data/cache-stores/) covers choosing between them. > The two backends are alternatives, not layers. Setting a variable that belongs to the backend you > didn't choose is refused, not silently ignored, so a leftover `SENRO_REMOTE_CACHE_ENDPOINT` > can't sit there looking like it's doing something. ## How a lookup works The shared cache is a **second tier behind the local one**, never a replacement: ```mermaid flowchart LR L[Lookup] --> D{Local disk} D -- hit --> U[Use it] D -- miss --> S{Shared store} S -- hit --> V[Verify digest] --> W[Write through
to local disk] --> U S -- miss --> R[Run the step] --> SV[Save locally,
then upload] ``` Anything fetched gets written through to local disk on the way past, so a second run on the same machine needs no network at all. A save writes to local disk first, then uploads. - **[Cache keys](/docs/data/cache-keys/) do not change.** A shared cache changes where a result is stored, never what it is keyed by. - **A [scratch cache](/docs/data/scratch/) stays local** and has no remote tier. Its entries are whole-tree tarballs, re-keyed on every lockfile edit, and its restore-key fallback depends on local recency. Uploading them would cost more than it would save. ## When the store is unreachable **Nothing fails.** If the shared cache is down, slow, unauthenticated, refusing writes, or serving garbage, that just means *no shared cache*. Nothing else happens. The run keeps going against the local cache alone and finishes with the exit code it would have had anyway. senro isn't quiet about it. One line on standard error: ``` senro: remote cache s3 bucket acme-senro-cache at s3.eu-west-1.amazonaws.com failed on head and is not used for the rest of this run: s3: HEAD senro/v1/cas/sha256/...: dial tcp: connect: connection refused ``` senro also logs one `cache.degraded` event in the run's ledger, which the plain renderer prints to your CI log. After that, it stops trying to reach the store for the rest of the run, so an outage costs one timeout, not one timeout per object across several hundred objects. A registry behaves the same way, through the same code. One kind of problem does **not** degrade gracefully: a configuration that could never work. These fail the run immediately, before the first step runs, because they're mistakes in what someone wrote, not conditions of the network: - No bucket, or an endpoint that is not a URL - Credentials embedded in the endpoint or the registry host - A registry target naming no repository, or a repository name a registry could not accept - `SENRO_REMOTE_CACHE=s3://...` with `SENRO_REMOTE_CACHE_ENDPOINT` missing > A misconfigured cache that silently degraded would look exactly like a cold one, and nobody ever > investigates a cold cache. ## What stops a bad cache A cache that returns the wrong bytes is worse than no cache at all: the damage is silent, and it spreads to every machine. Two checks guard against this, and neither is optional or configurable: - **Every object is verified against the digest it was asked for**, hashed as it's read. A truncated download, a substituted error page, an object overwritten by hand: each of these counts as a cache **miss**, and the step just runs. The corrupt bytes never reach your workspace or local cache. - **Every cache entry is verified against the key it was filed under.** A hit skips the step entirely, so an entry served under the wrong key would mean a build that silently didn't do what it was told. Each entry carries its own key and is only trusted if that key matches the one asked for. Both problems are reported, and neither one turns the cache off, because a single bad object doesn't say anything about the rest of the store. ## Two machines finishing at once Neither backend locks anything, coordinates anything, or elects a leader: - **Objects.** If two runners complete the same step, they write the same object under the same name. The name is the content's digest, so both write identical bytes. A single write is atomic at the store, and the reader verifies whatever it gets. - **Cache entries.** Each one carries the run that produced it, so **last writer wins**. Both runners ran the same action under the same key, so either result is safe for a later run to reuse. The only difference is which run's logs a hit replays. ## Configuration `SENRO_REMOTE_CACHE` turns the cache on, says where it is, and picks the backend. The rest of the variables belong to one backend or the other. Setting one that belongs to the backend you didn't choose is an error, not a no-op: | Variable | Backend | Meaning | | --- | --- | --- | | `SENRO_REMOTE_CACHE` | both | Turns it on: `s3://`, `s3:///`, or `oci:///`. Unset means no shared cache | | `SENRO_REMOTE_CACHE_TIMEOUT` | both | Bounds one request, as a Go duration (`45s`). Default five minutes | | `SENRO_REMOTE_CACHE_READ_ONLY` | both | `1` reads the cache and never writes it | | `SENRO_REMOTE_CACHE_ENDPOINT` | bucket | The store's URL, e.g. `https://s3.eu-west-1.amazonaws.com`. Required | | `SENRO_REMOTE_CACHE_REGION` | bucket | Scopes the request signature. Required. `us-east-1` is the conventional answer for a store with no regions of its own | | `SENRO_REMOTE_CACHE_PATH_STYLE` | bucket | Overrides bucket addressing. Unset works it out from the endpoint | | `AWS_ACCESS_KEY_ID` | bucket | The credentials. Standard names, because CI already sets them | | `AWS_SECRET_ACCESS_KEY` | bucket | | | `AWS_SESSION_TOKEN` | bucket | Set when the credentials are temporary, the usual case for an assumed role | | `SENRO_REMOTE_CACHE_USERNAME` | registry | The credential presented to the registry's token endpoint. Both unset means anonymous | | `SENRO_REMOTE_CACHE_PASSWORD` | registry | | | `SENRO_REMOTE_CACHE_PLAIN_HTTP` | registry | `1` talks `http` rather than `https`. For a registry on a trusted network with no certificate | > The registry credentials are senro's own names rather than borrowed ones, because no standard > pair exists that a CI job already exports for a registry. Docker keeps its credential in a config > file senro deliberately does not read. senro reads no credential file and contacts no metadata service. The credentials are whatever the process was given, read once at the start of the run; a run outlasting a temporary credential degrades like any other authentication failure. ### In Go ```go senro.Run(ctx, p, senro.WithRemoteCache(senro.RemoteCache{ Endpoint: "https://s3.eu-west-1.amazonaws.com", Region: "eu-west-1", Bucket: "acme-senro-cache", Prefix: "pipelines", AccessKeyID: os.Getenv("AWS_ACCESS_KEY_ID"), SecretAccessKey: os.Getenv("AWS_SECRET_ACCESS_KEY"), })) senro.Run(ctx, p, senro.WithRemoteCache(senro.RemoteCache{ Registry: senro.RegistryCache{ Host: "ghcr.io", Repository: "acme/senro-cache", Username: "x-access-token", Password: os.Getenv("GITHUB_TOKEN"), }, })) ``` A registry's fields live in their own struct because a bucket and a repository share almost nothing. `Timeout` and `ReadOnly` stay on `RemoteCache` itself, and mean the same thing for either backend. Naming both a bucket and a registry at once is refused when the run starts. **Precedence**: an explicit `WithRemoteCache` wins over the environment: code that states what it wants shouldn't be silently overridden by an ambient variable. To read the environment yourself, pass `senro.RemoteCacheFromEnv()` to `WithRemoteCache`. Its zero value configures nothing, so this is safe whether or not anything is actually set. If a trunk build fills the cache while pull-request builds only read it, you'll need one more variable and a matching credential. See [Running it in CI](/docs/data/cache-stores/#running-it-in-ci). ## Credentials never travel A secret never reaches the shared cache, for the same reason it never reaches the local one: only a secret's [identity, not its value](/docs/data/cache-keys/), enters a cache key. See [Secrets](/docs/secrets/). The store's own credentials never enter a cache key, an object, a bucket key, or a registry tag, and are scrubbed from any error message before it reaches a log or an event. An endpoint or registry host that embeds a credential is refused outright, since both can appear in error messages. ## Housekeeping `senro cache` commands operate on the **local** cache directory: `senro cache gc` prunes your disk and does not touch the shared store. senro never deletes from a shared cache, so expiry there is the store's job. Expiring anything is always safe: objects are content-addressed and immutable, so the worst outcome is a miss, and the step just runs. See [Retention](/docs/run/archiving/#retention) for the lifecycle rule, and why logs usually deserve a longer one than the cache. ## Where to go next - **[Cache stores](/docs/data/cache-stores/)**: picking a bucket or a registry, and what each costs. - **[Cache keys](/docs/data/cache-keys/)**: what a key is made of, here and locally. - **[Archiving a run](/docs/run/archiving/)**: the same store holding a run's logs and ledger. - **[Caching a step](/docs/data/caching/)**: making a step eligible in the first place. --- > Source: https://xavidop.github.io/senro/docs/data/scratch-sharing # Sharing scratch caches A [scratch cache](/docs/data/scratch/) is local by default: it lives on the machine that filled it, and a fresh CI runner starts with none. `SENRO_REMOTE_SCRATCH` puts it in the same bucket the [shared cache](/docs/data/shared-cache/) already uses, so a cold runner starts from the tree the last one built. ```sh export SENRO_REMOTE_CACHE="s3://acme-senro-cache" export SENRO_REMOTE_CACHE_ENDPOINT="https://s3.eu-west-1.amazonaws.com" export SENRO_REMOTE_CACHE_REGION="eu-west-1" export SENRO_REMOTE_SCRATCH=1 # off unless you set this ``` It is a **separate variable on purpose**. Turning on the shared cache does not turn this on, because it is not always the right trade and it needs a permission nothing else in senro does. ## Whether you want it Turn it on when your runners start cold and installing dependencies dominates the build. That is the case it exists for: `npm ci` or `go mod download` on every job, against a bucket that already holds the answer. Leave it off when the tree is large and the key churns. An entry is one whole-tree tarball, and the key changes on every lockfile edit, so a dependency bump re-uploads the whole thing to save a download your toolchain already does incrementally. A two-gigabyte `node_modules` whose lockfile moves daily costs more in transfer than it saves. ## What it needs from your credential **`s3:ListBucket`, which nothing else senro does requires.** The `RestoreKeys` fallback is a prefix listing, so a credential scoped to `GetObject` and `PutObject` alone works for everything else and fails here. ```json { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::acme-senro-cache", "arn:aws:s3:::acme-senro-cache/*"] } ``` **A bucket, not a registry.** `oci://` targets ignore this variable: prefix fallback is a listing, and the registry API cannot list by prefix. This is the one place the two backends are not at parity ([Buckets & registries](/docs/data/cache-stores/)). ## How entries are kept apart Entries live at `scratch//`, namespaced by the name you passed to `senro.New`. The namespace matters because a scratch key names no repository: it renders from lockfile content alone, so two projects that both declare `RestoreKeys("gomod-")` would otherwise match each other's entries on one bucket. Your pipeline's name separates them at no configuration cost. If two projects share a name, give them different bucket prefixes: `s3://acme-cache/web` and `s3://acme-cache/api`. A run started with `senro.RunPlan` has no pipeline and therefore no name. Its scratch caches stay local rather than being written somewhere nothing distinguishes, since an entry is immutable once stored. ## The platform trap **A scratch key says nothing about the machine that filled it.** senro will happily restore a tree built on `darwin/arm64` into a `linux/amd64` pod, because the key is yours and senro does not inspect what is in the tree. That is fine for a Go module cache, which is sources. It is not fine for `node_modules` carrying compiled native addons, or any tree with platform-specific binaries in it. Put the platform in the key yourself when the content is not portable: ```go senro.ScratchCache("node", senro.Key(`node-linux-amd64-{{ hashFiles "package-lock.json" }}`), senro.RestoreKeys("node-linux-amd64-")) ``` The key is a template evaluated once per run on the coordinator, before any step runs, so it cannot know which executor will mount it. One cache mounted by both a local step and a pod has one key and one entry, by construction. ## When the bucket is unreachable Exactly what happens with no bucket configured: the run reads and writes the local scratch cache and carries on. Sharing never turns a miss into a failure, because a scratch cache's whole contract is that a miss costs time and nothing else. `SENRO_REMOTE_CACHE_READ_ONLY` applies here too. A fork's pull-request build reads what trunk filled and writes nothing back. ## Where to go next - **[Scratch caches](/docs/data/scratch/)**: `Key`, `RestoreKeys`, and what a scratch cache is not. - **[Sharing it across machines](/docs/data/shared-cache/)**: the action cache's own shared tier, which this rides alongside. - **[Buckets & registries](/docs/data/cache-stores/)**: choosing a store, and what a registry cannot do. --- > Source: https://xavidop.github.io/senro/docs/notifications/slack # Slack Posts a short line a person can read in a channel, to a Slack incoming webhook. ```go import "github.com/xavidop/senro/notify" n := notify.New(notify.Slack(os.Getenv("SLACK_WEBHOOK_URL"))) defer func() { _ = n.Close() }() err := senro.Run(ctx, p, senro.WithSink(n)) ``` You need an [incoming webhook URL](https://api.slack.com/messaging/webhooks) from Slack. senro does not read your environment looking for one: the URL is an argument. ## What lands in the channel By default, one message per run, when it finishes: ``` senro: run 20260807T101503-a1b2c3 succeeded in 1m12s (7 succeeded, 1 cached) senro: run 20260807T101503-a1b2c3 failed in 41s (5 succeeded, 1 failed, 2 skipped_upstream_failed) ``` Widen it with `On` and you get a line per event type you asked for: ```go notify.Slack(url, notify.On(api.RunStarted, api.StepFinished, api.RunFinished)) ``` ``` senro: pipeline ci started (run 20260807T101503-a1b2c3) senro: step test succeeded (run 20260807T101503-a1b2c3) senro: step build failed (run 20260807T101503-a1b2c3): exit status 2 senro: run 20260807T101503-a1b2c3 failed in 41s (1 succeeded, 1 failed) ``` > **`StepFinished` on a fan-out is one message per unit.** A two hundred step expansion is two > hundred Slack messages. The default is `run.finished` only for exactly this reason. ## Sending to more than one channel One destination per channel, each with its own webhook URL and its own name: ```go n := notify.New( notify.Slack(buildsURL, notify.Named("slack-builds")), notify.Slack(alertsURL, notify.Named("slack-oncall"), notify.On(api.RunFinished)), ) ``` `Named` matters here: both would otherwise be called `slack` in the run's `notify.delivered` events and in the shutdown report, and you could not tell which one failed. ## The URL is a credential A Slack incoming webhook URL is the whole of one: anybody holding it can post to that channel. So `notify` strips it out of every error it records or prints. It never appears in an event, in the shutdown report, or in a log line. Keep it out of your source. `os.Getenv` in the example above is the shape to copy. ## Options Every [option](/docs/notifications/#options) works here. The two you are most likely to want: | | | |---|---| | `On(types...)` | Which events reach the channel. Default: `api.RunFinished` only. | | `Named(name)` | The name in the run's ledger and the shutdown report. Default: `slack`. | ## Rendering it yourself `notify.Slack(url)` is exactly `notify.To(url, notify.SlackText(), notify.Named("slack"), notify.On(api.RunFinished))`. If you want Slack Block Kit, threading, or a different wording, write a renderer and post to the same URL: ```go notify.To(os.Getenv("SLACK_WEBHOOK_URL"), notify.RendererFunc(func(e api.Event) ([]byte, error) { var b api.RunFinishedBody if err := e.Decode(&b); err != nil { return nil, err } emoji := ":white_check_mark:" if b.Status != api.StatusSucceeded { emoji = ":x:" } return json.Marshal(map[string]string{ "text": fmt.Sprintf("%s `%s` %s in %s", emoji, e.Run, b.Status, b.Duration), }) }), notify.Named("slack"), notify.On(api.RunFinished), ) ``` See [Write your own](/docs/notifications/custom/) for the full seam. ## Where to go next - **[Notifications](/docs/notifications/)**: the options, the headers, and what happens when a delivery fails. - **[The event stream](/docs/run/event-stream/)**: every event type `On` can name. - **[Write your own](/docs/notifications/custom/)**: a destination senro does not ship. --- > Source: https://xavidop.github.io/senro/docs/executors/ssh # SSH `ssh.Host(dest)` targets a workflow at a remote machine. Every step runs as a process on that host: the step's command becomes the process, its output comes back on the connection's two streams, and its exit code comes from the host. ```go import "github.com/xavidop/senro/executor/ssh" builder := ssh.Host("deploy@build-07.internal", ssh.CacheClass("ubuntu-24.04/amd64/go1.26")) release := p.Workflow("release", senro.Needs("verify"), senro.On(builder)) release.Step("restart", exec.Command("systemctl", "restart", "web")) ``` ## Point a workflow at a host senro shells out to the `ssh` binary already on the machine running the pipeline. It embeds no SSH implementation of its own, and there's no key, password, or host key anywhere in a senro pipeline. **The destination you write is the destination you'd type yourself.** `ssh.Host("build-07")` connects to exactly what `ssh build-07` connects to: your `~/.ssh/config` (`Host`, `Match`, `Include`), your `known_hosts` (hashed entries, `@cert-authority` lines, host certificates), your `ProxyJump` and `ProxyCommand` bastions, your agent, hardware keys, PKCS#11 tokens, and any wrapper in front of `ssh`. If it works from your shell, it works here. senro adds `-T` (pipes rather than a terminal) and exactly one hardening option of its own, `-o BatchMode=yes`. It overrides none of your own settings: the only other options it passes are the [multiplexing](#connection-multiplexing) ones, and only when you haven't configured any yourself. `BatchMode` means a coordinator with no terminal fails instead of hanging on a passphrase prompt, and an unknown host key is refused instead of prompting, since nobody is there to answer. senro never passes `StrictHostKeyChecking` in either direction: `no` would hand a step's credentials to whatever answered, and `yes` would override an operator who deliberately chose `accept-new`. > A host new to your `known_hosts` fails the run with `ssh`'s own message. Add it the way you always > do. That's intended behavior, not a rough edge. ## What the host needs A POSIX shell, `tar`, and the ordinary utilities around them (`mkdir`, `rm`, `cat`, `printf`, `uname`, `nohup`, `sleep`). Nothing gets installed and no package manager runs. senro doesn't need root: everything it creates lives under `~/.senro/work` in the account's own space, or wherever you point it with `ssh.Host("build-07", ssh.WorkspaceRoot("/var/lib/senro/ws"))` on a fleet with small home directories. ## What runs where, at a glance | Behavior | On this executor | |---|---| | Workspaces | Carried to the host and back, both directions, as `tar` over the connection | | `senro.RO` mounts | A request, not enforceable remotely; writes through one are caught on read-back | | Secrets | Files on the host, delivered over stdin, removed at step end, reaped after six hours | | Scratch caches | Carried to the host and back, like a workspace. Two full transfers per step | | `Func` steps | Supported; the binary is staged once per host per release | | `senro shell` | Supported. `--tty` is refused with `executor_no_terminal` | | Connections | One per host for the whole run, over an OpenSSH control master, unless you opt out | | Environment | `env -i` with your declared variables plus the host's own `PATH` | | Cache class | `ssh//` by default; declare `ssh.CacheClass` for a fleet | ## Where things land on the host Each step attempt gets its own directory, named after the run, step, attempt and a random nonce: ``` ~/.senro/work/---/ ws// the step's workspaces status the command's exit code pid the wrapper's process id ``` - **A mount lives inside the attempt directory**, not at the host's root. `At("/src")` becomes `/ws/src`, since senro isn't root on the host and won't pretend it can create `/src`. - **`WorkDir` resolves against mounts.** `WorkDir("/src")` alongside a mount at `/src` resolves to that mount's directory. A working directory that no mount touches is used as written, so `WorkDir("/opt/app")` means `/opt/app` on the host, which is what makes an ordinary deploy step work. - **Nothing survives a run** except what a step itself creates, plus a `Func` step's staged binary, which sits alongside the attempt directories rather than inside one. ## Workspaces cross the connection, in both directions A mounted workspace is filled on the host before the step runs, and read back afterwards, as `tar` over the connection: a file you send is there, a file the step writes comes back, and a file the step deletes is gone from your copy too. Declaring one works the same way everywhere ([Workspaces](/docs/data/workspaces/)). ```mermaid flowchart LR Co["Coordinator"] -->|"tar in"| Ws["workspace dir
on the host"] Ws --> Cmd["step's command runs"] Cmd -->|"tar out"| Co ``` - **A mount carries exactly what a snapshot carries.** Paths excluded from snapshots (`.git` and `node_modules` by default) aren't sent, don't come back, and so aren't in your workspace directory afterwards. A step that needs the repository's history on the far side should fetch it there itself. - **The upside of that same rule**: your directory ends up exactly matching its recorded digest, where the local and container executors leave the excluded paths sitting alongside it. - **The bytes cross twice per step, on every attempt.** There's no incremental transfer and no resumption: a one-gigabyte workspace costs a gigabyte each way. - **A read-only mount is read back and hashed, but never written over your copy.** A remote step that wrote through one is caught and reported rather than carried home. As with the local executor, read-only is a request senro can't enforce on a far-side directory the step actually owns. A [scratch cache](/docs/data/scratch/) crosses the same way and is read back before the run saves it, so what lands under the key is whatever the host left behind. Two things differ: nothing is excluded from a scratch cache (`node_modules` is usually the point of it), and there's no digest, since a scratch cache never enters a cache key. It still costs the same two transfers per step, so a multi-gigabyte dependency tree can be slower to carry over than to just download again. If the copy doesn't come back, the run saves nothing rather than storing your stale copy under a key nothing can rewrite. Two ways to get more out of one. **Across runs and machines**, set `SENRO_REMOTE_SCRATCH` and the cache is kept in the bucket, so a fresh coordinator starts from what the last run built instead of sending an empty tree across ([Sharing scratch caches](/docs/data/scratch-sharing/)). **Within one run**, an ssh step can share a cache with a local or container step as long as a `Needs` orders the two: a coordinator step warms a module cache, the ssh step reuses it, or the reverse ([handing one over](/docs/data/scratch/#handing-one-between-a-remote-step-and-a-local-one)). ## Secrets are files on the host, and they are removed A secret's value crosses as bytes on the connection's standard input, into a file created under `umask 077` inside a directory senro creates at `0700`. The step learns the path through its environment, as on every executor. The value is never an argument (visible to `ps`, auditd's `execve` rules, or shell history), never an environment variable (`/proc//environ`), and never sent via `SendEnv`. The file lives in the host's own runtime directory, chosen in this order, and deliberately never in the attempt directory that senro reads back, because a credential must never be something that could end up in a snapshot: 1. `$XDG_RUNTIME_DIR`, when set and a directory. Per-user and tmpfs-backed. 2. `/dev/shm`, tmpfs by definition. senro creates a `0700` directory inside it. 3. `$TMPDIR`, or `/tmp`. Disk-backed, and senro doesn't claim otherwise. **When it goes away.** `Close` removes it at the end of the step, on every path, including a kept sandbox. Where the host has `shred`, the file is shredded first; elsewhere it's just removed. A detached reaper, armed before anything is written, covers the case where the coordinator dies first: it fires after six hours, and a step that outlives it loses its credential files and fails loudly on the next read. Steps on one host share an account, so a step can read another step's secret directory. This is the weakest isolation of the four executors ([Secret channels](/docs/secrets/channels/)). ## The cache class is not the hostname Left undeclared, `Class()` reports `ssh//`, read from the host with `uname`. It's deliberately not the hostname: a class built from host identity would mean a fleet of forty identical machines never shares a cache entry, and nothing would tell you why. Declare what actually makes two hosts interchangeable, with `ssh.CacheClass("ubuntu-24.04/amd64/go1.26")` on both, and they share cache entries without senro contacting either one. Keeping the class honest is on you: senro can't tell that `build-07` quietly picked up a different Go toolchain. ## What a failure means `ssh` exits with the remote command's status, or with 255 for its own failures, so a bare 255 could mean either "the connection broke" or "your command exited 255". senro resolves the ambiguity on the host: a wrapper writes the command's real status to a file before exiting with it. On any code but 255, senro trusts the code directly. On 255, it opens one extra session to read the file. If the file is present, the command ran and the file is the verdict. If it's missing, nothing ran, and this counts as infrastructure. You get the command's exit code back, with no error and no retry from `retry.OnInfra()`, when: - the command exited non-zero, whatever the code, including 255 - the command doesn't exist on the host (exit 127) - the command was killed on the host, by the OOM killer or anything else (exit 128 + signal) It counts as an infrastructure failure, which [`retry.OnInfra()`](/docs/steps/retries/) retries, when: - the host could not be reached, authentication failed, or the host key did not verify - the connection dropped before the command recorded a status - the step's working directory does not exist on the host - a workspace could not be sent or read back - the run was cancelled ## Traps - **A login banner can land in your step's output.** Shells that print from startup files share your command's stream even non-interactively. senro's own scripts read only marked lines and are immune to this, but your step's output is not. Silence a printing `.bashrc`, or expect the banner to show up. - **A step gets the environment you declared, and nothing else**: `env -i` with the plan's variables plus the host's own `PATH`. It can't inherit the remote login environment, so it won't pick up `SSH_AUTH_SOCK` and hand a build step your keys. The [trace context](/docs/extend/exporter/) travels on that same list, visible in `ps` like every declared variable, which is why a secret crosses as stdin bytes instead. - **Every phase opens its own session** (host prep, each workspace, each secret, the command, read-back, cleanup, a `Func` step's binary check and push), all riding the same [connection per host](#connection-multiplexing). - **Cancelling a run doesn't guarantee the remote command is dead.** senro closes the session and signals the wrapper's recorded pid, and `sshd` tears the session down, but a command that detached from its session outlives all three. This is not the same guarantee as deleting a pod. - **senro depends on a binary it doesn't control.** An `ssh` old enough to lack an option, or a wrapper named `ssh` that does something else, changes what a step does. ## `Func` steps run here too A registered Go function runs on the host's filesystem, against its network. senro stages your pipeline binary at the content-addressed `/bin/senro-sha256-` and re-enters it there, once per host per release rather than per step. A differing platform costs a `CGO_ENABLED=0` cross-compile ([Func steps off the coordinator](/docs/executors/func-remote/)). ## Connection multiplexing A run pays for connection setup once per host, not once per command: the first session opens an OpenSSH control master (OpenSSH's own name for a shared connection that later sessions multiplex over, the same thing `ControlMaster`/`ControlPath` configure in `ssh_config`), and every later session rides it, so a step costs a handshake instead of six. - **The control socket is guarded like a session**, because opening it is one: a random name in the same private (`0700`) runtime directory as the attach socket. senro closes the master when the run ends, on every path, and its `ControlPersist` setting removes it if the coordinator is killed first. - **Your configuration wins.** If your `ssh_config` already resolves a `ControlPath` for the destination, senro adds no multiplexing option at all, and yours stays in force, the same as with host key policy. It checks this with `ssh -G`, which connects to nothing. - **A master that won't open isn't fatal.** The run carries on with a connection per command and says so once on standard error. `ssh.Host(dest, ssh.NoMultiplexing())` chooses that deliberately, for a fleet where one shared connection is the wrong trade-off. - **One broken master fails the commands riding it.** They retry as infrastructure failures, and the next one opens a new master. - **`sshd`'s `MaxSessions` (default 10) caps how many steps share a connection.** senro keeps at most 8 on the master and gives anything over that its own connection, so parallelism is never capped by this setting. - **The cap is 8 rather than 10 because exceeding it doesn't fail a step.** `ssh` prints `Session open refused by peer` into that step's own stderr and then succeeds on a fresh connection. Lower `MaxSessions` below 8 on a host and that line starts appearing in your logs. ## What is not here - Bastion support beyond the `ProxyJump` and `ProxyCommand` you already have. - A host-facts cache across runs: `uname` is read once per host per run. - Incremental workspace transfer, and a disk-space check before one. - A terminal for `senro shell`, refused with `executor_no_terminal`, because `ssh` driven from pipes has no window size to give a remote pty ([Shell](/docs/attach/shell/)). - A [scratch cache](/docs/data/scratch/) shared with a step on the coordinator's own filesystem **when nothing orders the two**, refused at build time: a local or container step writes that directory while it runs, and an ssh step tarring the same directory at that same moment would send a half-written tree and then save it under a key nothing can rewrite. Put a `Needs` between them and it is allowed, because there is no longer a same moment: the cache is handed from whichever runs first to whichever runs second. Two ssh steps can still share one freely, ordered or not. --- > Source: https://xavidop.github.io/senro/docs/steps/states # Step states A step ends in exactly one of ten states, never a boolean, because "did it pass" hides information a build system should surface. This is the full set the wire protocol declares, and all ten happen today. | State | How a step ends there | What its dependents get | |---|---|---| | `succeeded` | Passed without ever failing | They run | | `recovered` | Failed at least once, then passed on retry | They run | | `cached` | A `Pure()` step hit the local action cache: skipped entirely, recorded outputs restored | They run | | `failed` | Ran, failed, and exhausted any retries | `skipped_upstream_failed`, unless the step declared `ContinueOnError` | | `timed_out` | An attempt outlived the step's `Timeout` | `skipped_upstream_failed`, unless the step declared `ContinueOnError` | | `cancelled` | The run was cancelled before the step could finish | Nothing further is dispatched; the run ends `cancelled` | | `panicked` | A `senro.Func` step's registered function panicked; the panic is caught and reported rather than crashing the run | `skipped_upstream_failed`, unless the step declared `ContinueOnError` | | `skipped_upstream_failed` | A step it depends on failed | `skipped_upstream_failed`, transitively | | `skipped_condition` | Its `When` condition was not met at run start | `skipped_condition`, transitively. `ContinueOnError` does not rescue them | | `skipped_manual` | An operator took it out of a live run with `step.skip` | `skipped_manual`, transitively. `ContinueOnError` does not rescue them | `cached` is a real hit, not a placeholder: the step is not run and its recorded outputs are restored from the action cache. See [Caching a step](/docs/data/caching/). ## `recovered` is not `succeeded` A step that failed an attempt and then passed settles as `recovered`. The two are deliberately kept apart: collapsing them is how flaky infrastructure stays invisible for months, with a build that needed three attempts looking identical to one that needed one. A run full of `recovered` steps is still a passing run. It is a passing run that is telling you something. See [Retries](/docs/steps/retries/). ## `skipped_condition` is not `skipped_upstream_failed` A step can end up not running for two very different reasons, and senro records which. ### Something broke → `skipped_upstream_failed` ```go verify.Step("build", exec.Command("make", "build")) verify.Step("test", exec.Command("make", "test")).Needs("build") ``` `build` fails. `test` never runs, and settles as **`skipped_upstream_failed`**. The run ends `failed`, because something did. ### Nothing broke → `skipped_condition` ```go deploy := p.Workflow("deploy", senro.When(senro.Branch("main"))) deploy.Step("apply", exec.Command("./deploy.sh")) deploy.Step("smoke", exec.Command("./smoke.sh")).Needs("apply") ``` On a pull request, `apply` is gated off and settles as **`skipped_condition`**. `smoke` settles the same way. The run ends **`succeeded`**, because nothing failed: your deploy was not supposed to run on a pull request, and it did not. That is the whole point of the distinction. **A pull request's run stays green when its main-only deploy does not fire.** ### Side by side | | Something broke | Nothing broke | |---|---|---| | **What happened upstream** | A step `failed`, `timed_out` or `panicked` | A step was gated off by `When`, or skipped by an operator with `step.skip` | | **Dependents end as** | `skipped_upstream_failed` | The **same** state as the cause: `skipped_condition` or `skipped_manual` | | **The run ends** | `partial` or `failed` | `succeeded` | | **Does `ContinueOnError` rescue them?** | **Yes.** That is what it is for. | **No.** | ### Why `ContinueOnError` only helps on the left `ContinueOnError` says "run my dependents even though I failed, against what I did produce". It is about surviving a *failure*. A `skipped_condition` step produced nothing at all, because it never ran. There is no output to run against, and nothing was blamed, so there is nothing for `ContinueOnError` to excuse. See [Conditions](/docs/steps/conditions/) and [Control operations](/docs/attach/control-ops/). ## How far a failure travels - **Only downstream.** A failing step settles its direct dependents, and theirs in turn. Unrelated branches are not cancelled: they run to completion, so a failure produces one clear report instead of a half-explored graph. - **A skip does not poison the graph either.** Only the transitive dependents of a `skipped_condition` or `skipped_manual` step are affected. ## The run's own rollup `run.finished` carries a status and a per-state count of the steps: ```json {"type":"run.finished","payload":{"status":"failed", "steps":{"failed":1,"skipped_upstream_failed":1,"succeeded":1}}} ``` Five statuses exist, in this precedence, strongest first: | Run status | When | |---|---| | `cancelled` | Any step is `cancelled`. It outranks failure, since a step that failed while the run was being torn down says nothing useful about the workload | | `failed` | Any step is `failed`, `timed_out` or `panicked` | | `partial` | Nothing failed, but some step is `skipped_upstream_failed` | | `succeeded_with_recovery` | Nothing above, and some step is `recovered` | | `succeeded` | Everything else. `cached`, `skipped_condition` and `skipped_manual` all roll up clean | ## Where to go next - **[Retries](/docs/steps/retries/)**: what produces `recovered` instead of `failed`. - **[Handlers](/docs/steps/handlers/)**: `SENRO_FAILURE_STATE` carries one of these values. - **[Conditions](/docs/steps/conditions/)**: what produces `skipped_condition`. - **[Control operations](/docs/attach/control-ops/)**: `step.skip` and `skipped_manual`. - **[The event stream](/docs/run/event-stream/)**: where a state is recorded. - **[Reading a failed run](/docs/run/debugging/)**: reading these states out of a run. --- > Source: https://xavidop.github.io/senro/docs/steps # Steps A step is one action with an id: a command, or a registered Go function. Steps are what a pipeline is made of, and every page in this section configures one. ```go p := senro.New("ci") verify := p.Workflow("verify") verify.Step("test", exec.Command("go", "test", "./...")). Timeout(5 * time.Minute) plan, err := p.Build() ``` `Workflow(name, opts...)` adds a named group and returns a `*senro.WorkflowBuilder`. `Step(id, action)` adds a step to it and returns a `*senro.StepBuilder`. Every builder method returns the same builder, so calls chain. `Build()` resolves the whole pipeline into a validated, immutable `Plan`; see [Concepts](/docs/concepts/) for why that boundary exists. **Step ids are unique across the whole pipeline**, not per workflow, because a plan is flat. `Build()` refuses a duplicate, and refuses a step with no action. ## The two step kinds | Kind | What it runs | |---|---| | `exec.Command(name, args...)` | A command, exactly as given | | `senro.Func(name, params)` | A Go function registered under `name`. See [Func steps](/docs/steps/functions/) | Both kinds are built, scheduled, retried, cached and handled by exactly the same code, so reach for `senro.Func` whenever the work is "call this Go function", not "shell out to a program". ### `exec.Command` interprets no shell `exec.Command("go", "test", "./...")` runs the program `go` with two arguments, exactly as written. There is no shell in between, so **none of the things a shell does happen**: | You write | What a shell would do | What senro does | |---|---|---| | `exec.Command("ls", "*.go")` | Expand `*.go` to your files | Passes the literal `*.go` to `ls` | | `exec.Command("echo", "$HOME")` | Substitute your home directory | Passes the literal `$HOME` | | `exec.Command("go build && ls")` | Run two programs in sequence | Looks for one program whose whole name is `go build && ls` | | `exec.Command("cd", "web")` | Change directory | Runs `/bin/cd`, which changes nothing | Each has a direct replacement: - **Globs, pipes, redirection, `&&`**: ask for a shell in so many words. ```go verify.Step("test", exec.Command("sh", "-c", "go test ./... | tee test.log")) ``` - **Environment variables**: declare them with `Env`, which the step sees for real. ```go verify.Step("test", exec.Command("go", "test", "./...")).Env("CGO_ENABLED", "0") ``` - **Changing directory**: use `WorkDir`. ```go verify.Step("build", exec.Command("pnpm", "build")).WorkDir("./web") ``` Both are on [Env, dir & timeout](/docs/steps/settings/). > This is a feature, not a restriction to work around. An argument that is never re-parsed cannot > be split on a space you did not expect, and a filename with a space in it is just a filename. ## What you can configure Everything below is a method on the `*senro.StepBuilder` that `Step(...)` returned. | I want to... | Call | Page | |---|---|---| | Run this after another step | `Needs` | [Ordering](/docs/steps/ordering/) | | Set env vars, a directory, a time limit | `Env`, `WorkDir`, `Timeout` | [Env, dir & timeout](/docs/steps/settings/) | | Let dependents run even if this fails | `ContinueOnError` | [Env, dir & timeout](/docs/steps/settings/) | | Try again when it breaks | `Retry`, `RetryPolicy` | [Retries](/docs/steps/retries/) | | Clean up or collect logs afterwards | `OnFailure`, `Always` | [Failure handlers](/docs/steps/handlers/) | | Skip it unless something is true | `When` | [Conditions](/docs/steps/conditions/) | | Give it files, and keep what it wrote | `Mount`, `NoSnapshot` | [Workspaces](/docs/data/workspaces/) | | Skip it when nothing changed | `Pure`, `Inputs`, `Outputs`, `CacheEnv` | [Caching a step](/docs/data/caching/) | | Give it a credential | `SecretEnv` | [Secrets](/docs/secrets/) | ## Two things that are not step settings **Where a step runs belongs to its workflow.** `senro.On(...)` on the `Workflow` call picks the executor for every step in it, so a step never carries one of its own: ```go remote := p.Workflow("remote", senro.On(ssh.Host("build@ci-1"))) remote.Step("build", exec.Command("make", "build")) // runs on ci-1 ``` See [Executors](/docs/executors/). **Generating steps instead of writing them** is a workflow-level call too. If your repository has many apps, modules or packages and you want one step each, that is [Monorepos](/docs/monorepo/), not a step setting. ## Where to go next - **[Ordering](/docs/steps/ordering/)**: the two `Needs`, and the graph they build. - **[Step states](/docs/steps/states/)**: the ten states a step can end in. - **[Executors](/docs/executors/)**: the four places a step can run. - **[Reading a failed run](/docs/run/debugging/)**: where the evidence is. --- > Source: https://xavidop.github.io/senro/docs/analyzers/genkit # The AI analyzer `contrib/genkitanalyzer` is a ready-made [failure analyzer](/docs/analyzers/) that asks a model what broke. You supply the model; senro supplies the failure. It is built on [Genkit](https://genkit.dev), Google's open-source AI framework for Go, so the provider is whatever Genkit has a plugin for: Gemini, Vertex AI, OpenAI, Anthropic, Ollama and the rest. ## Install ```bash go get github.com/xavidop/senro/contrib/genkitanalyzer ``` It is a **nested module with its own `go.mod`**, so senro itself never pulls in an AI SDK. If you never install this, your dependency graph never hears about Genkit. ## Use it ```go package main import ( "context" "log" "time" "github.com/firebase/genkit/go/genkit" "github.com/firebase/genkit/go/plugins/googlegenai" "github.com/xavidop/senro" "github.com/xavidop/senro/contrib/genkitanalyzer" ) func main() { ctx := context.Background() // Your Genkit instance: your plugin, your credential, your telemetry. g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{})) p := senro.New("ci") // ... workflows and steps ... err := senro.Run(ctx, p, senro.WithAnalyzer( genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash")), senro.AnalyzerName("genkit"), senro.AnalyzeTimeout(20*time.Second))) if err != nil { log.Fatal(err) } } ``` That is the whole integration. When a step fails, its explanation shows up in the TUI and in the run's event stream, exactly as [Failure analyzers](/docs/analyzers/) describes. `New` takes the `*genkit.Genkit` **you** built. The package never calls `genkit.Init`, never registers a plugin, never reads an API key and never falls back to a provider of its own. ## Options | Option | What it does | | --- | --- | | `Model(name)` | The model, in Genkit's `provider/model` spelling. Leave it out and Genkit uses whatever you set with `genkit.WithDefaultModel`. | | `Prompt(fn)` | Replaces the prompt built per failure. `DefaultPrompt` is exported, so add to it rather than starting over. | | `Remedy(fn)` | Replaces the retry policy below. It is handed the `api.Failure`, never the model's answer. | ```go a := genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash"), genkitanalyzer.Prompt(func(f api.Failure) string { return genkitanalyzer.DefaultPrompt(f) + "\nThis pipeline builds a Go module.\n" })) ``` ## What leaves your machine The prompt is built from [`api.Failure`](/docs/analyzers/custom/#what-you-are-handed-apifailure) and nothing else. Every string on that struct has already been through the run's redactor. If you replace the prompt, **build it from `f` alone**. A value you read from an environment variable, a file or the workspace has been through no redactor at all, and everything in the prompt is sent to your provider. ## The model does not choose the remedy Whether a proposal offers "retry this step" is decided from the failure senro recorded, never parsed out of the model's answer. A model writing "just retry it" is not evidence that retrying is safe. ```go func DefaultRemedy(f api.Failure) api.Remedy { if f.State != api.StateFailed { return api.RemedyNone } if f.Error != "" && f.ExitCode == 0 { return api.RemedyRetry } return api.RemedyNone } ``` In words: **only a step whose process failed to start or run at all is offered a retry.** A non-zero exit code is the workload's own verdict, and running it again until it passes deletes what it just told you. A `timed_out` step hit a budget you wrote down; a `panicked` step is a bug in your Go code. `Remedy(fn)` swaps that policy out if you know your own infrastructure better. What it cannot do is widen what a proposal may ask for: the vocabulary stays [one word long](/docs/analyzers/custom/#the-remedy-vocabulary-is-one-word-long). ## When the model has nothing useful to say `Analyze` returns an error wrapping `ErrNoAnswer` rather than a proposal with an empty summary, and senro reads that as "no comment": nothing is appended to the ledger, and the run's own result is unchanged. `ErrNoAnswer` is exported, so an analyzer composing several sources can tell "the model had nothing to say" apart from "the provider was unreachable", and fall back to a local classifier for the first without retrying the second. ## Testing The package's own tests define a model in-process with `genkit.DefineModel` and reach no network, which is worth copying: a test that needs an API key cannot run in CI and cannot run on a fork. ## Where to go next - **[Failure analyzers](/docs/analyzers/)**: the gate, and who turns a proposal into an action. - **[Write your own](/docs/analyzers/custom/)**: the interface, when a model is not the right answer. - **[Control operations](/docs/attach/control-ops/)**: `analysis.accept` and `analysis.reject`. --- > Source: https://xavidop.github.io/senro/docs/attach/browser # The browser UI `senro ui` shows a **live** run in a browser, with controls. Run it alongside the pipeline. It prints a link that works once. ```console $ senro ui http://127.0.0.1:52413/h/kQ8x... senro ui: serving a view of this run, with controls. The link above works once. Press Ctrl-C to stop. ``` Open the link. The page it lands on carries no credential in its URL. ## What it shows Steps appear in the same order as the terminal UI, so the two match. Each step shows its state, elapsed time, and dependencies. Steps expanded from a fold are indented under their parent, which shows the group's summary. Selecting a step shows its detail (kind, attempt, exit code, error, any handlers that ran and how they ended) and tails its output. ## Controls The page offers the same control operations [the TUI](/docs/attach/tui/) does, except one. It decides which to show based on each step's current state, not a fixed list: | Scope | Offered | When | | --- | --- | --- | | Run | Pause / Resume | While the run is live. Follows a pause applied from anywhere, including `senro attach` or the CLI | | Run | Cancel run | While the run is live. Asks first | | Step | Release | The step is held at a breakpoint | | Step | Retry, Rerun from here | The step has finished. `Rerun from here` asks first | | Step | Break before, Skip | The step has not started. `Skip` asks first | A finished run offers no controls, and neither does a step that's currently running. If the engine would refuse an operation, the button for it isn't shown at all. A button that just produces a refusal trains people to ignore it. The one exception is [`ws.snapshot`](/docs/attach/control-ops/#forcing-a-snapshot). The UI server can forward this operation, but no button triggers it, because the page can't tell in advance whether a step has a workspace to snapshot. Use the TUI's `w` key for this instead: it shows the engine's refusal if there's nothing to capture. Nothing is applied optimistically. A control request's answer arrives as an event in the stream, so the page updates exactly when the run does, and in the same order the TUI shows. ### What is not forwarded, and why `POST /api/shell` has no route on the UI server and no handler in it. Use [`senro shell`](/docs/attach/shell/) from `senro attach` instead. A page that can steer a run is a reasonable thing to hand to whoever holds the one-time link. A page that can run arbitrary commands is not. That boundary is enforced by routing: the shell endpoint simply doesn't exist here, rather than by a check that could be forgotten later. Control requests are held to a check the read routes aren't: a `POST` must carry an `Origin` header that exactly matches the UI server's own. A request without one is refused. This isn't redundant with the `SameSite=Strict` cookie. A "site" doesn't include the port, so a page served by any other process on `127.0.0.1` counts as same-site with this server, and its requests would still carry the cookie. The `Origin` check is what tells them apart. ## Where the token goes The run's bearer credential ([Security](/docs/attach/security/)) never reaches the browser. `senro ui` holds it in its own process and adds it to the routes it forwards. A token in a URL ends up in browser history, `Referer` headers, and screenshots. A token in `localStorage` or a readable cookie is one injected script away from being stolen. Neither risk applies here, because the page never has the credential at all. What the browser holds instead is a session cookie for the UI server alone: - **HttpOnly**: neither the page's scripts nor the WebAssembly module can read it. - **SameSite=Strict**: no other page can cause it to be sent. - **Session-scoped**, with no `Expires`: it never reaches disk, and it is meaningless once `senro ui` exits anyway. The cookie is minted from the one-time nonce in the printed link. That nonce is the only place a credential ever touches a URL: it ends up in your terminal scrollback, and possibly in browser history, since some browsers record redirect chains. It's not reusable. The first use redirects to `/` and spends it; a second attempt gets the same 404 as any unknown path. ## What it binds Always loopback. There's no flag to widen it. A browser UI on a routable address would put a live build's view, and the session cookie that opens it, on the network in plaintext. The attach server already refuses a non-loopback bind without TLS; this follows the same rule. Every request's `Host` header is checked against the loopback names the server actually bound to. That defends against DNS rebinding, where an attacker's domain resolves to `127.0.0.1` after the page loads, tricking the browser into treating it as same-origin. The `Host` header is the one thing that trick can't fake. To watch a run on another machine, forward the port (`ssh -L`) and point a local `senro ui --addr` at the forward, with the run's token in `$SENRO_ATTACH_TOKEN`. That way both the credential and the traffic are protected. ## Resuming The client speaks the protocol every other client speaks: `GET /api/state` for a snapshot, then `GET /api/stream?from=` to tail it. It also handles the case where the resume point has fallen out of the retained event buffer. Whether that shows up as a `410 Gone` before the stream opens or as a terminal `overflowed` marker mid-stream, the client just takes a fresh snapshot and subscribes from there. The new snapshot **replaces** the old state instead of merging with it: the events in between are gone, and merging would show a run that never actually happened. If a stream ends without saying why (the connection died before the server could write its marker), the client reconnects from where it left off, with pacing and a limit, so it won't loop forever against a server that immediately closes every subscription. ## What it does not do A finished run has no attach server, and this client only speaks HTTP to one. To read a finished run, use `senro attach --run --follow`, which tails the run's own files from disk. It does not open a shell; see [What is not forwarded, and why](#what-is-not-forwarded-and-why). ## Why the client is Go and not JavaScript The browser page is a Go client compiled to WebAssembly. It folds the run's events using the exact same code the TUI, the attach server, and offline replay all use (`api.RunState.Apply`). That's deliberate: a separate JavaScript implementation of that logic could drift out of sync, and the failure would be silent: the browser showing a pass while the terminal shows a fail, for the same run. Using one shared implementation rules that out. ## What it costs The client is about **4.0MB** of WebAssembly (**1.1MB gzipped**), embedded in the `senro` binary and downloaded by the browser once per session. Most of that size is unavoidable: a minimal Go WebAssembly binary already starts around 2MB. The binary embeds the client gzipped, and everything it serves carries an `ETag`, so a reload after the first load is a fast conditional request with no body. --- > Source: https://xavidop.github.io/senro/docs/triggers/events # The event file The file a dispatcher hands your pipeline. Read [Triggers](/docs/triggers/) first for the wiring; this page is the format, and where to find the details of the source you use. > **Running as a server?** You need none of this. `trigger.FromRequest` reads a webhook delivery > straight off the wire, headers and signature included, with no file and no envelope. See > [Run it as a server](/docs/triggers/server/). ## Load one ```go func LoadEvent(path string, providers ...Provider) (*Event, error) func ReadEvent(r io.Reader, providers ...Provider) (*Event, error) ``` `path` is a file, `-` for standard input, or `""` for no event at all. **The empty case is not an error.** A pipeline run by hand has no event, and `senro.Run` given a nil event gates nothing and runs, so `./pipeline` builds everything, which is what the local loop wants. A dispatcher that forgets the flag therefore over-runs, which somebody notices, rather than never running, which nobody does. ## The format An envelope naming where the event came from, wrapped around the source's own payload: ```json { "provider": "github", "event": "push", "payload": { "ref": "refs/heads/main", "before": "...", "after": "...", "commits": [] } } ``` `provider` and `event` are both **required**. None of the four webhook bodies says which event it is (that is the `X-GitHub-Event`, `X-Gitlab-Event`, `X-Event-Key` or `X-Gitea-Event` header), and guessing from the payload's shape is how a `create` gets read as a `push`. A shell one-liner for a webhook receiver: ```sh jq -n --arg e "$GITHUB_EVENT_NAME" --slurpfile p body.json \ '{provider:"github", event:$e, payload:$p[0]}' > event.json ``` ## The sources this build reads | `provider` | `event` values | Its own header | | --- | --- | --- | | [`github`](/docs/triggers/github/) | `push`, `pull_request` | `X-GitHub-Event` | | [`gitlab`](/docs/triggers/gitlab/) | `push`, `tag_push`, `merge_request` | `X-Gitlab-Event` | | [`bitbucket`](/docs/triggers/bitbucket/) | `repo:push`, every `pullrequest:*` | `X-Event-Key` | | [`gitea`](/docs/triggers/gitea/) | `push`, `pull_request`, `create` | `X-Gitea-Event` | | [`senro`](/docs/triggers/manual/) | `schedule`, `manual` | none, no webhook behind it | Each page covers what that source sends, what it leaves out, and the traps worth knowing before you rely on a matcher. None of the five is privileged: all are `trigger.Provider` values dispatched through the same function [yours](/docs/triggers/custom/) is. Every one of them translates into senro's single vocabulary, so `Branches` always tests a pull request's **target** branch and `Actions` always matches the **source's own** action words, untranslated. A provider of yours may **not** claim any of those five names. One that shadowed a built-in would make the same event file mean different things in two binaries. ## When there is no file list `Paths` filters on the changed-file list the event carries. An event whose source supplied no list is an **error** from `Paths`, not a no-match: - **every Bitbucket payload**, which never carries one; - a GitHub, GitLab or Gitea **pull request** payload, none of which carries one; - a GitLab push over 20 commits, or a Gitea push whose commit list was truncated. If you fetched the list yourself, supply it through [the neutral shape's](/docs/triggers/manual/) `files` field. > The distinction is load-bearing if you [write a provider](/docs/triggers/custom/): nil means > "this source did not say", empty means "it said, and nothing changed". ## Where to go next - **[Triggers](/docs/triggers/)**: the wiring, the matchers and the three outcomes. - **[Run it as a server](/docs/triggers/server/)**: skipping the file entirely. - **[Write your own](/docs/triggers/custom/)**: a source that is not on this page. - **[Affected sets](/docs/monorepo/affected/)**: the precise narrowing `Paths` is not. --- > Source: https://xavidop.github.io/senro/docs/run/event-stream # The event stream Every observable fact about a run is one `api.Event`, appended to an ordered, append-only stream. This page covers the envelope, the event types senro sends today, and `Apply`, the function that turns the stream into something a client can render. The types live in [`github.com/xavidop/senro/api`](/docs/run/api/). Every view of a run reads this one stream: ```mermaid flowchart LR Engine["engine"] -->|"api.Event"| Stream["append-only stream"] Stream --> TUI["senro attach (TUI)"] Stream --> UI["senro ui (browser)"] Stream --> Disk["events.jsonl on disk"] Disk --> Replay["offline replay"] ``` ## The envelope ```go type Event struct { V int `json:"v"` Seq uint64 `json:"seq"` TS time.Time `json:"ts"` Type Type `json:"type"` Run string `json:"run,omitempty"` Step string `json:"step,omitempty"` // stable base ID, never "id@2" Attempt int `json:"attempt,omitempty"` // 0 when not step-scoped Group string `json:"group,omitempty"` // expansion parent, for aggregation TraceID string `json:"trace_id,omitempty"` // W3C trace, identical on every event Payload json.RawMessage `json:"payload,omitempty"` } ``` The routing fields (`Type`, `Step`, `Attempt`, `Group`) sit outside `Payload` so you can filter events without decoding the payload first. `(Event).Decode(v any)` unmarshals `Payload` into a typed struct; it's safe to call even when there's no payload, so just call it every time. ### The trace `TraceID` identifies the [W3C Trace Context](https://www.w3.org/TR/trace-context/) trace the run belongs to: 32 lowercase hex characters, the same on every event of a run, and different for every run. If the run was started by a CI job or a webhook delivery that carried a `traceparent` header, the run joins that job's trace instead of starting its own. The span details live in the payloads, not in the envelope, because they change per event: - `run.started` carries `span_id`, `parent_span_id`, `trace_flags` and `tracestate`. - `step.started`, `step.finished` and the `handler.*` events carry their own `span_id` and `parent_span_id`. The same trace context is also passed into every step's command as environment variables, so a traced tool running inside a step joins the run's trace. See [Writing a trace exporter](/docs/extend/exporter/) for how this turns into spans. ## Types sent today All thirty-four, from `api.DeclaredTypes()`: ``` run.started run.finished plan.resolved plan.expanded plan.expansion_skipped step.created step.started step.finished step.retried step.log.appended cache.hit cache.miss cache.saved cache.degraded ws.snapshot ws.restored ws.evicted binary.staged secret.resolved secret.redacted control.applied breakpoint.hit handler.started handler.succeeded handler.failed handler.superseded shell.opened shell.closed notify.delivered notify.failed notify.dropped analysis.proposed analysis.applied analysis.rejected ``` Some only show up in certain situations: - **`notify.*`**: the outcome of one outbound notification. A run with no notifier configured emits none. See [Notifications](/docs/notifications/). - **`cache.degraded`**: a **shared** cache stopped working (unreachable, bad credentials, or an unexpected response) and the run kept going anyway. It's not a failure or a miss, just slower than it should have been. See [Shared cache](/docs/data/shared-cache/). - **`breakpoint.hit`**: fires once, when the scheduler first holds back a step that has a breakpoint set on it. It's the only way to tell "stopped on purpose" apart from "hung," since a held step never gets a `step.started` or `step.finished`. Clients show it as `StepState.Paused`. See [Control operations](/docs/attach/control-ops/). - **`shell.opened` / `shell.closed`**: mark the start and end of an interactive session on a step's workspaces (`senro shell`, or the TUI's `s` key). See [The shell](/docs/attach/shell/). - **`ws.evicted`**: a [persistent workspace](/docs/data/persistent/) was cleared out because it went unused past its `MaxAge`, or grew past its `MaxSize`. The event carries the measurement and which limit it hit. - **`binary.staged`**: senro copied its own binary onto a target host so a [func step off the coordinator](/docs/executors/func-remote/) can run there. Its `reused` field tells you whether that copy was already there: on SSH, watch for `false` on every step, which means you're paying for a fresh transfer each time instead of once per host. On the container executor `reused` is always `true`, since the binary is mounted straight from the coordinator. ## Reserved names `api.Type.Known()` also recognizes a few names reserved for future use: ``` plan.generated client.attached client.detached ``` These exist so that when the matching feature ships, adding the event is a compatible change for existing clients rather than one they need to update for. `breakpoint.hit`, `shell.opened`, `shell.closed` and `binary.staged` all went through this before their features landed. > **Clients should ignore types they don't recognize.** A newer engine may emit event types this > build has never seen. Skip them instead of erroring, so your client keeps working across > versions. ## Turning events into `RunState` ```go func (s *RunState) Apply(e Event) error ``` `Apply` takes each event, one at a time, and updates `RunState` with it: the run's status, every step's state, every expansion, every handler. A renderer never has to replay the raw stream itself; it just keeps calling `Apply`. Its rules: - **A sequence number that goes backwards is an error**: `api: out-of-order event: seq 4 after 7`. - **The same sequence number twice is fine.** If a client resumes one event early, replaying an event it already applied doesn't change anything. - **A forward gap is accepted silently.** `Apply` doesn't detect missing events; check sequence numbers yourself if you need that. - **An unknown `Type` is ignored**, and so is an unknown field inside a payload. - **A malformed payload on a type `Apply` does recognize is an error**, not something it skips. The same function backs every client: the attach server's own state, the terminal UI, offline replay, and [the browser UI](/docs/attach/browser/). They all call `Apply`, so they can't disagree about what a stream means. ## Attach is a different layer This page covers the event stream: what's written to disk and turned into `RunState`. **Attach** is the live protocol a second process uses to talk to a running engine: sending commands, subscribing to events, and resuming a stream after a disconnect. See [Attach](/docs/attach/), and [Step states](/docs/steps/states/) for what a step's final `State` means. --- > Source: https://xavidop.github.io/senro/docs/attach # Attach: the protocol A `senro` pipeline is just a Go program. `senro attach` is a separate process that connects to it. It renders a terminal UI, sends control commands back, and can open a session inside a live step. This page explains how the two find each other, and what they say to each other. ## Opting in A pipeline that never calls `attach.Listen` costs nothing extra: no goroutine, no socket. To opt in, add one call before `senro.Run`: ```go att, err := attach.Listen(ctx, attach.Options{ Bind: attach.AutoUnixSocket, // /senro/.sock; see Discovery below Dir: "", // run directory; empty derives runs/ RunID: "", // empty generates one; match senro.WithRunID if you set it Pipeline: p.Name(), // shown in `senro attach`'s multi-run listing WaitForClient: false, // block until a client attaches, if true ReadOnly: false, // reject control operations and shells, if true }) if err != nil { log.Fatal(err) } defer att.Close() err = senro.Run(ctx, p, senro.WithAttach(att)) ``` `Listen` starts an HTTP server on the bound socket. It also registers the run in a small on-disk registry, so a bare `senro attach` can find it. It returns an `*attach.Attach`. Its `.Sink()` method is what you pass to `senro.WithAttach`: every event the run produces goes through it. `Options` has eight fields, and all are optional. Two aren't shown above: `TLSCertFile` and `TLSKeyFile` (see [Transport](#transport-unix-socket-or-tcp)). Here's what each field does: | Field | Notes | |---|---| | `Dir` | The run directory: the same one `senro.WithDir` sets. `GET /api/plan` and `GET /api/logs/{step}` read `plan.json` and `logs/` from here, so a live client sees the same files you'd see later with `senro attach --run `. Leave it empty and it defaults to `runs/`; `att.Dir()` tells you which was used | | `RunID` | Set it to match `senro.WithRunID`, so `senro attach --run ` finds the run again after it has finished | | `Pipeline` | Empty shows the run as `-` in a bare `senro attach`'s multi-run listing | | `WaitForClient` | Blocks `Listen` until a client subscribes. Use this to debug a pipeline that fails during its own setup, before the first step runs | | `ReadOnly` | Answers every control request with HTTP 403, and refuses a [shell](/docs/attach/shell/) the same way, before the connection is hijacked. Good for a shared dashboard: people can watch, but not drive | ## Transport: unix socket or TCP `Bind` picks the transport based on its shape. A filesystem path (or `attach.AutoUnixSocket`, or nothing at all) binds a unix socket. A `host:port` value binds TCP. Anything starting with `/`, `./` or `../` is always treated as a path. ```go attach.Listen(ctx, attach.Options{}) // unix socket, discovered automatically attach.Listen(ctx, attach.Options{Bind: "/tmp/run.sock"}) // unix socket, your path attach.Listen(ctx, attach.Options{Bind: "127.0.0.1:0"}) // TCP on a free loopback port attach.Listen(ctx, attach.Options{Bind: "0.0.0.0:8443", // TCP, reachable, over TLS TLSCertFile: "/etc/senro/tls.crt", TLSKeyFile: "/etc/senro/tls.key"}) ``` > **The two are not equivalent.** A unix socket is `0600` inside a `0700` directory, and it checks > peer credentials, so another local user can't open it at all. A TCP listener is guarded only by a > per-run bearer token, and that token can cancel the run, skip steps, and open a shell inside a > step's workspace. > > Loopback binds work without TLS. Anything else needs a certificate. There's no flag to skip > that. `att.Token()` gives you the credential, and `att.Addr()` gives you the resolved address with the real port. A bare `senro attach` works on either transport: it reads the token from the run's registry entry. If there's no local registry entry (for example, over a port-forward), pass the token in `$SENRO_ATTACH_TOKEN` and the address on a flag: `senro attach --addr 127.0.0.1:8443 --tls`. See [Security](/docs/attach/security/) for the full comparison, and for why TCP is best used for a browser on the same machine or over a port-forward. ## Discovery `Listen` writes a small JSON file, `/senro/.json`. It records the address, the transport, the run ID, the pipeline name, and the working directory. For a unix run, the socket file sits right next to it. For a TCP run, the file also carries the run's bearer token. That's why it's written `0600` inside a `0700` directory. See [Getting the token](/docs/attach/security/#getting-the-token). The runtime directory depends on the platform. `$XDG_RUNTIME_DIR` is only used on Linux: | Platform | Runtime directory | Example socket | |---|---|---| | Linux, `$XDG_RUNTIME_DIR` set | `$XDG_RUNTIME_DIR` | `/run/user/1000/senro/4711.sock` | | Linux, unset (common in a container, a `sudo` shell, or a systemd unit without `PAMName`) | `/dev/shm` | `/dev/shm/senro/4711.sock` | | Linux, unset and no `/dev/shm` | none | `Listen` fails: `neither $XDG_RUNTIME_DIR nor /dev/shm is available to resolve a runtime dir` | | macOS and everything else | `os.UserCacheDir()` | `~/Library/Caches/senro/4711.sock` | - A bare `senro attach` reads that registry, removes entries whose process has died, and attaches to the one live entry. If there's more than one, it lists them so you can pick with `--pid`. - `senro attach --run ` looks up a run by ID instead. It checks live over the socket first, then falls back to the recorded run under `runs//` on disk. See [Reading a failed run](/docs/run/debugging/) for what that directory contains. ## Snapshot, then subscribe A client attaches with two requests, in order: ``` GET /api/state the current RunState, and the seq it reflects GET /api/stream?from= everything that happened since ``` `RunState` bundles the status of every step in the run as of that instant: the engine's live equivalent of the snapshot a finished run leaves on disk. Attaching costs the same no matter how long the run has been going: there's no replay of a million old events. The snapshot carries the sequence number the subscription resumes from, so there's no race where the two disagree. ```mermaid flowchart LR client["Client"] -->|"1. GET /api/state"| engine["Engine"] engine -->|"RunState + seq"| client client -->|"2. GET /api/stream?from=seq+1"| engine engine -.->|"events, as they happen"| client ``` ## One client, two sources The client that renders a live run and the client that renders a finished one from disk are the *same code*. They run against two implementations of one interface: ```go type Source interface { State(ctx context.Context) (*api.RunState, error) Subscribe(ctx context.Context, fromSeq uint64) (<-chan api.Event, error) Logs(ctx context.Context, step string, attempt int, stream string, from int64) (io.ReadCloser, error) Control(ctx context.Context, req api.Frame) (api.Frame, error) Close() error } ``` ```mermaid flowchart LR live["Live run
(unix socket or TCP)"] -->|wire protocol| src["Source"] disk["Finished run
(runs/<id>/ on disk)"] -->|recorded events, log files| src src --> ui["One client:
TUI, plain, browser"] ``` The live source speaks the wire protocol. The disk source reads recorded events and log files directly, and answers `Control` with `ErrReadOnly` (there's no engine to send an operation to). Otherwise the two look identical to the renderer. That's what makes the handoff seamless: when the pipeline process exits while a client is attached, the client just switches sources and scrollback keeps working. A live source also satisfies a second, optional interface. That's how `senro shell` and the TUI's `s` key reach a step. The disk source doesn't implement it, so opening a shell against a finished run is refused with a clear answer instead of hanging. Use `senro ws pull` instead. ## Where to go next - **[The TUI](/docs/attach/tui/)**: the rendered client, its layout and key bindings. - **[Control operations](/docs/attach/control-ops/)**: the frame protocol and the eleven operations. - **[The shell](/docs/attach/shell/)**: what a session on a live step can and cannot do. - **[Security](/docs/attach/security/)**: the two transports and the bearer token. - **[senro run and attach](/docs/cli/run/)**: the commands, flag by flag. --- > Source: https://xavidop.github.io/senro/docs/attach/shell # The shell A step failed. You have its log and exit code, but the answer is in a file the log never printed. `senro shell` opens an interactive session inside the step's own workspaces, on the step's own executor, at the same paths the step saw, while the run is still alive: ```bash senro shell --step build ``` It's the last piece of the debugging loop: [`senro ws`](/docs/cli/workspaces/) lists a run's workspaces, writes one out, and compares two of them. This lets you stand inside one instead. ## What happens on the wire A shell session isn't a control operation like `step.retry`. `POST /api/shell` takes over ("hijacks") the plain HTTP connection and turns it into a raw, two-way byte stream instead of a JSON request and response. That's why it needs its own route: an open-ended interactive session can't be squeezed into a single [`Frame`](/docs/attach/control-ops/#frame-shape). ```mermaid flowchart LR client["senro shell"] -->|"POST /api/shell"| engine["Engine"] engine -->|"connection hijacked:
raw bytes, not JSON"| session["Session in the
step's sandbox"] session -->|"stdin / stdout / resize"| client session -.->|"shell.opened, shell.closed"| stream["Event stream"] ``` ## Pair it with a breakpoint Pairing a shell with a **breakpoint** is usually what you actually want: stop the run *before* a step runs, then look at what it was about to run against. Without the breakpoint, you're racing the step. With it, the workspace stays put for as long as you need. ```bash # in the TUI: focus the step and press b. The run stops before it and waits, # making whatever other progress it can, until you press B to release it. senro shell --step deploy ``` Or stay in the TUI entirely and press **`s`** on the focused step. The TUI releases the terminal, the run keeps going underneath, and the footer says how the session ended once you leave with `^D`. The whole loop is `b` to stop before a step, `s` to look, `B` to release it. See [Control operations](/docs/attach/control-ops/#breakpoints). ## What the session sees | | | |---|---| | **Workspaces** | Every workspace the step mounts, at the same paths, carrying whatever is in them now. | | **Working directory** | The step's own, so a bare `ls` means what it means in the step. | | **Executor** | The step's. A container step's session runs inside the same image. | | **Environment** | The step's declared environment, minus anything naming a secret. | | **Secrets** | **None.** Never. See below. | Every mount is read-only, whatever mode the step declared, and on the container and Kubernetes executors that is enforced by the kernel: ``` sh: can't create /repo/planted.txt: Read-only file system ``` Here's why: a step's workspace snapshot is taken while its sandbox is still open, so the digest in the event stream, and every cache key computed from it, already describes those exact bytes. A debugging shell must not be able to change what a run says its steps produced. > On local and SSH, read-only is intent, not an enforced restriction: neither executor reaches a > workspace through anything with a per-process mode. The same caveat applies to a step's own > read-only mounts and to handlers. ## No secrets, ever A session is delivered **no secrets**: no secret files, no `SENRO_SECRET_*` variables, and not the alias variable a step declared with `SecretEnv`. senro delivers a secret as a file and removes it once the step's sandbox closes. A shell session can stay open indefinitely, so re-delivering a cleaned-up credential would put it back on disk for as long as the window stays open, and for anyone with access to it. If a failure only reproduces with the credential present, re-run the step instead. See [Secrets](/docs/secrets/). Session output is also **not** redacted, unlike a step's. It goes straight to your terminal instead of into permanent log files, and a redactor that holds back partial matches until more bytes arrive wouldn't work for something interactive anyway. There's no secret in there to redact regardless. ## Two kinds of session By default a session runs against **pipes**, like `docker exec -i` without `-t`: no prompt, no line editing, no job control, no `Ctrl-C` as a signal. You type a line and the answer comes back. `--tty` gives you a **real terminal** instead: ```bash senro shell --step build --tty ``` You get job control, line editing, history, and `^C` delivered as a real signal to the remote command instead of killing your client. `senro shell` puts your terminal into raw mode and restores it no matter how the session ends, including on a panic, so a bad session never leaves your shell without echo. You have to explicitly ask for a terminal. senro won't upgrade you automatically. A pty is one device, so a command's stdout and stderr merge irreversibly once you're in one. Picking for you would silently cost you either job control or separate streams. ### Which executors can host one | Executor | Shell | Terminal | | --- | --- | --- | | Local | Yes | Yes | | Container | Yes | Yes | | Kubernetes | Yes | Yes | | SSH | Yes | **No** | - Asking for a terminal where none can be hosted is refused with `executor_no_terminal`: a different reason from `executor_no_shell`, since only one of these is fixed by dropping `--tty`. Nothing in this build actually refuses a shell outright, so you should never see `executor_no_shell` in practice; it exists because the capability is still checked at run time. - **SSH** can't host a terminal, because of window size. senro drives the `ssh` binary over pipes, and `ssh` normally gets its window size (and every resize) from its own stdin's terminal. Driven from pipes, it has none, so the remote pty would just report `0 0` with no way to fix it. - **Kubernetes** hosts both, but in a pod of its own: not the step's own pod, because that one has the step's secrets projected into it. The shell pod gets the step's image, with the step's workspaces staged into it read-only at the same paths, and your command runs in a container held open for the session. This costs a second pod and one more workspace transfer, and the image needs `sh` (and `tar`, if the step mounts anything). See [Kubernetes](/docs/executors/kubernetes/#senro-shell-is-a-pod-of-its-own). ### Resizing Your terminal's size travels with the request, so the remote terminal is created at the right size from the start: a pty created with no size reports `0 0`, and a full-screen program reading that draws nothing. Every later resize (`SIGWINCH`) travels on the same connection, so it's never out of sync. ### End of input A terminal has no EOF. `^D` is a byte, not a closed file descriptor. When your input ends, senro sends the `VEOF` byte, exactly what pressing `^D` would send. A command that ignores it just keeps going. For anything a prompt would have been convenient for, pass the command instead: ```bash senro shell --step build -- sh -c 'ls -la && cat go.mod' senro shell --step build -- test -f out/binary # exits with the command's own status ``` The session's exit code becomes `senro shell`'s own exit code, so the last example drops straight into a script unchanged. ## It needs a live run A session runs inside a **running** engine's workspace directories, in a sandbox that engine opens. A finished run has neither: ``` senro shell: no LIVE run named "20260812T151058-540c8ca44b": a session needs the running engine that owns the run's workspaces. If the run has finished, `senro ws pull 20260812T151058-540c8ca44b ` writes its files out instead ``` `ws pull` is the right tool once a run is over. It writes the same files out from the content-addressed store long after the process has exited. Pressing `s` in the TUI against a run tailed from disk tells you the same thing. A read-only attach server also refuses a session, before it's even opened. ## Over TCP, this is a remote shell A session works over the [TCP transport](/docs/attach/#transport-unix-socket-or-tcp), behind the same per-run bearer token as every other endpoint: ```bash export SENRO_ATTACH_TOKEN='...' senro shell --addr 127.0.0.1:8443 --tls --step build ``` Keep this in mind before binding a port: **anyone holding that token gets a command prompt inside a step's workspace, from wherever they can reach it.** - Over a unix socket the boundary is "whoever can already run code as you". Over TCP it is "whoever has the token", which over loopback includes any other user who can open the port. - Blocking just this one route while allowing the rest wouldn't help much: `step.retry` and `run.rerun_from` on the same listener already re-run a step's own command. See [Security](/docs/attach/security/) for the full comparison, and why a non-loopback bind needs TLS. ## What it looks like in the event stream Every session brackets itself with two events in the run's permanent ledger: ```json {"seq":41,"type":"shell.opened","step":"build","payload":{"session":"s1","client_id":"c2","cmd":["sh"],"workspaces":["src"]}} {"seq":57,"type":"shell.closed","step":"build","payload":{"session":"s1","client_id":"c2","exit_code":0,"duration_ns":42000000000}} ``` - Exactly one `shell.closed` follows every `shell.opened`, however the session ends: the command exiting, your connection breaking, or the run ending underneath you. A `shell.opened` with no matching close means the engine died while somebody was inside it. - Neither event carries a byte the session produced: your terminal, not the run's record. - The ledger records that a session existed, whose it was, what it ran, and how it ended. The `cmd` field tells "somebody opened a shell" apart from "somebody ran one command": useful if you're deciding whether an alert is worth waking up for. - Both event names were [reserved from the start](/docs/run/event-stream/): adding them was an additive change to the protocol. ### If your connection drops The session ends and the command inside it is killed. This isn't best-effort: the engine actively watches the connection, because the most common thing an abandoned session leaves behind is a command that never reads its input (a `tail`, a `sleep`, an editor), which would otherwise run forever with nobody watching. `shell.closed` records this as `"error":"client_disconnected"`. If a run finishes while you're still in a step, your session ends the same way, with `run_ended`. A session can't hold a run open, and a run can't end while leaving someone inside it. ## Where to go next - **[Control operations](/docs/attach/control-ops/)**: breakpoints, retry, skip and rerun-from. - **[The TUI](/docs/attach/tui/)**: the full key list, including `s`. - **[Security](/docs/attach/security/)**: what it means that this works over TCP. - **[Reading a failed run](/docs/run/debugging/)**: what to do once there is no live run. --- > Source: https://xavidop.github.io/senro/docs/attach/tui # The TUI `senro attach` (and `senro run`, on a TTY) renders an interactive terminal UI: a step list on the left, the focused step's log on the right, and a footer showing the run's status. It's a plain client of the [`Source` interface](/docs/attach/#one-client-two-sources), so it behaves the same whether it's watching a live run or replaying a finished one from disk. ## Choosing a renderer ``` --ui=auto tui on a TTY, plain streaming lines otherwise (the default) --ui=tui force the terminal UI: an error, not a silent downgrade, if stdout isn't a TTY --ui=plain line-streaming output, adding no escape sequences of its own --ui=none watch the run silently and report only the final status ``` `--ui=tui` refuses to run against a non-TTY: redirected into a file, piped into `less`, most CI logs. That's better than filling a log with garbled escape sequences nobody can read later. `--ui=plain` is built on the same `Source` client as the TUI. It isn't a separate code path, so a TTY run and a CI log never disagree about what happened. ## What `--ui=plain` prints Two kinds of line, both attributed to a step: ``` build started build stdout | go: downloading github.com/xavidop/mamori v1.12.1 build stderr | # github.com/xavidop/senro/internal/engine build failed: exit status 2 test started test stdout | ok github.com/xavidop/senro/api 0.4s test succeeded run failed ``` A step's lifecycle line reads ` `, or ` : ` when it failed. Every line the step wrote is relayed as ` | `, so you always know which step a line belongs to. Lines from different steps interleave because those steps really did run at the same time. Within one step's own stream, though, the order matches what the step produced, and its output always appears above its own lifecycle line. Plain adds no escape sequences of its own, and doesn't strip a step's either. Whatever a colorized test runner wrote is exactly what the log shows. Secret values are already gone by this point. The engine redacts them before they reach the log file, so `[REDACTED]` is what's on disk, and it's all any renderer can ever fetch. See [Secrets](/docs/secrets/). The full log bytes are on disk no matter which renderer you use, at `runs//logs///{stdout,stderr}`. `senro attach --run ` replays a finished run from those files. ## Keys | Key | Action | |---|---| | `up` / `k`, `down` / `j` | Move the selection | | `enter` | Focus the selected step | | `r` | Retry the focused step (`step.retry`) | | `R` | Rerun the focused step and everything downstream of it (`run.rerun_from`) | | `x` | Skip the focused step (`step.skip`). It, and every step that needs it, settle as `skipped_manual` | | `b` | Set a breakpoint on the focused step (`breakpoint.set`). The run stops before it | | `B` | Clear that breakpoint, releasing the step (`breakpoint.clear`) | | `w` | Snapshot the focused step's workspaces now (`ws.snapshot`), for `senro ws pull`. Answerable for a step that has not run, so pair it with `b` | | `p` | Pause the whole run (`run.pause`). Nothing new is dispatched; whatever is running finishes | | `P` | Resume it (`run.resume`) | | `s` | Open a [shell](/docs/attach/shell/) on the focused step: its workspaces, read-only, on its own executor. The TUI releases the terminal until you leave with `^D` | | `a` | Approve the [analyzer](/docs/analyzers/custom/)'s proposal for the focused step (`analysis.accept`). The engine performs the remedy it named, and records who approved it | | `A` | Reject that proposal instead (`analysis.reject`). Nothing is performed | | `c` / `Ctrl-C` | Cancel the run (`run.cancel`) | | `pgup` | Load older log history for the focused step | | `/` | Filter the step list | | `?` | Show the help overlay; `esc` closes it | | `q` | Detach. The run keeps going, and is never killed by quitting the UI | In filter mode, every other keystroke goes into the filter text, so typing `r` while filtering edits the query instead of retrying a step. `enter` applies the filter and exits; `esc` discards it and exits. `?` lists exactly the keys above, and there are no hidden or unused keys. `a` and `A` are two separate keys rather than one toggle, like `b`/`B` and `p`/`P`. Each maps to exactly one wire operation, and if the engine refuses it, the footer shows why. Pause state and breakpoints are the engine's call, not the TUI's: another client may have just changed them. A toggle based on what the TUI last saw locally could send the wrong operation, so these stay as separate keys. What goes on the wire is the proposal's id, never the step. The proposal's summary is rendered above the focused step's log, so you can read what `a` approves before you press it. `s` is the only key that takes the terminal away from the TUI. It suspends the renderer for the length of the session and redraws once you're done, so the run keeps going and the screen is up to date the moment you leave. Against a run tailed from disk, `s` refuses and tells you why: there's no engine to host a session. Use `senro ws pull` instead. A step held at a breakpoint renders as `paused`, not as a blank row, and the plain renderer prints ` paused at breakpoint`. A run stopped on purpose shouldn't look identical to one that hung. A run paused with `p` reads the same way at the run level, with the footer showing `run: paused`. ## Exit codes Whichever renderer you use, the process exits with the *run's* exit code, not the UI's: `0` for success, `1` for failure, `2` for a usage error, `130` for cancelled (`Ctrl-C`, or an external `SIGINT`/`SIGTERM`). Detaching with `q` is not a failure. The run was never asked to stop. A script that drives `senro attach` and then hits `q` won't be misread as "the run failed." See [CLI](/docs/cli/#exit-codes) for the full contract. ## Where to go next - **[Control operations](/docs/attach/control-ops/)**: what `r`, `R`, `x`, `b`/`B`, `w` and `c` send. - **[The protocol](/docs/attach/)**: the `Source` interface every renderer is built against. --- > Source: https://xavidop.github.io/senro/docs/triggers # Triggers A push to `main` should run your pipeline. A push to somebody's feature branch should not. `senro.WithTrigger` is how your pipeline binary decides for itself whether an event is its business. ## Gate a run on an event ```go ev, err := trigger.LoadEvent(*eventPath) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main"), trigger.Paths("services/**")), trigger.OnPullRequest(trigger.Actions("opened", "synchronize")), trigger.OnTag(trigger.Semver(">=1.0.0")), trigger.OnSchedule("0 3 * * *", trigger.Params{"suite": "full"}), )) if errors.Is(err, trigger.ErrNoMatch) { os.Exit(78) } ``` ```sh ./pipeline --trigger-event event.json # 0 if it matched, 78 if it did not ``` The binary is the matcher. Whatever dispatches runs hands it an event file and reads its exit code; the dispatcher decides nothing, holds no configuration and remembers nothing. A complete, runnable version is in [`examples/trigger`](https://github.com/xavidop/senro/tree/main/examples/trigger), with sample events beside it, and [The event file](/docs/triggers/events/) says what to put in one. > senro never reads `os.Args` and never exits. A library that inspects its host's argv surprises a > host with flags of its own, so your `main` parses the flag and maps the sentinel error to an exit > code. 78 is a convention, yours to choose. ## Three outcomes, never two An event nobody wanted and an event nobody wired correctly must not look the same from outside. ```mermaid flowchart LR ev["event.json"] --> bin["pipeline binary"] bin -->|"a trigger matched"| run["the run starts"] bin -->|"no trigger matched"| no["ErrNoMatch, exit 78"] bin -->|"wired wrong"| err["ordinary error, exit 1"] ``` | Outcome | What `Run` returns | Conventional exit code | | --- | --- | --- | | A trigger matched | whatever the run itself produced | 0, or 1 if the run failed | | No trigger matched | `trigger.ErrNoMatch` | 78 (`EX_CONFIG`) | | The wiring is wrong | an ordinary error | 1 | The third row is the one worth understanding, because it is what keeps the second row honest: - **No trigger matched** means the filters worked and the answer was no. This event is simply not one you asked to run for, such as a push to a branch other than `main`. `ErrNoMatch`'s message says why, one line per declared trigger — which kind it only answers, or which predicate rejected the event (`branches=[main]`, say) — so `os.Stderr` alone answers "why didn't this fire" without adding a flag or a second binary to run. - **The wiring is wrong** means senro could not work out an answer at all, and refuses to report that as a no. Two ways to land in the third row: ```go trigger.OnTag(trigger.Semver("~>1.0")) // error: Semver("~>1.0"): ... "~>1.0" is not a constraint senro can parse, so there is // nothing to compare a tag against. Write ">=1.0.0" or "^1.0.0". trigger.OnPullRequest(trigger.Paths("services/**")) // error: Paths was asked of a pull_request event that carries no changed-file list, so it can // be neither true nor false. A GitHub pull_request payload does not include one; fetching it is // a separate API call. Put the list in the event's "files" field if you have it. ``` senro could have called either one "no match" and exited 78. Then the pipeline would never run, every event would look like a clean skip, and nobody would find out until someone asked why the last release was never built. An ordinary error and exit 1 is loud on the first event instead. **A no-match is genuinely inert.** No run directory, no `events.jsonl`, no partial state. `Run` decides before it has opened the cache or started a goroutine, so being fired for every push to every branch costs a process start and nothing else. ## Declare what you run for ```go func OnPush(opts ...Option) Trigger func OnPullRequest(opts ...Option) Trigger func OnTag(opts ...Option) Trigger func OnSchedule(cron string, opts ...Option) Trigger func OnManual(opts ...Option) Trigger ``` Triggers are tried in declaration order and the first match wins. With no options, a trigger matches every event of its kind. `OnSchedule` matches the event that says "this is the 03:00 run". **senro grows no scheduler**: something outside it still starts the binary at 03:00. The cron string is compared to the event's own as text, whitespace normalised, which is what two crontab lines pointing at one binary need to select different work. senro does not parse cron, so `0 3 * * *` and `0 3 * * 0-6` are not equal. ### Matchers - **`Branches(patterns...)`**: the branch. On a pull request this is the **base** branch, not the head, the same question GitHub Actions' `branches:` filter answers. - **`Paths(patterns...)`**: the event's changed-file list. See below. - **`Actions(names...)`**: a pull request's action, by exact name. `"synchronize"` fires when an open pull request gets new commits, so you usually want it alongside `"opened"`. - **`Semver(constraint)`**: a tag that is a semantic version satisfying the constraint. - **`Params{...}`**: not a filter. Parameters a matched trigger contributes to the run. - **`Matcher{...}`**: a question of your own. See [Writing a trigger source](/docs/triggers/custom/). `Branches` and `Paths` use senro's one glob syntax, the same one workspace excludes and `Inputs` use: `*` and `?` match within a path segment, `**` spans segments, and a pattern with no `/` is the whole name. So `main` is not `feat/main`, `release/*` is `release/1.0` but not `release/1.0/hotfix`, and `feat/**` is everything under `feat/`. Asking a matcher of a kind that has no answer is an error: a tag has no branch, so `OnTag(Branches("main"))` is refused rather than left to never match. ### Semver `trigger.Semver(">=1.0.0 <2.0.0")` takes comparisons separated by spaces or commas, all of which must hold. The operators are `>=`, `<=`, `>`, `<`, `!=`, `=` and `==`; a bare version means `=`. A leading `v` is accepted; everything else follows semver 2.0.0, including no leading zeros. - **A tag that is not a version is not a match, and is never read as zero.** `release-2024`, `latest` and `""` are rejected: that is the difference between ignoring a docs tag and deploying version 0.0.0 from it. - Prereleases order the way semver says, so `1.0.0-rc.1` is **below** `1.0.0` and `Semver(">=1.0.0")` does not match a release candidate. There is no separate "exclude prereleases" rule on top. ### `Paths` is not an affected set `Paths` filters on the event's own changed-file list, supplied by the provider. It never looks at the working tree, which is what lets it run before a checkout exists. It is **not** a replacement for an [affected set](/docs/monorepo/affected/). A path filter is cheap and avoids starting a run at all, but cannot tell that a change to a shared library breaks a service containing none of the changed files. An affected set is precise and narrows a run already started. An event whose provider supplied no changed-file list is an **error**, not a no-match, per [provider](/docs/triggers/events/). ## What a match carries into the run Two things, the two an [affected set](/docs/monorepo/affected/) is computed from: - **The mode**: `all` or `affected`. A pull request is `affected`; a push to the default branch, a tag and a scheduled run are `all`; any other push is `affected`. - **The base**: what to diff against and what to diff. A push supplies its `before` and `after`; a pull request supplies its base and head commits. A tag and a schedule supply neither. `change.FromTrigger(ev)` reads these two and hands them to an expansion's `Affected`. They land in the run's manifest, not in parameters, and nothing here resolves a ref or computes a merge base: a trigger reports what the event said. The matched trigger's `Params`, laid over the event's own, become the run's [parameters](/docs/steps/conditions/), read by a condition such as `senro.ParamIs("suite", "full")`. The event's branch becomes the `branch` parameter `senro.Branch` reads. `senro.WithParams` wins over both, so you can override a trigger without editing it. ## Provenance: `runs//run.json` Every run writes a manifest beside its ledger, before its first event, saying what triggered it: ```json "trigger": { "kind": "push", "provider": "github", "repo": "acme/app", "ref": "refs/heads/main", "branch": "main", "mode": "all", "matched": "push(branches=[main], paths=[services/**])", "base": {"from": "737d38c5...", "to": "fd489864..."}, "files": 2 } ``` Read it with `senro.ReadRunManifest("runs/")`. A run nobody triggered still gets a manifest, with no `trigger` field, so a reader never has to ask whether this run happens to have one. > **The manifest carries no parameter values.** `senro.WithParams` promises that a parameter value > lands in nothing durable, and this file is durable with no redactor in front of it. `files` is a > count for the same reason, and is `-1` when the provider supplied no list. ## Through the CLI `senro run ./ci --trigger-event event.json` **forwards** the flag to the pipeline binary it builds and propagates that binary's exit code; the pipeline is the matcher. On a 78 it adds one line of explanation, because a bare exit 78 with no output reads like a crash. This assumes your pipeline accepts `--trigger-event`, the convention `examples/trigger` establishes; if yours spells it differently, pass it yourself after `--`. See [`senro run`](/docs/cli/run/). ## Wire a dispatcher A dispatcher is a lock and an exec: take a per-repository lock (`flock` will do), write the event file, start the binary, read `$?`. senro ships one in [`contrib/dispatcher`](https://github.com/xavidop/senro/tree/main/contrib/dispatcher), which is that plus an HMAC check in front: ```sh dispatcher -addr :8080 -secret-file /etc/senro/webhook-secret \ -pipeline ./ci -group ci-main ``` - A delivery arriving while another holds the group is **rejected with a reason**, not buffered: concurrency is a lock, never a queue. `-cancel-in-progress` is the other honest answer, and still not a queue, since the displaced run is gone rather than deferred. - `-namespace` makes the lock a `coordination.k8s.io` `Lease` instead of a file, so several replicas exclude each other. - It stays deliberately small, with a standing size limit, because buffering is the first feature of a CI platform and senro is not becoming one. ## Where to go next - **[Run it as a server](/docs/triggers/server/)**: your pipeline binary as the webhook endpoint, with no event file in between. - **[The event file](/docs/triggers/events/)**: the envelope, the providers and their traps. - **[Writing a trigger source](/docs/triggers/custom/)**: a provider and a matcher of your own. - **[Affected sets](/docs/monorepo/affected/)**: what consumes the mode and base a match carries. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs # Unit graphs A unit graph is the second argument to [`Expand`](/docs/monorepo/fan-out/). It tells senro what to fan out over: the list of apps, modules, crates or packages your repository already has. ```go import "github.com/xavidop/senro/unit/gowork" verify.Expand("test", gowork.Modules()). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("go", "test", "./...")).WorkDir(u.Dir) }) ``` Eight ship under `github.com/xavidop/senro/unit`. Pick the one that matches your repository. ## Choosing one | Graph | A unit is | Can narrow to affected units? | |---|---|---| | [`glob`](/docs/monorepo/unit-graphs/glob/) | Any directory matching a pattern | No | | [`gowork`](/docs/monorepo/unit-graphs/gowork/) | A Go module, or a Go package | **Yes** | | [`cargo`](/docs/monorepo/unit-graphs/cargo/) | A Rust crate | **Yes** | | [`jswork`](/docs/monorepo/unit-graphs/jswork/) | An npm, pnpm, Yarn or Bun workspace package | **Yes** | | [`maven`](/docs/monorepo/unit-graphs/maven/) | A Maven reactor project | **Yes** | | [`gradle`](/docs/monorepo/unit-graphs/gradle/) | A Gradle project | **Yes** | | [`pyproject`](/docs/monorepo/unit-graphs/pyproject/) | A Python distribution | No | | [`bazel`](/docs/monorepo/unit-graphs/bazel/) | A Bazel package | Only with `bazel.Query()` | None of these fits? [Write your own](/docs/monorepo/unit-graphs/custom/): two methods, and `Expand` cannot tell the difference. ## What "can narrow" means Every graph can list units. Only some can also answer **"which units does this change reach?"**, which is what [`Affected`](/docs/monorepo/affected/) needs: ```go verify.Expand("test", gowork.Modules()). Affected(change.FromTrigger(ev)). // only gowork, cargo, jswork, maven, gradle, bazel.Query Template(...) ``` Adding `.Affected(...)` over a graph that cannot answer is **refused at build time**, with a message naming the graphs that can: ``` senro: expansion "lint": unit: glob dirs apps/* cannot compute an affected set: it discovers units but knows nothing about which unit depends on which. Fan out over a graph that knows which unit depends on which (gowork, cargo, jswork, maven or gradle, under github.com/xavidop/senro/unit), or drop Affected and run every unit ``` Quietly running every unit would look friendlier and be wrong: an expansion that covered everything looks exactly like one that computed a real answer, and a CI that cannot tell them apart will eventually trust a green build that skipped the unit a change broke. The question every graph is answering when it declines is the same one: **can a unit in this ecosystem depend on another without saying so anywhere the graph can read?** For a glob pattern and for Python, the answer is yes. ## No toolchain required Every graph except `gowork` and `bazel.Query()` reads manifests or walks the tree, without running the ecosystem's own tool. They work on a machine with no cargo, node, mvn, gradle, JDK or bazel installed, which is usually the machine planning the run. ## What a unit is Whatever the graph, a `senro.Unit` is three fields: | Field | What it is | |---|---| | `ID` | The unit's stable identity, and what lands in the child step's id: `test[unit=apps/web]`. | | `Name` | What a tool calls the unit: a module path, a crate name, a `groupId:artifactId`. What a template passes to `--filter`, `-p` or `-pl`. | | `Dir` | The unit's directory, relative to the root. `u.Sources()` builds a `Pure()` step's inputs from it. | ## Where to go next - **[Fan-out](/docs/monorepo/fan-out/)**: `Expand`, `Template`, `MaxParallel` and `MaxNodes`. - **[Affected units](/docs/monorepo/affected/)**: narrowing a run to what a change reaches. - **[Write your own](/docs/monorepo/unit-graphs/custom/)**: a layout no shipped graph reads. --- > Source: https://xavidop.github.io/senro/docs/notifications/webhook # Webhook Posts each event to an HTTP endpoint as JSON: the `api.Event` itself, exactly as [`event.schema.json`](/docs/run/api/) describes it. A receiver needs no senro-specific knowledge, just the schema. ```go import "github.com/xavidop/senro/notify" n := notify.New( notify.Webhook("https://ci.example.com/hooks/senro", notify.Sign(os.Getenv("SENRO_HOOK_SECRET"))), ) defer func() { _ = n.Close() }() err := senro.Run(ctx, p, senro.WithSink(n)) ``` **A webhook receives every event by default.** That is the opposite of Slack's default, because a program can drop what it does not want. Narrow it when you know what you need: ```go notify.Webhook(url, notify.On(api.RunStarted, api.StepFinished, api.RunFinished)) ``` ## What a request looks like ```http POST /hooks/senro HTTP/1.1 Content-Type: application/json X-Senro-Event: step.finished X-Senro-Run: 20260807T101503-a1b2c3 X-Senro-Seq: 42 X-Senro-Delivery: 20260807T101503-a1b2c3/42 X-Senro-Timestamp: 1786000000 X-Senro-Signature: v1=9f0c... {"v":1,"seq":42,"ts":"2026-08-07T10:15:41Z","run":"20260807T101503-a1b2c3", "type":"step.finished","step":"build", "payload":{"state":"failed","exit_code":2,"duration":"12.4s"}} ``` Route on `X-Senro-Event` without parsing the body. Deduplicate on `X-Senro-Delivery`, which identifies one event permanently: [delivery is at-least-once](/docs/notifications/#what-every-request-carries). ## Verify a signed request `Sign(secret)` adds two headers, computed over the raw request body: ``` X-Senro-Timestamp: X-Senro-Signature: v1= ``` To verify, recompute the same HMAC over the exact bytes you received: ```go func verify(r *http.Request, secret []byte) ([]byte, error) { body, err := io.ReadAll(r.Body) if err != nil { return nil, err } ts := r.Header.Get("X-Senro-Timestamp") secs, err := strconv.ParseInt(ts, 10, 64) if err != nil { return nil, errors.New("no usable timestamp") } // Reject anything far from your own clock, so a captured request cannot // be replayed forever. if d := time.Since(time.Unix(secs, 0)); d > 5*time.Minute || d < -5*time.Minute { return nil, errors.New("timestamp outside the accepted window") } mac := hmac.New(sha256.New, secret) fmt.Fprintf(mac, "%s.%s", ts, body) want := "v1=" + hex.EncodeToString(mac.Sum(nil)) // Constant-time. Never ==. if !hmac.Equal([]byte(want), []byte(r.Header.Get("X-Senro-Signature"))) { return nil, errors.New("signature mismatch") } return body, nil } ``` Two things people get wrong: reading the body twice (the signature is over the **raw** bytes, so verify before decoding), and comparing with `==` instead of `hmac.Equal`. ## Sending somewhere that is not senro-shaped `Header` and `ContentType` cover an endpoint that wants its own envelope, an API key, or a different media type: ```go notify.Webhook("https://http-intake.logs.datadoghq.com/api/v2/logs", notify.Named("datadog"), notify.ContentType("application/x-ndjson"), notify.Header("DD-API-KEY", os.Getenv("DD_API_KEY")), ) ``` If the **body** has to change shape, that is a renderer, not an option. See [Write your own](/docs/notifications/custom/). > senro's own headers are set **after** yours and win. A destination may add to a request, but may > not forge the key a receiver deduplicates on. ## Options Every [option](/docs/notifications/#options) works here. The ones that matter most for a webhook: | | | |---|---| | `On(types...)` | Which events are sent. Default: all of them. | | `Sign(secret)` | HMAC-sign every request. Off by default. | | `Retry(attempts, base)` | Total requests per event, and the first backoff. Default: 3, 250ms. | | `Timeout(d)` | Bounds one request. Default: 5s. | | `Client(c)` | Your own `*http.Client`, for a proxy or a pinned CA. | ## Where to go next - **[The `api` package](/docs/run/api/)**: the schema your receiver decodes. - **[The event stream](/docs/run/event-stream/)**: every event type, and what its payload holds. - **[Notifications](/docs/notifications/)**: retries, drops, and how a failed delivery is reported. --- > Source: https://xavidop.github.io/senro/docs/data/workspaces # Workspaces One step compiles a binary and another step tests it. A `Workspace` is how the first hands its files to the second: a named, versioned directory that both steps mount. senro snapshots it into a local content-addressed store (files are keyed by the hash of their contents, so identical files are only ever stored once) when a step that mounts it finishes, and restores it for the next step that needs it. ```go src := senro.Workspace("src", senro.Scope(senro.ScopeRun)) build.Step("compile", exec.Command("go", "build", "-o", "bin/app", "./...")). WorkDir("/src"). Mount(src.At("/src", senro.RW)) // RW: this step writes into the workspace verify.Step("test", exec.Command("go", "test", "./...")). Needs("compile"). Mount(src.At("/src", senro.RO)) // RO: this step only reads ``` ## Mount one - **`ws.At(path, mode)` always needs a mode.** Use `senro.RW` for a step that writes into the workspace, and `senro.RO` for one that only reads it. If a step's declared [`Outputs`](/docs/data/caching/) land in a workspace, it needs `RW`. - **`Mount(...)` accumulates.** One step can mount several workspaces and [scratch caches](/docs/data/scratch/) at once. - **Two mounts at one path are refused.** `Build()` rejects it: `plan: step "s" mounts two things at "/src", so which one it sees is undefined`. - **How strictly `senro.RO` is enforced depends on the executor.** The container executor makes it a real read-only bind mount; the local executor detects the write afterwards instead. See [Executors](/docs/executors/). - **A handler can't mount anything of its own.** It automatically gets its parent step's workspaces, read-only, at the same paths. Calling `Mount` on a handler is refused. See [Handlers](/docs/steps/handlers/). ## Scopes | Scope | What you get | | --- | --- | | `senro.ScopeRun` | The default. One directory for this run, shared by every step that mounts it, gone when the run ends | | `senro.ScopePersistent` | One directory on this machine that outlives the run. Requires bounds; see [Persistent workspaces](/docs/data/persistent/) | | `senro.ScopeStep` | One fresh directory **per step**, shared with nobody and discarded with the run. Nothing is snapshotted from one, because no later step reads it | senro also refuses `MaxAge` or `MaxSize` on any scope other than `ScopePersistent`, since nothing would ever use them. ## Snapshots senro snapshots a workspace when a step that mounts it finishes. That snapshot's digest is what a later step, a cache key, and `senro ws` all use. ```mermaid flowchart LR F["step finishes"] --> S["snapshot"] --> D["digest"] D --> N["next step's mount"] D --> K["cache key"] D --> C["senro ws ls / pull / diff"] ``` - **`NoSnapshot()`** on a step skips this snapshot. Use it when nothing downstream needs the step's filesystem output. - **`senro.Exclude(patterns ...string)`** is a `Workspace` option that keeps matching paths out of the snapshot, in addition to the default excludes (`.git` and `node_modules`). For example: `senro.Exclude("**/*.log", "tmp/")`. - **`senro.PreserveSymlinks()`** is a `Workspace` option that keeps real `node_modules` directories in the snapshot too, not just symlinks pointing at them. Use it when `node_modules` is itself a tree of symlinks into a store, as with pnpm. Most workspaces don't need it. - **A snapshot stores the executable bit and nothing else.** File modes are normalized, mtimes are fixed, and senro doesn't store uid, gid, extended attributes, ACLs, hard links, devices, sockets, or fifos. This means a snapshot's digest depends only on file content, not on which machine produced it. See [`senro ws pull`](/docs/cli/workspaces/). - **Excludes apply to transfers too.** On the [Kubernetes](/docs/executors/kubernetes/) and [SSH](/docs/executors/ssh/) executors, an excluded path is never sent to the remote target, so it can't come back either. - **You can force a snapshot mid-run.** The [`ws.snapshot`](/docs/attach/control-ops/#forcing-a-snapshot) control operation (the TUI's `w` key) captures a step's workspaces on demand, useful when you've paused at a breakpoint. The event this creates is marked `"forced": true`. The digest is real and `ws pull` works on it, but it never enters a cache key, and `ws ls`, `ws pull`, and `ws diff` skip it. That keeps their reports matching what the run actually produced. ## Pattern syntax `*` and `?` match within one path segment. `**` matches across segments. senro uses this same matcher everywhere: `Exclude`, `artifact.Glob`, and fan-out unit globs. Two rules are easy to miss: - **A pattern with no `/` matches the whole relative path**, not any depth. This is narrower than `.gitignore`: `**/*.log` matches `sub/a.log`, but `*.log` does not. - **A trailing `/` (meaning "this directory and everything under it") only works in `Exclude`.** `artifact.Glob` and unit globs have no directory form, so `tmp/` matches nothing there. Negation (`!pattern`) isn't supported anywhere. In a `.senroignore` file, a line starting with `!` is rejected outright, naming the file and line number. Everywhere else, a leading `!` is just treated as a literal character. So `senro.Exclude("!vendor")` excludes a path that starts with the character `!`, not "everything except vendor". ## Look at what a run produced ```sh senro ws ls # every workspace the latest run declared senro ws pull RUN src /tmp/broken # the files a failed step left behind senro ws diff RUN-A RUN-B src # what changed between two runs ``` `ws ls` and `ws diff` read stored indexes and never pull the actual files, so both are instant even on a multi-gigabyte tree. Use `ws pull` when a workspace's last state came from a cache hit, since a cache entry stores only a body digest, not an index. See [Workspace commands](/docs/cli/workspaces/). ## Where to go next - **[Persistent workspaces](/docs/data/persistent/)**: a tree that survives between runs. - **[Scratch caches](/docs/data/scratch/)**: the best-effort, key-restored sibling. - **[Caching a step](/docs/data/caching/)**: how a workspace's content reaches a cache key. - **[Concepts](/docs/concepts/)**: why everything is content-addressed. --- > Source: https://xavidop.github.io/senro/docs/notifications/custom # Write a destination A destination is a URL plus one method that turns an event into a request body. Everything else (the queue, retries, signing, headers, outcome events, the end-of-run flush) you inherit. Reach for this when you want a run's result somewhere senro does not ship: PagerDuty, Datadog, Discord, an internal incident bus. ```go type Renderer interface { Render(api.Event) ([]byte, error) } func To(rawURL string, r Renderer, opts ...DestinationOption) *Destination ``` ## Build one in three steps ### 1. Render the body `notify.RendererFunc` adapts a plain function, which is enough when you have no state to carry: ```go d := notify.To("https://example.com/incidents", notify.RendererFunc(func(e api.Event) ([]byte, error) { var b api.RunFinishedBody if err := e.Decode(&b); err != nil { return nil, fmt.Errorf("reading the run.finished payload: %w", err) } return json.Marshal(map[string]string{ "run": e.Run, "status": string(b.Status), }) }), notify.Named("incidents"), notify.On(api.RunFinished), ) ``` > **Always set `Named`.** Unnamed destinations are all called `destination`, in the ledger and in > the shutdown report, so two of them are two lines nobody can tell apart. ### 2. Filter with `On`, not inside `Render` Filtering happens **before** the queue. A destination that only wants `run.finished` must not have its queue filled with events it is going to discard, or the one event it exists for is the one that gets dropped. ```go notify.On(api.RunFinished) // yes // if e.Type != api.RunFinished { return nil, nil } // no ``` ### 3. Wire it in ```go n := notify.New(notify.Slack(slackURL), d) defer func() { _ = n.Close() }() err := senro.Run(ctx, pipeline, senro.WithSink(n)) ``` ## Ship it as a package If other people will use your destination, give them a constructor so it is one line: ```go package pagerduty const EventsAPI = "https://events.pagerduty.com/v2/enqueue" func Destination(routingKey, source string, opts ...notify.DestinationOption) *notify.Destination { defaults := []notify.DestinationOption{ notify.Named("pagerduty"), notify.On(api.RunFinished), } return notify.To(EventsAPI, Renderer{RoutingKey: routingKey, Source: source}, append(defaults, opts...)...) } ``` **Put your defaults first and the caller's options after.** `append(defaults, opts...)` means the last option wins, so a caller's `notify.Retry(5, time.Second)` overrides yours without you having to decide field by field what they may change. Every built-in destination is written this way. ## The rules ### What you must do | | | |---|---| | **Return eventually** | A `Render` that blocks forever gets a short window after the shutdown grace expires, then is abandoned and named in the shutdown report so the process can exit. | | **Do not retain or mutate the event** | The same `api.Event` goes to every other observer. | | **Be deterministic for one event** | The body is rendered once and reused across retries, so a renderer that counts calls is counting events, not requests. | | **Filter with `On`** | See above. | ### What you get for free | | | |---|---| | **A queue** | Bounded, per destination, on its own goroutine. `Render` runs there, one event at a time, never on the goroutine running the build. Being slow delays only this destination; a full queue drops rather than waits, counted as `notify.dropped`. | | **Retries** | Up to `Retry(attempts, base)` requests per event, doubling and jittered, retrying only what is worth retrying (no answer, a 429, any 5xx) and never a 400 that will be a 400 again. | | **Timeouts** | `Timeout(d)` bounds one request; the notifier's grace bounds the whole shutdown. | | **Signing and headers** | `Sign(secret)` HMAC-signs the exact bytes you rendered, and `X-Senro-Event`, `X-Senro-Run`, `X-Senro-Seq` and `X-Senro-Delivery` go on every request. senro's headers are set after yours and win. | | **Outcome events** | Every delivery becomes `notify.delivered`, `notify.failed` or `notify.dropped` in the run's ledger, naming your destination. | | **The flush** | `senro.Run` flushes before it returns, so `run.finished` is on its way out before your process is. | | **Redaction** | The engine redacts every event payload before the event exists, upstream of every sink. Your renderer has nothing left to do about it. See [Secrets](/docs/secrets/). | Every [option](/docs/notifications/#options) applies to a destination built with `To`, exactly as it does to `Webhook` and `Slack`. ### What happens on error An error from `Render` means **no request for this event**. It is recorded as `notify.failed` with your error's text, and not retried: a body that would not render will not render differently a second later. A panic is caught and becomes the same failed delivery. It does not fail the run and does not stop the destination, which delivers the next event normally. It is still a bug. ## When the URL changes per event: `Requester` A destination that **creates** a resource on one event and **updates** it on the next does not fit `Renderer`, which renders a body for a fixed URL. Implement `notify.Requester` instead: ```go type Request struct { Method string // defaults to POST when empty URL string // required Body []byte // may be nil for a method that has none } type Requester interface { Request(api.Event) (*Request, error) } notify.To("", nil, notify.WithRequester(myRequester), notify.Named("mydest")) ``` - A destination with a Requester uses it **instead of** its Renderer; its configured URL becomes a base the Requester may use or ignore. - Returning a nil `*Request` with a nil error means "nothing to send for this event", recorded as neither delivered nor failed. That is how a Requester filters. - `Request` has no headers field: headers are declared once with `Header`, and senro's envelope headers still win. - It is called on the delivery goroutine, one event at a time per destination, so you may keep state without a lock. It must not block for long, or the bounded queue behind it drops. - A Requester that needs to **read the response** (the id of the resource it just created) implements `notify.ResponseReader`, called only on a 2xx. [GitHub Checks](/docs/notifications/github-checks/) is the built-in worked example: a check run POSTed once and PATCHed thereafter, with the id learned from the create's response (`notify/githubchecks.go`). ## Your credential is yours A routing key in a request body and an API key passed to `Header` go through **no redactor**, which only ever sees event payloads. senro redacts the run's secrets out of the events you render from, and takes no responsibility for a credential your renderer holds. Keep it out of the strings you put in the body. The destination URL is treated as a credential regardless, since for a Slack incoming webhook it is the whole of one: `notify` strips it out of every error it records or prints, for your destination as much as for its own. ## Testing Stand up an `httptest.Server`, point the destination at it, and assert on the bytes that arrived. The one thing a test substitutes is the URL, which is why `To` takes it: ```go func TestBodyShape(t *testing.T) { var got []byte srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { got, _ = io.ReadAll(r.Body) })) defer srv.Close() n := notify.New(notify.To(srv.URL, Renderer{}, notify.On(api.RunFinished))) // ... emit an event, then n.Flush(ctx) ... if !strings.Contains(string(got), `"status":"failed"`) { t.Errorf("body %s", got) } } ``` For the outcome events as well as the request, hand the notifier an appender with `n.SetAppender(...)`, which is what `senro.Run` does. ## The worked example [`examples/extensions/pagerduty`](https://github.com/xavidop/senro/tree/main/examples/extensions/pagerduty) is a full destination for PagerDuty's Events API v2, commented and driven end to end by senro's own tests. It maps a run's status to a `trigger` or `resolve` action, and uses the run's own ID as the PagerDuty dedup key, so an at-least-once delivery is one incident rather than several, and a later resolve closes the one it opened. ## Where to go next - **[Notifications](/docs/notifications/)**: the options, the headers, the failure reporting. - **[The event stream](/docs/run/event-stream/)**: what an `api.Event` contains, which is what you are rendering. - **[Writing a trace exporter](/docs/extend/exporter/)**: the same sink, folded into spans instead. --- > Source: https://xavidop.github.io/senro/docs/triggers/custom # Write a trigger source A `trigger.Provider` turns one source's own payload into the neutral `trigger.Event` every matcher reads. Write one when you want to run on something the [five built-in sources](/docs/triggers/events/#the-sources-this-build-reads) do not parse: an internal event bus, a webhook senro has never seen, or one more event from a source it already reads. **Every matcher senro ships reads only the neutral `Event`**, so `Branches`, `Paths`, `Actions` and `Semver` work on your source's events without knowing your source exists. That is the payoff: you write a parser, and you get the whole matcher vocabulary for free. ```go type Provider interface { Name() string Parse(event string, payload []byte) (*Event, error) } ``` ## You may not need one `trigger.Event` is an ordinary struct with exported fields. If your dispatcher already has the facts in its own form, build one and hand it straight to `senro.WithTrigger`: ```go ev := &trigger.Event{Kind: trigger.Push, Repo: "acme/app", Ref: "refs/heads/main", Branch: "main"} err := senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")))) ``` A `Provider` is for **the file envelope**: what a dispatcher writes to disk and what `senro run --trigger-event` forwards. If you have no file, you need no provider. Simpler still, for facts you already hold: write [the neutral `senro` shape](/docs/triggers/manual/) as JSON and skip the Go entirely. ## Build one in three steps ### 1. Name it `Name` is the value the envelope's `provider` field must carry to reach you: ```json {"provider": "deploy-bus", "event": "release", "payload": { ...your body... }} ``` ```go type Provider struct{} func (Provider) Name() string { return "deploy-bus" } ``` The five built-in names (`github`, `gitlab`, `bitbucket`, `gitea`, `senro`) are refused, as are two providers sharing a name. ### 2. Parse into the neutral event ```go func (Provider) Parse(event string, payload []byte) (*trigger.Event, error) { if event != "release" { return nil, fmt.Errorf("deploy-bus reads a release event, not %q", event) } var p struct { Repo string `json:"repo"` Tag string `json:"tag"` SHA string `json:"sha"` By string `json:"requested_by"` } if err := json.Unmarshal(payload, &p); err != nil { return nil, fmt.Errorf("the release payload: %w", err) } if p.Tag == "" { return nil, errors.New("the release payload names no tag, " + "which is what a Semver matcher tests") } return &trigger.Event{ Kind: trigger.Tag, Repo: p.Repo, Ref: "refs/tags/" + p.Tag, Tag: p.Tag, Files: nil, // nil, not empty: this source says nothing about changed files Base: trigger.Base{To: p.SHA}, Params: map[string]string{"author": p.By}, }, nil } ``` ### 3. Pass it to `LoadEvent` ```go ev, err := trigger.LoadEvent(*eventPath, deploybus.Provider{}) if err != nil { return err } err = senro.Run(ctx, pipeline, senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main")), trigger.OnTag(trigger.Semver(">=1.0.0")), // now matches your bus's events )) ``` `trigger.OnTag(trigger.Semver(">=1.0.0"))` matches events from your bus, unchanged. You wrote no matcher. ## Filling in the event | Field | What to put in it | |---|---| | `Kind` | **Required.** One of `Push`, `PullRequest`, `Tag`, `Schedule`, `Manual`: the kinds a trigger can be declared for. | | `Repo`, `Ref`, `Branch`, `Tag` | Whatever your source says. `Branches` reads `Branch`, `Semver` reads `Tag`. | | `Files` | The changed-file list, or **nil** if your source does not carry one. | | `Base` | `{From, To}`: what to diff against and what to diff. An [affected set](/docs/monorepo/affected/) reads it. | | `Params` | Everything senro has no field for: a username, a label, a project ID. | Three rules decide whether your provider is trustworthy: - **Translate into senro's vocabulary.** A GitLab merge request is senro's `PullRequest`; a tag push is `Tag`, even when your source delivers it as a push to `refs/tags/...`. This is exactly what makes the built-in matchers work on your events. - **`Files` nil and `Files` empty mean different things.** Nil is "this source did not say", and `Paths` against it is an error. Empty and non-nil is "it said, and nothing changed". If your payload carries no file list, leave it nil. - **Everything that goes wrong is an error.** An unparseable payload, an event name you do not handle, a missing required field: all errors. Never a nil `Event` with a nil error, and never a silent no-match. ## What you get back - **Declaration order and first-match-wins**, across triggers mixing built-in and custom matchers. - **Every declaration is checked before any of it is matched**, so a mistake in the third trigger is reported even when the first would have matched. - **The mode and the base** are computed from your `Event`, so a monorepo [affected set](/docs/monorepo/affected/) works on your source's events unchanged. - **Parameters and provenance**: the matched trigger's `Params` become the run's parameters, and `runs//run.json` records which trigger matched and how it was declared. - **[Three outcomes](/docs/triggers/#three-outcomes-never-two)**, preserved through your provider. Your error is reported as an ordinary error naming your provider, which is **exit 1** rather than the **78** a no-match gets. A provider that returns nothing at all, an empty or unknown `Kind`, or a panic becomes the same thing. senro fills in `Event.Provider` from your `Name` if you leave it empty. ## Asking a question senro has no matcher for `Branches`, `Paths`, `Actions` and `Semver` ask about fields the neutral `Event` has. For something only your source carries, write a `trigger.Matcher`: ```go func Author(usernames ...string) trigger.Option { return trigger.Matcher{ Name: "author", Args: usernames, Kinds: []trigger.Kind{trigger.Push, trigger.PullRequest, trigger.Tag}, Match: func(ev *trigger.Event) (bool, error) { who, ok := ev.Params["author"] if !ok { // Neither true nor false: this event does not say. return false, fmt.Errorf("Author%v was asked of a %s event "+ "that carries no author", usernames, ev.Kind) } return slices.Contains(usernames, who), nil }, } } ``` ```go trigger.OnPush(trigger.Branches("main"), deploybus.Author("dependabot")) ``` | Field | What it is | |---|---| | `Name`, `Args` | What the run's provenance record and every error message show. The trigger above renders as `push(branches=[main], author=[dependabot])`. They land in `runs//run.json` and in CI log text, neither of which the redactor sits in front of, so **do not put a credential in one**. | | `Kinds` | The kinds this question has an answer for. Declaring it on any other kind is an error, the same treatment `OnTag(Branches("main"))` gets. Leave it empty for a question every event can answer, such as one about `Params`. | | `Match` | `true`, `false`, or an error meaning "this event carries nothing to answer with". | **Getting that last distinction right is most of what makes a matcher trustworthy**: "nobody asked for anything" is a plain `false`, "this event does not say who" is an error. `Match` must not block. It runs before any run exists, so a trigger is a cheap filter on what the event already says, never a question about the working tree or the network. ## Layering on a built-in The most useful shape is usually not a competing parser, but **one more event on top of a source senro already reads**: ```go func (p Provider) Parse(event string, payload []byte) (*trigger.Event, error) { if event == "note" { return p.parseComment(payload) // the event GitLab's built-in skips } return trigger.GitLab().Parse(event, payload) // everything else, unchanged } ``` `trigger.GitHub()`, `trigger.GitLab()`, `trigger.Bitbucket()` and `trigger.Gitea()` are all exported for this. ## Testing Test through `ReadEvent`, so the envelope is exercised too, then assert on what the matchers make of the result. **Use real captured payloads as fixtures**, not ones written from a field list you remember. That cross-check is what caught the example below reading `user_username` on a payload carrying `user.username`. Test the error cases too, and assert they are **not** `trigger.ErrNoMatch`. ## The worked example [`examples/extensions/gitlabcomment`](https://github.com/xavidop/senro/tree/main/examples/extensions/gitlabcomment) is a provider for a comment on a GitLab merge request, so `/retest` in a comment is a reason to build. Its `Parse` handles the comment and delegates everything else to `trigger.GitLab()`, so triggers for pushes and merge requests keep working with no code of its own. ## Where to go next - **[Triggers](/docs/triggers/)**: the wiring, every matcher, and what a match carries into the run. - **[Run it as a server](/docs/triggers/server/)**: reaching your provider from an HTTP handler, with `trigger.As`. - **[The event file](/docs/triggers/events/)**: the envelope your provider is reached through. - **[Write a destination](/docs/notifications/custom/)**: the other end, for telling somebody how the run went. --- > Source: https://xavidop.github.io/senro/docs/monorepo/unit-graphs/custom # Write a unit graph A unit graph tells [`Expand`](/docs/monorepo/fan-out/) what to fan out over: a list of units, each with an id, a name and a directory. Write one when your repository's layout is described by a file no [shipped graph](/docs/monorepo/unit-graphs/) reads, such as a `components.json` only you have. ```go // Every graph implements this. type UnitGraph interface { Units(ctx context.Context, root string) ([]Unit, error) Describe() string } ``` That is the whole requirement. A second, optional interface adds the ability to narrow a run to an [affected set](/docs/monorepo/affected/); [see below](#adding-affected-set-support). ## Build one in two steps ### 1. Return the units ```go type ComponentGraph struct{ File string } func (g ComponentGraph) Describe() string { return "components in " + g.File } func (g ComponentGraph) Units(ctx context.Context, root string) ([]senro.Unit, error) { if err := ctx.Err(); err != nil { return nil, err } body, err := os.ReadFile(filepath.Join(root, g.File)) if err != nil { return nil, fmt.Errorf("%s: %w", g.Describe(), err) } var doc struct { Components []struct { Name string `json:"name"` Dir string `json:"dir"` } `json:"components"` } if err := json.Unmarshal(body, &doc); err != nil { return nil, fmt.Errorf("%s: %w", g.Describe(), err) } if len(doc.Components) == 0 { return nil, fmt.Errorf("%s: declares no components", g.Describe()) } out := make([]senro.Unit, 0, len(doc.Components)) for _, c := range doc.Components { out = append(out, senro.Unit{ID: c.Dir, Name: c.Name, Dir: c.Dir}) } sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID }) return out, nil } ``` ### 2. Fan out over it Exactly like a shipped graph: ```go verify.Expand("test", ComponentGraph{File: "components.json"}). Template(func(u senro.Unit) *senro.StepBuilder { return senro.NewStep(exec.Command("make", "test")).WorkDir(u.Dir) }) ``` That is a complete, working graph. It cannot narrow a fan-out to an affected set, and it says so rather than guessing: `.Affected(...)` over it is refused at build time. ## Filling in a `Unit` | Field | What it is | |---|---| | `ID` | The unit's stable identity, and what lands in the child step's id: `test[unit=apps/web]`. **It must not contain any of `[]=,@`**; the step-id grammar has no escape for them. | | `Name` | What a tool calls the unit: a module path, a crate name, a `groupId:artifactId`. What a template passes to `--filter`, `-p` or `-pl`. | | `Dir` | The unit's directory, relative to the root, forward slashes on every platform. `u.Sources()` builds a `Pure()` step's input selector from it. | `ID` and `Dir` are usually equal but need not be. Every shipped graph uses a slash-separated path relative to the root, because an id has to be stable across runs and machines. ## Four rules **Return a deterministic order.** Child step ids derive from the unit set in the order you returned it, so the order decides the plan and the plan digest every cache entry hangs off. A graph that ranged over a map gives the same pipeline a different identity on every build. Sort before returning. **Write a `Describe` that names the thing to look at.** One short phrase, present tense: `"gowork modules"`, `"components in components.json"`. It is what a person reads in a plan and what senro builds errors out of, so `"my graph"` is useless and the second one says which file to open. **Honour the context.** A graph that walks a large tree or shells out to a toolchain is doing the work somebody presses Ctrl-C on: check `ctx.Err()` before starting and again inside a walk. A graph reading one small file checks once at the top. **Refuse rather than shrink.** Treat an unreadable manifest as an error, never as a smaller graph. A manifest half-read is a missing edge, a missing edge is a unit left out of an affected set, and that is a green build for a broken tree. A graph that found nothing at all should error too: an expansion that silently produced no steps looks, in a log, exactly like one that passed. senro calls `Units` **once per expansion at build time**, before any step runs. ## Adding affected-set support To let `.Affected(...)` work over your graph, implement two more methods: ```go type UnitAffector interface { UnitGraph Owns(ctx context.Context, root string, files []string) ([][]string, error) ReverseDeps(ctx context.Context, root string) (map[string][]string, error) } ``` **Only implement this if you can answer both questions honestly.** > A wrong affected set is worse than no affected set. The failure modes are not symmetric. An unneeded unit costs CI minutes. Skipping a unit the change broke reports a green build for a broken tree, which is the failure that makes a team turn the feature off for good. Ask yourself: **can a unit in my repository depend on another without saying so anywhere I can read?** If yes, implement `UnitGraph` alone and say why in your package doc. Declining is first-class, and three shipped graphs decline: [`glob`](/docs/monorepo/unit-graphs/glob/), [`pyproject`](/docs/monorepo/unit-graphs/pyproject/) and [`bazel.Packages()`](/docs/monorepo/unit-graphs/bazel/). ### `Owns`: what a changed file belongs to - The result is **parallel** to `files`: element `i` holds the ids of the units owning `files[i]`. - An **empty** element means no unit owns the file, which senro turns into a run of every unit. That is the answer whenever you are unsure, and it is not a failure: a `Makefile` above every unit genuinely can change what all of them build. - One file may belong to several units. `gowork` attributes a `go.mod` to every package of its module, because a dependency bump changes what all of them compile against. - **Never stat the paths.** A deleted file must be answered from its path alone: it is gone from disk by plan time, and a deletion is exactly the change whose dependents most need rebuilding. ### `ReverseDeps`: who breaks when a unit changes - Keyed and valued by `Unit.ID`: `ReverseDeps[X]` is who depends on **X**, not what X depends on. - **Direct edges only.** senro computes the transitive closure itself, once, and marks on push so a cycle terminates rather than hanging a build. - Values must be sorted, for the same reason `Units` must be. Then add `.Affected(...)` to the expansion: ```go verify.Expand("test", ComponentGraph{File: "components.json"}). Affected(change.FromTrigger(ev)). Template(...) ``` A graph that is not a `UnitAffector` gets `senro.ErrNoAffectedSet` from `Affected` at build time, wrapped so `errors.Is` can tell "this graph cannot narrow" apart from "narrowing failed". ## The worked example [`examples/customgraph`](https://github.com/xavidop/senro/tree/main/examples/customgraph) is the graph above grown into a full `UnitAffector`, with `Owns` resolving to the nearest declaring directory and `ReverseDeps` built from a `needs` field. It compiles and is tested on every commit. Two of its tests are worth more than the rest together: - **The transitive case.** A changes, B depends on A, C depends on B, and all three must run. The most common bug is a closure that only goes one hop. - **A file nothing owns must run everything**, which is the over-approximation working. > Build fixtures out of real manifests, not plausible-looking ones. A wrong field name is a bug no > test will catch, because your fixture and your parser agree with each other and disagree with > the tool. ## Where to go next - **[Unit graphs](/docs/monorepo/unit-graphs/)**: the eight that ship, in case one already fits. - **[Fan-out](/docs/monorepo/fan-out/)**: `Expand`, `Template`, `MaxParallel` and `MaxNodes`. - **[Affected units](/docs/monorepo/affected/)**: what `Owns` and `ReverseDeps` feed. --- > Source: https://xavidop.github.io/senro/docs/analyzers/custom # Write an analyzer One method. senro hands you a failed step, you hand back a sentence. ```go type Analyzer interface { Analyze(context.Context, api.Failure) (api.Proposal, error) } ``` Read [Failure analyzers](/docs/analyzers/) first if you have not: it covers what an analyzer is for and who decides whether its suggestion is acted on. ## Build one in three steps ### 1. Write the type Any type with that one method is an `Analyzer`. There is nothing to register and nothing to embed. ```go package runbook import ( "context" "strings" "github.com/xavidop/senro/api" ) type Analyzer struct{} func (Analyzer) Analyze(ctx context.Context, f api.Failure) (api.Proposal, error) { return api.Proposal{}, nil // no Summary means "no comment" } ``` `github.com/xavidop/senro/api` is the only import you need. ### 2. Answer the failure `f` is everything senro knows about the failed step. The four fields you will reach for most: ```go f.Step // "fetch" f.Cmd // []string{"curl", "-fsSL", "https://registry.example.com/pkg.tar.gz"} f.ExitCode // 7 f.LogTail // the last few KB the step printed, already redacted ``` Match on whatever tells you the most, and return a `Proposal`: ```go func (Analyzer) Analyze(ctx context.Context, f api.Failure) (api.Proposal, error) { switch { case strings.Contains(f.LogTail, "i/o timeout"): return api.Proposal{ Summary: f.Step + " failed on the network, not on its own work", Detail: "The request never got an answer, so the command never reached a verdict.", Remedy: api.RemedyRetry, }, nil case strings.Contains(f.LogTail, "no space left on device"): return api.Proposal{ Summary: f.Step + " filled the disk", Detail: "Clear the runner's build cache. Retrying fails identically.", // No Remedy: a retry cannot fix a full disk. }, nil } return api.Proposal{}, nil } ``` A real analyzer replaces the switch with one call to whatever answers best for you: a model, a log classifier, an internal incident API. The shape of the function does not change. ### 3. Wire it in ```go err := senro.Run(ctx, p, senro.WithAnalyzer(runbook.Analyzer{}, senro.AnalyzerName("runbook"))) ``` `AnalyzerName` is the name that appears in the ledger and in the TUI, so name the analyzer, not the model behind it. `WithAnalyzer` takes exactly one analyzer: to consult two sources, write one `Analyze` that asks both and returns a single answer. ## What you are handed: `api.Failure` ```go type Failure struct { RunID, Pipeline, Step string Attempt int State State // failed, timed_out or panicked ExitCode int Error string Duration time.Duration Cmd, Needs []string LogTail string } ``` Two things worth knowing before you build a prompt or a request out of it: - **It is already redacted.** `LogTail` comes from downstream of the same redactor that writes the log file; every other field is plan data. There is no second redactor after you, so whatever you put in a request is what leaves the machine. - **`LogTail` is bounded** to the last few kilobytes, so a step that printed a gigabyte cannot become a gigabyte-sized request body. There is no handle on `Failure` for reading more: no workspace, no file system, no run. What is on the struct is what an analyzer gets to reason about. ## What you return: `api.Proposal` ```go type Proposal struct { Summary string // one line, the thing a person reads first Detail string // the reasoning, as long as it needs to be Remedy Remedy // at most one action to offer } ``` | Field | Rule | |---|---| | `Summary` | **Required.** A proposal without one is discarded, and no event is appended. That makes "no comment" a real answer that costs the run nothing. | | `Detail` | Free text. senro never parses it, so this is where advice like "apply this patch" or "bump the timeout" belongs. | | `Remedy` | Either `api.RemedyRetry` or nothing. Anything else counts as nothing, with no trimming and no case folding: `"RETRY "` and `"patch"` are both no remedy. | ### The remedy vocabulary is one word long ```go const ( RemedyNone Remedy = "" // explain, ask for nothing RemedyRetry Remedy = "retry" // run the failed step again ) ``` Retrying is the only action a proposal can ask for, because it is the only one you could already do by hand in the TUI. Editing a workspace file, rewriting a command or injecting an environment variable are not in the vocabulary and never will be. Say those things in `Detail` instead. A person applying advice they read is a different thing from a program applying text it was sent. ### When to return a remedy, and when not to This is the judgement call that decides whether an analyzer is worth having. ```go // Worth retrying: the command never reached a verdict. case strings.Contains(f.LogTail, "connection reset by peer"): return api.Proposal{Summary: "the registry dropped the connection", Remedy: api.RemedyRetry}, nil // Worth retrying: the process never started. case f.Error != "" && f.ExitCode == 0: return api.Proposal{Summary: "the step's process never started", Remedy: api.RemedyRetry}, nil // NOT worth retrying: the workload gave its verdict, and it was "no". case f.ExitCode == 1 && strings.Contains(f.LogTail, "--- FAIL:"): return api.Proposal{ Summary: "a test failed: " + firstFailingTest(f.LogTail), Detail: "Retrying deletes the information the test just gave you.", }, nil ``` An analyzer that proposes a retry for everything is worse than no analyzer: it turns a policy into a loop that hides real failures. Being able to say "a person needs to look at this" is the point. ## The rules, in one table | | | |---|---| | **Return quickly** | One call is bounded by `senro.AnalyzeTimeout` (30s). Past it, the call is cancelled and counted. | | **Errors are fine** | An error means "no comment": no proposal, the run's own error unchanged, your error named in the shutdown report. A run does not fail because somebody's API was down. | | **Panics are fine too** | Recovered into an ordinary error. Still a bug in your analyzer. | | **You are called once per failed step** | Not per attempt, and never for a step skipped because something upstream failed. | | **Being slow is free** | `Analyze` never runs on the engine's goroutine. If the queue behind it is full the offer is dropped and counted, and no scheduling decision waits on you. | A proposal that arrives after the run's event stream has closed is printed on standard error rather than lost, which is also where a failed step your analyzer never answered about is named: ``` senro analyze: 1 proposal arrived after this run's event stream closed, so it is reported here instead of in the ledger: runbook: deploy attempt 2: the cluster rejected the manifest's apiVersion ``` Redirect that with `senro.AnalyzeReportWriter`. ## Test it without an engine An analyzer is a pure function of a struct. Feed it a `Failure`, assert on the `Proposal`. No engine, no network, no key: ```go func TestDiskFull(t *testing.T) { got, err := Analyzer{}.Analyze(context.Background(), api.Failure{ Step: "audit", ExitCode: 1, LogTail: "cp: error writing '/out/x': no space left on device\n", }) if err != nil { t.Fatal(err) } if got.Remedy != api.RemedyNone { t.Errorf("a full disk must not propose a retry, got %q", got.Remedy) } if !strings.Contains(got.Summary, "disk") { t.Errorf("summary %q says nothing about the disk", got.Summary) } } ``` ## The worked example [`examples/extensions/fakeanalyzer`](https://github.com/xavidop/senro/tree/main/examples/extensions/fakeanalyzer) is the analyzer on this page in full. Its "provider" is a switch statement, so it runs with no network and no key while showing identical wiring. `go run ./examples/analyze` explains two failures and applies nothing, because nobody approved anything. With `-auto`, a policy is configured: ``` proposed audit@1 audit filled the disk no remedy: a person has to look at this proposed fetch@1 fetch failed on the network, not on its own work applied fetch@1 by a configured policy, with no human involved ``` `audit` is the step that matters. It failed, it was explained, and nothing was applied to it even with the policy wide open, because the analyzer proposed no remedy for a full disk. ## Where to go next - **[The AI analyzer](/docs/analyzers/genkit/)**: this interface with a real model behind it, ready to install. - **[Failure analyzers](/docs/analyzers/)**: the gate, and who decides a proposal becomes an action. - **[Control operations](/docs/attach/control-ops/)**: `analysis.accept` and `analysis.reject` on the wire. - **[Reading a failed run](/docs/run/debugging/)**: what senro tells you without any analyzer. --- > Source: https://xavidop.github.io/senro/docs/extend/exporter # Writing a trace exporter A `senro.Sink` receives every event the engine appends to a run's ledger, in ledger order. Reach for one to fold that stream into something else: OpenTelemetry spans, metrics, a database row per step. senro carries W3C Trace Context through the stream but ships no OpenTelemetry exporter, so this page uses spans as the example. The split is deliberate: senro's job is trace context that is **correct**, and turning that into spans is arithmetic over data anybody can read, against the `go.opentelemetry.io/otel` version you already have. ## The interface ```go package senro type Sink interface { Emit(api.Event) } func WithSink(s Sink) Option ``` That is the whole surface, and it is the same one [notifications](/docs/notifications/) are built on. Two optional interfaces are under [the contract](#what-senro-guarantees-you). ## The smallest one that works ```go func (e *Exporter) Emit(ev api.Event) { e.mu.Lock() defer e.mu.Unlock() switch ev.Type { case api.StepStarted: var b api.StepStartedBody if ev.Decode(&b) != nil || b.SpanID == "" { return } e.open[ev.Step] = &Span{ TraceID: ev.TraceID, SpanID: b.SpanID, Parent: b.ParentSpanID, Links: b.LinkedSpanIDs, Name: ev.Step, Start: ev.TS, } case api.StepFinished: var b api.StepFinishedBody if ev.Decode(&b) != nil { return } e.close(ev.Step, ev.TS, b.State.Failed()) } } ``` That produces a span per step and misses three cases, which are [below](#the-four-events-that-end-a-span). ## What is in the stream Every event carries the trace. The span structure is in the payloads, because unlike the trace ID it is not constant. | Where | Field | What it is | | --- | --- | --- | | every event | `trace_id` | 32 lowercase hex characters, identical on every event of the run | | `run.started` | `span_id` | the run's own span, the root of everything this run emits | | `run.started` | `parent_span_id` | the inbound span, absent when senro started the trace | | `run.started` | `trace_flags` | the W3C flags byte, two lowercase hex characters | | `run.started` | `tracestate` | the inbound vendor state, verbatim | | `run.finished` | `span_id` | the run span again, so the last event closes it on its own | | `step.started` | `span_id` | this **attempt's** span | | `step.started` | `parent_span_id` | from the graph, never from the clock | | `step.started` | `linked_span_ids` | the needs that could not be the parent | | `step.finished` | `span_id` | the span to close | | `step.finished` | `parent_span_id` | present only when this event opened the span | | `handler.*` | `span_id`, `parent_span_id` | one handler run, parented on the attempt that triggered it | Everything above is `omitempty`, and every field senro adds within protocol v1 will be. Ignore what you do not recognise. ## The span model - **One span per run.** Its parent is whatever started senro, when something did. - **One span per attempt, not per step.** A step retried three times produces three `step.started` events with three span IDs. One merged span would report a step that took the sum of every attempt and succeeded, which did not happen. - **Parentage comes from the graph, never from the clock.** A step's span hangs off the first of its `Needs` in plan order; a step with no needs hangs off the run. Two steps that ran back to back only because `MaxParallel` was 1 are **siblings**, since nesting them would report a pipeline with no parallelism at all, which is exactly what somebody opened the trace to find out about. - **A span has one parent; a step may wait on many.** The needs that could not be the parent are in `linked_span_ids`, which is what OpenTelemetry links are for: causality without containment. ### The four events that end a span Three are easy to miss, and each one missed leaves a span that never closes or work absent from the trace. - **`step.finished` closes the attempt.** The ordinary case. - **`step.retried` also closes an attempt.** An attempt that will be retried emits no `step.finished` at all. Close the step's currently open span here; no span ID is carried and none is needed, because a step has at most one attempt in flight. - **`step.finished` sometimes opens the span too.** A step restored from cache emits `cache.hit`, `ws.restored` and `step.finished` with no `step.started` anywhere, and a step skipped by a false `When` condition or a failed upstream is the same shape. Their finish event carries `parent_span_id` and `linked_span_ids` precisely so the span can exist, and the start time is the event's timestamp less the `duration_ns` it reports. - **`handler.succeeded` and `handler.failed` close a handler.** Handlers emit **no** `step.log.appended` markers, so anything that models a run by walking what steps logged misses them completely. ## The contract ### What you must guarantee **`Emit` must not block.** It is called inline on the engine's goroutine, holding the lock that makes a ledger append and its delivery a single atomic unit, so a slow `Emit` slows the whole run and a wedged one stops it. Do only what is cheap: touch a map and return, and hand each finished span to a batching span processor. ### What senro guarantees you - Every event the run appends, in ledger order, exactly once. `WithSink` is repeatable and composes with `senro.WithAttach`: a run given both feeds the attach socket and every sink from one stream. - **`senro.Flusher`** gives you a bounded chance to finish after the run's stream is sealed, on a context derived with `context.WithoutCancel`, so a **cancelled** run still exports its trace. That is the run whose trace somebody actually wants. Shut your span processor down here. - **`senro.Reporter`** hands you an appender for recording your own outcomes in the run's ledger. An exporter rarely needs it; a notifier does. ### What happens on error There is nowhere to return one, deliberately: an observer must not be able to end a build. A panic in `Emit` is recovered and the event dropped. It is still a bug in your sink. ## Wire it into a run ```go err := senro.Run(ctx, pipeline, senro.WithSink(otelspan.New(os.Stdout))) ``` ### Continue an inbound trace This is the highest-value part and needs no code from you. senro reads `TRACEPARENT` and `TRACESTATE` from its own environment, both spellings, which every CI system and deploy tool that has a trace exports. Such a run carries the upstream trace ID on every event and the upstream span as `run.started`'s `parent_span_id`. From an embedder that already holds a span: ```go sc := trace.SpanContextFromContext(ctx) err := senro.Run(ctx, pipeline, senro.WithSink(exporter), senro.WithTraceContext( fmt.Sprintf("00-%s-%s-%02x", sc.TraceID(), sc.SpanID(), sc.TraceFlags()), sc.TraceState().String())) ``` `WithTraceContext` wins over the environment, including when given empty strings, which is how a caller says "this run is a root, ignore the ambient variables". > A **malformed** traceparent is ignored and the run starts a fresh trace. It is never salvaged and > never a reason to refuse to run: half a broken trace ID joined to a fresh span would be a link to a > trace that does not exist. ### The trace continues outward Every step's command is launched with `TRACEPARENT` set to **that attempt's own span**, so a tool that reads it becomes a child of the step it ran in. Your exporter does nothing to get this. - **The attempt's span, not the run's**: one shared parent would flatten the trace into a list. Handlers too, from the span `handler.started` published, and `TRACESTATE` travels with it. - **On every executor**: local, container, Kubernetes and ssh. On Kubernetes it is ordinary pod env, readable by anybody who can read the pod, which is fine for two random identifiers and is exactly why secrets are files there instead. - **A step that declares its own `TRACEPARENT` (or `traceparent`) wins**: senro leaves it alone and exports no tracestate beside it, because vendor state belongs to the trace its traceparent named. - **It never enters the cache key.** The key's env component digests only the names a step declared in [`CacheEnv`](/docs/data/caching/), built from the step's declared environment rather than what the command is finally launched with. A value that changes every run would otherwise mean a pure step never hits again. ## The worked example [`examples/extensions/otelspan`](https://github.com/xavidop/senro/tree/main/examples/extensions/otelspan) handles all four closing events and imports `github.com/xavidop/senro/api` and the standard library and nothing else from senro, which the test suite checks mechanically. Run it against a real pipeline with `go run ./examples/otelexport`, whose pipeline is deliberately awkward: `lint` and `test` in parallel off `fetch`, `test` failing once and recovering, `audit` skipped by a false condition, `package` waiting on two things, `deploy` with an `Always` handler. Six steps, nine spans: ``` trace 254ace9a1767055bbbec4d35943b14ba release cd2d9cef6e314db2 1.419s senro.status=succeeded_with_recovery fetch 663ca0a6d8208e8b 11ms senro.attempt=1 senro.state=succeeded audit 65431cefc7ffe417 0s senro.state=skipped_condition ! test 01ef82c82a778c69 21ms senro.attempt=1 senro.state=retried lint 170842380fc784dd 229ms senro.attempt=1 senro.state=succeeded package d314461326c97a28 16ms link=ed641a4c5d0081c8 deploy d370199f0c116957 12ms senro.state=succeeded deploy/always/release-lock ae63ed6490deb3d6 13ms senro.handler_kind=always test ed641a4c5d0081c8 16ms senro.attempt=2 senro.state=recovered ``` Both attempts at `test` are children of `fetch`, not of each other; `lint` and `test` are siblings even though one finished first; `audit` is present with a zero duration and a reason rather than absent; `package` links the attempt at `test` that actually succeeded. For unit tests, feed it events and assert on the spans: you need no engine and no collector. For the end-to-end shape, run a real pipeline with `senro.WithSink(exp)` and check that every span walks up to a single root with no dangling parent and no cycle. That one assertion catches every parentage mistake worth catching. ## Where to go next - **[The event stream](/docs/run/event-stream/)**: every event type and payload, which is what you are folding. - **[Writing a notifier](/docs/notifications/custom/)**: another extension point built on the same sink. - **[Run options and outcomes](/docs/run/options/)**: where `WithTraceContext` earns its keep.