diff --git a/docs/workpad/designs/2026-10-16-01-headless-forks.md b/docs/workpad/designs/2026-10-16-01-headless-forks.md new file mode 100644 index 0000000..7b1273d --- /dev/null +++ b/docs/workpad/designs/2026-10-16-01-headless-forks.md @@ -0,0 +1,430 @@ +# Design: Trained implementer, forked per block (`spire:forks`) + +**Date**: 2026-10-16 +**Designer**: model id `claude-opus-5-5` (stage 3; does not plan or build) +**Goal**: [goals/headless-forks.md](../goals/headless-forks.md), including its "Decided 2026-10-16" section +**Inputs**: [research/2026-10-16-00-claude-headless-fork-sessions.md](../research/2026-10-16-00-claude-headless-fork-sessions.md) (claim ids such as `3a`, `F4c` refer to it), [reports/2026-10-15-23-athanor-trunk-fork-provenance.md](../reports/2026-10-15-23-athanor-trunk-fork-provenance.md), [reports/2026-10-15-22-spire-vs-superpowers-plan-and-build-experiment.md](../reports/2026-10-15-22-spire-vs-superpowers-plan-and-build-experiment.md) §5 +**Touches**: new skill `plugins/spire/skills/forks/` (`SKILL.md`, `forks.mjs`, `preamble.md`, `templates/`), `plugins/spire/agents/athanor.md`, `plugins/spire/skills/implementation/SKILL.md`, `references/spawning-claude-code.md`, `references/run-log.md`, `references/models.md`, `plugins/spire/.claude-plugin/plugin.json` (version) +**Status**: one approach, for the planner (stage 4) and the attacker (4b) + +Marking: **Fact** means verified in the research (claim id given) or checked by the designer today (`claude --help` of 2.1.283, `node --version`, `which`). **Assumed** means not verified; the planner's preflight must check it (§10). + +--- + +## 1. The shape in one picture + +``` +Igor ──Agent──► Athanor (Opus, plan level: plan, design, trunk notes, reports, run log; no code reads) + │ + │ Bash (foreground, ≤ 9 min per call) + ▼ + node forks.mjs train / fork / rebase / continue / wait / usage / list / expire + │ spawns detached supervisor per call + ▼ + supervisor ──► claude -p --resume --fork-session (headless, scrubbed env, + │ full fingerprint) + └─► registry.json, ledger.jsonl, results/.json (next to the run log) + +Base B1 (trained once: reads code, writes trunk notes) ──fork──► F1.1 F1.2 F1.3 … (one per complex block) + └─rebase at phase boundary──► B2 ──fork──► F2.1 … +Mechanical / distant blocks: fresh Agent-tool agents reading the trunk notes (unchanged path). +``` + +The Athanor never builds a block itself and never reads code. The helper owns every `claude -p` command line, the registry, the cache bookkeeping and the usage ledger. The Athanor writes briefs, reads handoff notes and helper JSON, runs mechanical gates and test commands, and keeps the run log. + +--- + +## 2. The skill: `spire:forks` + +**Name**: `forks` (directory `plugins/spire/skills/forks/`). Short, says what it does, and lifts to Dirigent as one folder. + +**Files**: + +| File | Role | +|---|---| +| `SKILL.md` | Contract: when to use, commands, the rules the Athanor follows, the JSON it gets back. Short. | +| `forks.mjs` | The helper. One file, Node built-ins only, no `npm install`. | +| `preamble.md` | Appended system prompt for every base (`--append-system-prompt-file`): role of a trained implementer that will be forked, handoff-note format, "do this yourself; do not dispatch agents", stamp model id. Part of the fingerprint (hashed). | +| `templates/train.md`, `templates/fork.md`, `templates/rebase.md` | Fixed prompt frames the helper wraps around the Athanor's brief. Versioned with the skill. | + +**Who loads it, when**: the Athanor, by frontmatter preload (`skills:` gets `spire:forks`), so it is present from intake. The implementation skill names it in "Context" (§7) as the build path for complex blocks. Igor does not load it; Igor only sees its numbers in the run log. Any other agent may load it for a non-build use, but the Athanor procedure is the only contract this design defines. + +**SKILL.md head** (pattern of `adepts`): + +```yaml +--- +name: forks +description: Trains one headless implementer per build and forks it per block via `claude -p --resume --fork-session`. Owns the command line, the cache bookkeeping and the usage ledger. Loaded by the Athanor. +allowed-tools: Bash(node:*) +--- +!`node "${CLAUDE_SKILL_DIR}/forks.mjs" where` +``` + +`where` prints the absolute path of `forks.mjs`, the resolved binary and its version, the node version, and whether the env scrub list matches the host. The Athanor uses the printed path in every later call. **Assumed**: a preloaded skill's `!` injection runs at agent start, and `allowed-tools: Bash(node:*)` pre-approves the later calls (the `adepts` skill relies on the same pattern for `bash`; its own comment calls it novel). + +--- + +## 3. The helper `forks.mjs` + +### 3.1 Runtime choice: Node + +Checked on this host today (Git Bash): `node` v24.8.0 at `/c/Program Files/nodejs/node`; `bash` 4.4.23; no `jq`; no `python`/`python3` (Python 3.7 directory is on PATH but no executable resolves); Strawberry `perl` present. + +- **Node, chosen.** JSON is native (the whole job is parsing `stream-json`, `modelUsage` and the registry). `child_process.spawn` takes an explicit `env` object, so the scrub is a dict operation, not `env -u` quoting through Git Bash. Detached spawn, kill-tree, atomic `rename`, `open(..., 'wx')` lockfiles and path normalisation work on Windows and POSIX. The research already used `node` for JSON here (claim 7). +- **Assumed** about other hosts: `node` ≥ 18 is on PATH wherever Spire runs (the native Claude Code installer does not need Node, so this is not guaranteed). `where` fails loudly with "node not found" and the Athanor falls back to the fresh-agent path, logged. +- Rejected: bash + jq (no jq here; JSON in bash is fragile). Python (not on PATH). PowerShell (not on Linux remote hosts; the repo's scripts are bash). Perl (works, but nobody in this repo maintains Perl). The Agent SDK in TypeScript (needs `npm install`, spawns the CLI anyway; that is the Dirigent-later path). + +### 3.2 State location: next to the run log, per run + +A trained base belongs to one build plan, and the 1 h TTL (`F4a`) means no base usefully outlives a run. The state lives beside the run log so a successor Athanor and the reviewer find it: + +``` +/reports/--run.md the run log (unchanged name rule) +/reports/--run-forks/ helper state, owned by forks.mjs + registry.json bases, jobs, config; never hand-edited + ledger.jsonl one line per finished call, append-only + briefs/.md the exact prompt sent (frame + Athanor brief + delta header) + notes/.md handoff notes written by forks (one file each: parallel-safe) + results/.json the per-call summary `wait` returns + logs/ gate and test output written by forks and the Athanor + raw/.jsonl stream-json stdout ┐ gitignored by the helper's own + raw/.err stderr ┘ .gitignore (raw/, *.lock) +``` + +Every command takes `--run `; the state dir is derived from it. The run log header gets one line, `Forks: `. This is a dated subfolder for a cascade of related files, as the workpad skill allows. + +### 3.3 Fingerprint + +Any difference in model, tools, permission mode or binary loses the cache (`3b`, `4b`, `F5b`), and flags are not inherited by a fork (`4b`, `4c`). So the Athanor passes launch settings **once**, at `train`; `fork`, `rebase`, `continue` and pings read them from the base record. The Athanor cannot get a fork's flags wrong because it never supplies them. + +Launch fields (hashed into `fingerprint.hash`, canonical JSON, sha256): + +| Field | Source / default | Why | +|---|---|---| +| `binary.path` | `--binary`, else `$CLAUDE_CODE_EXECPATH` if set, else `command -v claude` | Two installs exist (`F5a`). The Desktop's bundled binary matches the host and runs the newest models: Sonnet 5.5 needs ≥ 2.1.284 (`F2c`), the PATH install is 2.1.283. | +| `binary.version`, `binary.size`, `binary.mtime` | ` --version`, `stat` | Path alone is not enough: a native install can auto-update in place. Hashing a large exe on every call is too slow; size + mtime + version is the check. | +| `model` | required, full id (`^claude-[a-z0-9-]+(\[1m\])?$`); aliases refused | Aliases move with the CLI version (`F2c`); plan mode swaps Haiku to Sonnet (`F2a`). Goal decision: always a full id. **Assumed**: `--model` accepts the `[1m]` suffix. | +| `effort` | required (`--effort`), skill default `high` | The host passes `CLAUDE_EFFORT=medium` (`F2b`) and user `settings.json` has `effortLevel: medium`; a user changing effort in the Desktop mid-run would change a fork's effort. **Assumed**: effort changes the request (thinking settings) and so the cache. | +| `permission_mode` | default `dontAsk` | Never `plan` (`F2a`). `dontAsk` denies anything not pre-allowed (`4a`) and so needs the explicit list below. | +| `tools` | default `Bash,Read,Edit,Write,Glob,Grep,Skill` | No `Task`/`Agent`: forks are leaves, which also removes the depth question (`F1c`, `F1d`). **Assumed**: `Skill` is a valid `--tools` name. | +| `allowed_tools`, `disallowed_tools` | default: allow = tools; deny = `Bash(git push:*)`, `Bash(git rebase:*)`, `Bash(git reset --hard:*)`, `Bash(claude:*)`, plus project additions | Same guard rails as today's children; no nested `claude` from a fork. | +| `mcp` | default `--strict-mcp-config --mcp-config /mcp-empty.json` | Not restored on resume (`4c`), so it is passed every time. Pins the tool list against claude.ai connectors toggling mid-run. **Assumed**: strict mode also drops claude.ai account connectors (`F3b` shows they load otherwise). | +| `agent` | optional `--agent `; stored as name + sha256 of the resolved file | Goal: a project agent file can specialise the implementer. **Assumed**: `--agent` resolves project `.claude/agents/.md`; plugin-namespaced names untested; how the agent body combines with the default system prompt untested. | +| `append_system_prompt` | `preamble.md`, stored as sha256 | Recorded at first request and reused (`4c`; `--system-prompt-snapshot on` is the default, designer read it in `--help` today). Passed identically every time anyway. | +| `cwd` | `--cwd`, default the git toplevel; stored normalised | Forks must run from the base's cwd (`F4d`); the transcript path derives from it (`6d`). | +| `env.pinned` | `CLAUDE_CODE_PROMPT_CACHE_TTL=1h`, `DISABLE_AUTOUPDATER=1` | TTL constant across base and forks (`F4b`). **Assumed**: `DISABLE_AUTOUPDATER` stops an in-place update of the PATH install mid-run. | +| `env.scrubbed` | list in §4 | | + +Runtime fields, taken from the `system/init` event of every call and compared with the base's: sorted tool names (hash), `permissionMode`, plugin names with versions, agent names, `apiKeySource` (**assumed** field name). A difference sets `fingerprint: drift:[fields]` on the result. Before every launch the helper re-reads `binary.version/size/mtime`; a change is a `binary` drift (rule in §5). + +`init.model` is **not** recorded as the model. It lies under plan mode (`F2a`, `F2c`); the real model comes from `modelUsage` (§6). + +### 3.4 Commands + +All commands print one JSON object on stdout and use exit codes: `0` done/ok, `2` usage or refused, `3` `wait` timed out with jobs still running, `4` a job failed. + +| Command | Inputs | Does | Returns | +|---|---|---|---| +| `where` | — | Preflight print for SKILL.md. | script path, binary, versions, node, scrub check | +| `init` | `--run`, launch fields (§3.3), `--parallel-cap` (default 2), thresholds (§5) | Creates the state dir, `.gitignore`, `mcp-empty.json`, the config block in `registry.json`. Idempotent. | config | +| `train` | `--run`, `--base `, `--brief `, optional launch overrides (else the `init` config) | New session: `claude -p --session-id ` with the full fingerprint and `templates/train.md` around the brief. On finish records `base_commit` (`git rev-parse HEAD`), training cost, context size, last hit. | `{job, base, session_id}` | +| `fork` | `--run`, `--base`, `--block `, `--phase `, `--brief `, `--mode sequential\|parallel`, `--max-minutes` (default 90), `--max-budget-usd` (optional) | `--resume --fork-session` with the base's fingerprint, `templates/fork.md` + delta header + brief. Refuses a different model or cwd (that is a new base), refuses when running jobs on the run would exceed the parallel cap, refuses `parallel` mode unless the brief names its file set. | `{job, base, base_state: warm\|expiring\|cold, warm_until, rebase_due, retrain_due, reasons[]}` | +| `rebase` | `--run`, `--base`, `--as `, `--brief ` | A fork whose job is to absorb what changed (`templates/rebase.md`: read the listed notes and diffs, do not edit code, update the trunk notes, reply READY). Its session becomes base `` with `parent`; the old base is `superseded`. | as `train` | +| `continue` | `--run`, `--job `, `--brief ` | `--resume ` **without** `--fork-session`: one more turn in that fork (gate failed, test fallout). Same fingerprint. Not a SendMessage resume, so no notification-routing hazard. | `{job}` | +| `wait` | `--run`, `[job…] \| --all`, `--any`, `--timeout ` (default 540, max 570) | Blocks until the condition holds or the timeout. Runs the keep-warm tick (§5) every 30 s while it waits. | `{done:[result…], running:[{job, block, elapsed_s, last_event_s_ago, turns}], pings:[…]}` | +| `usage` | `--run`, `--phase`, `--block`, `--format json\|md` | Aggregates `ledger.jsonl`. `md` prints rows for the run log. | totals and rows (§6) | +| `list` | `--run` | Bases (status, warm state, `warm_until`, context tokens, training cost, forks, lineage) and running jobs. | | +| `expire` | `--run`, `--base \| --all` | Marks bases `retired`: no forks without `--force`, no pings. Transcripts stay on disk (30 days, `6d`). | | +| `kill` | `--run`, `--job` | Kills the supervisor and its `claude` process tree (`taskkill /T /F` on Windows, process group on POSIX); result `killed`. | | + +Internal: `_supervise ` (the detached worker) and `tick` (one keep-warm pass, also callable by hand). + +### 3.5 Launch and supervision + +- Each launching command writes the job record, then spawns `node forks.mjs _supervise ` **detached** (`detached: true`, `windowsHide: true`, stdio to `raw/`) and returns at once. The supervisor spawns `claude` with the scrubbed env, `--output-format stream-json --verbose`, and the prompt from `briefs/.md` on stdin. +- The supervisor reads the stream as it arrives: the `system/init` event gives the session id and the runtime fingerprint; each assistant event bumps `turns` and `last_event_at`; the final `result` event gives `usage`, `modelUsage`, `total_cost_usd`, `num_turns`, `is_error`, `permission_denials`. **Assumed**: the stream's `result` event carries the same fields as `--output-format json` (`1a` verified them for `json`). +- On exit it writes `results/.json`, appends to `ledger.jsonl`, updates `registry.json` under the lock (`registry.lock`, `open wx`, stale after 30 s, write-temp-then-rename). +- Watchdogs: `--max-minutes` exceeded → kill, status `timeout`. No stream output for 15 min → status flag `stalled` (reported by `wait`, not killed). `permission_denials` non-empty → `needs_attention` (a denied `git commit` would otherwise pass silently, `4a`). +- Fork session id: taken from `init`. **Unknown** whether `--session-id` combines with `--resume --fork-session`; if it does, the helper may pre-assign ids, but nothing depends on it. +- Fallback if detached children do not survive on this host (R1): `--attach` runs the supervisor in the calling process; the Athanor starts it with the Bash tool's `run_in_background: true` and still polls with `wait`. The Athanor never relies on the background notification. + +--- + +## 4. Env scrub and auth + +Dropped from the child env (the twelve the research removed in E1 with auth still working, `F3a`, `F3b`): + +`CLAUDE_CODE_MESSAGING_SOCKET`, `CLAUDE_CODE_MESSAGING_TOKEN`, `CLAUDE_CODE_HOST_SESSION_ID`, `CLAUDE_CODE_ENTRYPOINT`, `CLAUDE_CODE_DESKTOP_APP_VERSION`, `CLAUDE_CODE_SDK_HAS_HOST_AUTH_REFRESH`, `CLAUDE_CODE_TERMINAL_MCP_TOOLS`, `CLAUDE_CODE_REPORT_FINDINGS`, `CLAUDE_CODE_SESSION_ID`, `CLAUDE_AGENT_SDK_VERSION`, `CLAUDE_CODE_ENABLE_ASK_USER_QUESTION_TOOL`, `CLAUDE_CODE_ENABLE_SDK_FILE_CHECKPOINTING`. + +Also dropped, because they would change the fingerprint or the auth path: + +- `CLAUDE_EFFORT` (effort comes from `--effort`), `ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_*_MODEL`, `ANTHROPIC_SMALL_FAST_MODEL`, `CLAUDE_CODE_SUBAGENT_MODEL` (none present today, `F2b`; dropped so a future host cannot slip one in). +- `ENABLE_PROMPT_CACHING_1H`, `FORCE_PROMPT_CACHING_5M`, `CLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL` (TTL is pinned, §3.3). +- `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_USE_BEDROCK`, `CLAUDE_CODE_USE_VERTEX`: with any of these the run would bill elsewhere and get 5 min TTL (`F4a`). Goal decision: OAuth. + +Kept as in the tested E1 run: `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `AI_AGENT`, `CLAUDE_CODE_EXECPATH`, `CLAUDE_PID`, `CLAUDE_CODE_GIT_BASH_PATH`, `PATH`, `HOME`/`USERPROFILE`. Each kept var's role is unverified (`F3a`); keeping E1's set means the design runs the configuration that was tested. + +**Auth stays OAuth**: no `--bare` (it refuses OAuth, `F3d`), no key in the env. The helper checks each `init`'s `apiKeySource` (**assumed** field; expected `none` under OAuth) and the observed TTL (§5): a call that wrote only `ephemeral_5m` means the account fell out of the 1 h bucket (over plan limit, `F4a`), and `wait` reports `ttl_downgrade: true`. The scrub list lives in one constant in `forks.mjs`; `where` reports host vars that look Desktop-specific (`CLAUDE_CODE_*` not on either list) so a new host variable shows up instead of leaking silently. + +Not settled by the scrub: whether the nested process's own messaging pipe can reach the host (`F3c`). The scrub removes the host socket and token; nothing more is claimed. + +--- + +## 5. Keep warm, expire, rebase, re-train: the rule + +All inputs come from the registry and the ledger; the Athanor only obeys the flags. + +**Per base**: + +``` +ttl = 3600 s if the base's last call wrote ephemeral_1h tokens, or read cache and the + previous known ttl was 3600; 300 s if it wrote only ephemeral_5m; else 3600 + (config default for subscription). [F4a, 3c] +last_hit = time of the first assistant event of the most recent call on this base or any of + its forks/pings (the first request reads the base prefix). [F4c] +warm_until = last_hit + ttl +margin = max(120 s, ttl / 10) → 360 s at 1 h, 120 s at 5 min +state = warm if now < warm_until - margin + expiring if now < warm_until + cold otherwise +``` + +**Assumed**: a hit on the base prefix refreshes that prefix's entry (`F4c` is docs, not tested over time), and only a request that reads exactly the base prefix (a fork's first request) refreshes it; later requests in a fork read longer prefixes. Using the first assistant event, not the call end, is the conservative reading. + +**Keep warm (pings)**: a ping is `--resume --fork-session`, prompt `Reply OK`, `--max-turns 1`, same fingerprint, logged as kind `ping`. Price ratios against the base input price, from Anthropic API pricing (**assumed** current, not re-checked in the research): cache read 0.1, 1 h write 2.0, 5 min write 1.25. One ping costs about 0.1 × prefix; one cold start about (write − read) × prefix, 1.9 × prefix at 1 h. So pings pay while fewer than `break_even = floor((write − read) / read)` are needed: 19 at 1 h, 11 at 5 min. + +``` +ping base when: base.status == active + and state == expiring + and the run has a running job or a wait call in the last 10 min (the Athanor is in its turn) + and pings since the base's last real call < break_even +``` + +The tick runs inside every supervisor (every 60 s) and inside `wait` (every 30 s). No daemon: when the Athanor has handed back and nothing is running, nobody pings, and the base cools after its TTL. That is intended: between phases the base is rebased anyway. + +**Expire**: a base is `retired` by `expire`, `superseded` by `rebase` or a re-train. A `cold` base stays forkable: the history is on disk (`2b`, `6d`), only the cache is lost. **Coldness is never a reason to re-train.** A fork of a cold base is logged `cold_start` with its creation tokens. + +**Rebase due** (flag on `fork`'s return; the Athanor rebases before forking): + +- phase boundary (procedural: the first fork of a new phase always comes from a rebased base); +- delta since `base_commit` above threshold: more than 6 commits, or more than 1,500 changed lines (`git diff --shortstat`), or a delta header plus listed notes above about 12k tokens (chars / 4); +- binary drift (version, size or mtime changed): the rebase runs under the new binary and becomes the new fingerprint; one cold write, recorded. + +**Re-train due** (fresh `train`, new lineage): + +- the requested model, cwd, tools, permission mode, agent or preamble differs from the base (these define the base; a fork cannot carry a change); +- base context above `max_base_tokens` (default 250k), measured from the last assistant message's usage in the base transcript (`~/.claude/projects//.jsonl`, `6d`; **assumed** message-level `usage` is present in the transcript lines), leaving room under the ~600k comfortable figure (models.md, a guesstimate); +- four rebases in one lineage (prefix growth). + +Thresholds are starting values in `init`; the trial (§9) tunes them. They are numbers, not judgment. + +--- + +## 6. Usage capture + +Per call, from the `result` event: + +- **Real model**: the keys of `modelUsage` whose counts grew in this call. Not `init.model` (`F2a`, `F2c`). +- **Tokens, four columns**: `input`, `output`, `cache_creation`, `cache_read`, plus `cache_creation` split into `1h` and `5m` from `usage.cache_creation`. Source: the per-model delta of `modelUsage` against the session's totals before the call (`modelUsage` and `total_cost_usd` are cumulative across the restored session, `3d`, `F4e`). Top-level `usage` is stored too; the T2 numbers (creation 343 against the base's 8,661) suggest it is per call, so the two must agree; disagreement sets `usage_mismatch` (**assumed** semantics for multi-turn calls, R5). +- **Cost**: `cost_delta_usd = total_cost_usd(call) − total_cost_usd(session before call)` (`3d`, `F4e`). For a fork, "session before" is the base's cumulative total at the fork point; for `continue`, the fork's own last total. Never sum `total_cost_usd` across calls. These are Claude Code's client-side estimates; under a subscription they are notional, which is still the right unit for comparison. +- **Cache verdict**: `hit` if `cache_read ≥ 0.9 × base prefix`, `partial` if > 0, `miss` if 0 (a miss on a fork is a fingerprint bug or a cold base; `wait` says which). +- Also: `wall_s`, `num_turns`, `is_error`, denials count, fingerprint verdict, binary version. + +**Ledger line** (`ledger.jsonl`): + +```json +{"job":"j0007","kind":"fork","phase":1,"block":"1.3","base":"B1","session":"7ccea6b5-…", + "parent_session":"433df7b7-…","started":"2026-10-16T14:02:11+02:00","ended":"…","wall_s":1310, + "binary":"2.1.299","model_requested":"claude-sonnet-5-5","models_used":["claude-sonnet-5-5"], + "tokens":{"input":5120,"output":38210,"cache_creation":61200,"cache_creation_1h":61200, + "cache_creation_5m":0,"cache_read":2840100}, + "cost_delta_usd":1.92,"cost_cumulative_usd":4.37,"turns":57,"cache":"hit","base_state":"warm", + "fingerprint":"ok","denials":0,"status":"ok","note":"notes/1.3.md"} +``` + +**Into the run log**: the Athanor fills each block row from `usage --block --format md` and pastes `usage --phase --format md` into the phase close: per-kind totals (train, rebase, fork, continue, ping), cold starts, cache hit ratio, priced total. The ledger file stays the source; the run log carries the copy the reviewer and Igor read. Goal said "writes them to the run log": the helper writes the ledger and renders the rows; the Athanor pastes them, because a script appending to a markdown file the Athanor is also editing would interleave. This is a design decision (see the end). + +Agent-tool agents (mechanical blocks) report usage only in their completion result; the Athanor copies what it sees, as today. The root and the Athanor's own tokens are counted from transcripts in the trial (§9), not by the helper. + +--- + +## 7. Blocking and polling + +The headless process is invisible to Agent-tool tracking (`6c`), and a background Bash notification from inside a subagent is not something to build on (routing hazard in the spawning reference; unverified for Bash). So: + +- **Launch returns at once** (detached supervisor). **Waiting is a foreground Bash call** to `wait`, which blocks up to 540 s, under the Bash tool's 600 s ceiling (`6c`). Exit code 3 means "still running, call again". The Athanor loops: `wait --all --timeout 540` until exit 0 or 4. A 36-minute block costs about four `wait` turns of a plan-level Athanor context, mostly cache reads. +- **Parallel forks**: only for blocks the plan marks disjoint (the implementation skill's existing rule). Launch each with `--mode parallel`, then `wait --all`. Cap 2 by default (`init --parallel-cap`): forks on a subscription share plan usage, and an over-limit account drops to 5 min TTL (`F4a`). Two forks from one base in parallel are verified (`2c`); tool-using parallel forks in one worktree are not (R3). +- **Timeouts**: per job `--max-minutes` (default 90) enforced by the supervisor; `--max-budget-usd` optional per call (counts the call's own spend, `F4e`). `stalled` is reported, not acted on; the Athanor kills or waits per the procedure (one more `wait`, then `kill` and flag). +- **Hand back only with nothing running**: before the deliverable, `list` must show no running jobs, else `kill` and flag. +- The Athanor never uses `SendMessage` on a fork and never needs `TaskStop`; `continue` and `kill` cover it. + +--- + +## 8. Athanor procedure + +### 8.1 What the Athanor holds + +The build plan, the design, the run log, trunk notes, handoff notes, review reports, helper JSON, `git log --oneline`, `git diff --stat`/`--numstat`, gate and test summaries. Not source files. Its context stays plan-sized, so `claude-opus-5-5[1m]` keeps its pin but is no longer what pays for reading code (in the experiment, 53 M of 55.6 M Athanor tokens were cache reads of a 1M-context session that had read the trunk). + +### 8.2 One phase per turn + +1. **Intake.** Read plan, design, run log, trunk notes, the last review. Check the branch (`git rev-parse --abbrev-ref HEAD`). First phase: `forks init` with the launch fields the build plan or project config names (model = the plan's owner model for complex blocks, full id; effort; agent file if the project has one). +2. **Train or rebase.** + - First phase: `train --base B1` with a brief naming the plan, design, the phase's blocks, the areas to read, and the trunk-notes path. The base reads the code, **writes the trunk notes**, changes no code, replies READY with a short summary. Trunk notes are now the base's product, not the Athanor's. + - Later phases, and whenever `fork` returns `rebase_due`: `rebase --as B` with the delta (handoff notes, review findings, commit range). `retrain_due`: `train` a new lineage. The helper decides which; the Athanor follows. +3. **Route blocks.** Complex block in the trained area → fork. Mechanical or distant block → fresh Agent-tool agent in the foreground with the trunk notes, as today. A complex block whose owner model differs from the base model either becomes a fork anyway (logged deviation) or gets its own base; it never gets a fork with `--model` (that loses the cache, `3b`, and the helper refuses it). +4. **Fork per block** with a **delta brief** (goal default): block id and plan section, the handoff notes of blocks done since the base, open review flags touching the block, the mode, the gate and test commands. The helper prepends the commit range since `base_commit` itself. Then `wait` until done, read `notes/.md` and the result JSON. +5. **Check, mechanically.** Commit present (`git log`); the evidence log under `logs/` exists and is non-empty (experiment proposal 4); `git diff --numstat` on test paths and a count of removed assertion lines (`git diff -- | grep -c '^-.*assert'`, pattern from project config). A non-zero count goes to the fork's note as a question and into the reviewer flags. This replaces the Athanor's former "read the fix's diff for loosened assertions", which was a code read; the fresh phase reviewer still looks for weakened checks. +6. **Gate** between blocks: the plan's command, output to `logs/`. Red → `continue --job ` with the log path, once; still red → flag and go on per "what may stay broken". +7. **Tests once** at phase end: the Athanor runs the test command to a file under `logs/` (never through `tail`, experiment incident 4; first run after a fresh worktree is a warm-up, rerun once). Fallout → one fork from the current base (or `continue` of the block that caused it) with the failing log as the delta. +8. **Run log**: block rows from `usage`, notes appended from `notes/`, phase close with the usage totals and every flag. +9. **Hand back** with no running jobs. Do not `expire` the base: if Igor's review and the fix pass come back within the TTL, the base is still warm; if not, it is cold but forkable. + +Fix pass (Igor resumes the Athanor with findings): forks of the current base with the findings as the delta; `rebase_due` decides whether to rebase first. + +### 8.3 Commits and tests: who runs them + +The experiment had index-lock races between parallel children and a red commit when the Athanor committed a child's block before calibrating it (M5). + +| Case | Who commits | Who runs tests | +|---|---|---| +| Sequential fork (default) | The fork, by explicit paths, **after** its block gate and the block's named tests pass. Calibrate, then commit. | The fork (block-scoped tests, output to `logs/`). | +| Parallel forks (disjoint files only) | Nobody inside the forks (no index-lock race). After `wait --all`, the Athanor runs the gate and each block's named test command, then commits each block by the explicit paths its note lists, one at a time. Mechanical; no code read. | The Athanor, after all parallel forks finish. Forks run nothing that touches shared state unless project config says parallel test runs are safe. | +| Fresh Agent-tool agent | As a sequential fork. | As a sequential fork. | +| Phase test suite | — | The Athanor, once, to a file. | + +Only one committer exists at any moment: sequential forks commit one after another; parallel forks never commit. + +### 8.4 Text changes + +- **`agents/athanor.md`**: frontmatter `skills:` adds `spire:forks`. Intake: drop "the code the phase touches … That reading is your trunk"; the Athanor reads plans and reports and trains the base. Step 2: replace "a fork of yourself (it inherits your model and, assumed, your context)" with "a headless fork of the trained implementer (`spire:forks`)"; fresh agents stay for mechanical or distant blocks. Remove inline building by the Athanor. Keepers: add "you do not read code; the base and its forks do", "nothing running at hand back", "commit rules per §8.3". Replace "read the fix's diff for loosened assertions" with the mechanical check of §8.2 step 5. Deliverable: add the usage totals and base state. +- **`skills/implementation/SKILL.md`** "Context: fresh, fork, or keep": rewrite "trunk and forks" as "trained implementer and forks": one base per build, trained early, forked per complex block with a delta brief, rebased at phase boundaries, fresh agents for mechanical blocks; cite `spire:forks` for mechanics. The Agent-tool fork path is gone. "Spawning and models": add that headless forks take a full model id. Keep the general "if one agent can do the whole change in one go, do it inline" for non-Athanor use. +- **`references/run-log.md`**: header gets `Forks: `; block table becomes `Block | Owner (model id from modelUsage) | Session | Commit | in | out | cache write | cache read | cost Δ`; phase close gets the `usage --phase` table; trunk notes are written by the base at training. +- **`references/models.md`**: "Future work: orchestrator introspection" gets a dated note that headless forks report per-call usage; add the plan-mode Haiku→Sonnet observation and the alias-by-version note (`F2a`–`F2c`). +- Out of scope, flagged for Igor: `skills/igor/SKILL.md:66` ("forking yourself … to brief a new expert") is now close to what the base does; it may want a line. + +--- + +## 9. Spawning-reference updates (`references/spawning-claude-code.md`) + +**Fork section** (Agent tool), rows changed: + +| Row today | New text and status | +|---|---| +| "`fork` is actually offered … unverified" | Not offered on 2026-10-05 or 2026-10-15. Spire no longer uses Agent-tool forks (goal decision 2026-10-16). observed | +| "A fork copies the caller … assumed" | A fork inherits the current conversation only; no parameter names another agent. verified 2026-10-15 (docs sub-agents, provenance §5) | +| "A fork starts with the caller's full transcript … assumed" | Same system prompt, tools, model, history as the forking session. verified 2026-10-15 (docs) | +| "Can a fork fork again? … unknown" | Agent-tool: "A fork can't spawn further forks." verified 2026-10-15 (docs). Headless `--fork-session`: a fork of a fork works. verified 2026-10-16 (`2d`, T6) | +| new | Fork mode is on by default in interactive sessions (v2.1.232+), off under `-p`/SDK unless `CLAUDE_CODE_FORK_SUBAGENT=1`; with it on, all subagents run in the background and `run_in_background` is removed. verified 2026-10-15 (docs) | +| new | Agent-tool forks and subagents use the 5 min cache bucket by default; the main conversation (incl. `-p`) gets 1 h on a subscription within plan usage. verified 2026-10-16 (`F4a`, docs) | + +**New section "Headless sessions (`claude -p`)"**, rows, each with its research id and date 2026-10-16: + +- `--output-format json` result fields; `stream-json` init fields (`1a`). `--resume` by id across projects; `--session-id` to choose an id (`1b`, `1d`); don't use `--continue` for fan-out (`1c`). +- `--fork-session`: new id, copy of history, base unchanged, forks isolated, parallel forks fine, resuming one id in two processes interleaves (`2a`–`2c`). +- Cache: hits need identical tools, model, system prompt; flags are not inherited by a fork; pass them every time (`3a`, `3b`, `4b`, `4c`). Model is restored on resume, `--model` overrides it (`4d`). The system prompt is recorded once and reused (`4c`; `--system-prompt-snapshot`, `--help` 2.1.283). +- Cost: `total_cost_usd`/`modelUsage` are cumulative across a resumed session; per-call cost is a difference (`3d`, `F4e`). +- TTL: 1 h main bucket on subscription, 5 min otherwise; configurable; a hit resets the timer; same cwd for shared prefix (`F4a`–`F4d`). +- Permissions under `-p`: nothing can prompt; denials are listed and the run continues (`4a`). +- `init.model` is not the model that ran; plan mode upgrades Haiku to Sonnet; aliases change with CLI version (`5d`, `F2a`–`F2c`). +- Discovery: without `--bare`, plugins, skills, hooks, MCP and CLAUDE.md load (`5a`); `--bare` needs an API key (`5b`, `F3d`). +- Nesting: inherits `CLAUDECODE=1`, no recursion guard (`6a`); Agent depth is per process and not passed down; at the limit the Agent tool is simply absent, no error (`F1a`–`F1d`). +- Tracking: invisible to task notifications, `ListAgents`, `TaskStop`; `--bg` does not combine with `-p` (`6c`, the invisibility row stays **assumed**). +- Transcripts per fork under `~/.claude/projects//` (`6d`). +- Desktop env inheritance and the scrub (`6e` as corrected by `F3a`–`F3c`). +- Two binaries; pin one and a full model id (`F5a`, `F5b`). + +**Models section**: the alias-only row stays for the Agent tool; add "headless `--model` takes full ids, so headless forks pin by full id". + +**Rules of thumb**: add "complex blocks go to headless forks via `spire:forks`; never resume or fork a headless session by hand". + +--- + +## 10. The success-criteria trial + +**Preflight first (Phase 0 of the plan, scratch repo, before any Spire text changes ship)**. Each item maps to a risk in §11: + +1. Detached supervisor survives the end of the Bash call that started it, from inside an Agent-spawned subagent on this Windows host (R1). If not, test `--attach` under `run_in_background`. +2. A base trained to ≥ 100k tokens with real tool use; three sequential forks that read and write files; each fork's first request hits the base prefix (R2). +3. Two parallel tool-using forks on disjoint files in one worktree; no index lock (they don't commit); gate afterwards (R3). +4. `modelUsage` delta equals top-level `usage` on a multi-turn fork; `stream-json` `result` has the `json` fields (R5). +5. A ping after 50 min keeps a fork at 60+ min warm; a fork after 65 min without ping is `cold` (R7). This is the only test that needs wall time; it can run beside the others. +6. `--effort`, `--strict-mcp-config`, `--agent`, `Skill` in `--tools`, `[1m]` in `--model`, `apiKeySource` in init: each accepted and visible in init (R8, R9, R15). + +Any failure in 1, 2 or 4 stops the plan and goes back to Igor (§11, "what would make this wrong"). + +**The trial** (goal: "Proven, not assumed"): + +- **Plan**: one real multi-block plan the user picks, with at least two phases and at least three complex blocks in one area (otherwise forks have nothing to share). Not this skill's own build: the tool under test cannot measure itself. +- **Arms**: F = Athanor with `spire:forks` (this design). C = the current 0.2.2 shape (Athanor reads the trunk, builds or delegates per its current text). Same build plan, same base commit, separate worktrees, same reviewer model and review structure (one fresh Opus reviewer per phase, one fix pass, re-check), Igor as root in both. +- **Isolation**: sealed inputs per arm and a transcript audit with a positive control, as in the 2026-10-15 experiment (§1 there). Arm F's headless transcripts live under the worktree's project slug and are audited too. +- **Quality**: blind implementation judge (Fable, as before) on the two diffs; reviewer findings by severity per phase; tests and gates. +- **Tokens, all four columns, priced**, for every agent including the root: Igor, the Athanor, Agent-tool children, every headless call (the ledger, cross-checked against the fork transcripts with copied base history counted once, deduplicated by message id; **assumed** copied messages keep their ids), reviewers. Price with one published per-model table, written into the report. +- **Wall time**: from `date` and git, start to last re-check, per arm; Athanor active time per phase. +- **Mechanism metrics (arm F)**: cache verdicts per fork, cold starts, pings and their cost, rebases, re-trains, permission denials, fingerprint drifts, `ttl_downgrade`. +- **Report**: `reports/-headless-forks-trial.md`. Thresholds in §5 are tuned from it. + +--- + +## 11. Risks and unknowns for the planner and attacker + +| # | Risk / unknown | Check | +|---|---|---| +| R1 | Detached children may be killed when the Bash tool call ends (Windows job object). Untested. | Preflight 1; fallback `--attach` + `run_in_background`. | +| R2 | Large tool-using bases are untested (research used tiny contexts). Cache behaviour, compaction inside a fork, and whether the recorded system prompt is reused for a 100k+ base. | Preflight 2. | +| R3 | Parallel tool-using forks in one worktree: file collisions, shared test resources. | Preflight 3; parallel only for disjoint files; cap 2. | +| R4 | Permissions. The Athanor's `node` calls may prompt in the host session (closed door). Forks under `dontAsk` deny silently. | `allowed-tools` on the skill (assumed effective); `needs_attention` on denials; the user may need an allow rule for `node */forks.mjs *`. | +| R5 | Top-level `usage` per call vs. aggregate over turns; `stream-json` `result` field parity with `json`. | Preflight 4. | +| R6 | `--session-id` with `--fork-session`. | Not depended on. | +| R7 | TTL refresh on hit and the base-prefix refresh semantics are docs-only; cross-cwd forks untested. | Preflight 5; forks always run in the base's cwd. | +| R8 | `--effort` vs. `CLAUDE_EFFORT` and `effortLevel` precedence, and whether effort affects the cache. | Preflight 6; env var dropped, flag pinned. | +| R9 | Mid-run drift outside the fingerprint: plugin updates, SessionStart hooks injecting context on resume, claude.ai connectors. | Runtime fingerprint compare on every init; `--strict-mcp-config`. | +| R10 | Whether the nested process's pipe can reach the host (`F3c`). | Not claimed; scrub only removes the host socket and token. | +| R11 | Plan usage. Parallel Opus/Sonnet forks consume the subscription; over the limit the TTL drops to 5 min and usage may bill as credits. | `ttl_downgrade` flag; parallel cap; per-call `--max-budget-usd`. | +| R12 | Binary changes mid-run (in-place auto-update; a Desktop update replacing the `$CLAUDE_CODE_EXECPATH` directory). | Version/size/mtime check before each launch; `DISABLE_AUTOUPDATER=1` (assumed honoured); binary drift → rebase. | +| R13 | A fork's recorded system prompt carries the base's git status and CLAUDE.md from training time. | The helper's delta header lists commits since `base_commit`; the fork template says the snapshot is stale. | +| R14 | Node may be missing on other hosts. | `where` fails loudly; fresh-agent fallback, logged. | +| R15 | `--agent` with project and plugin agent files; how the agent body combines with the default system prompt and `--append-system-prompt-file`; agent `tools:` frontmatter against `--tools`. | Preflight 6. | +| R16 | Safety. Forks hold full Bash under `dontAsk`; deny patterns are easy to sidestep. | Same exposure as today's children; project deny rules apply; the brief's scope fence. | +| R17 | Delta briefs may leave forks with a stale view of neighbouring blocks, so they collide or contradict. | The rebase threshold; the trial's quality arm. | + +**What would make this approach wrong**: + +- Forks of a large trained base do not hit the cache reliably (R2). Every fork then pays full creation at 2× input price on top of the training, and fresh agents with trunk notes are cheaper. Stop and go back to Igor. +- Headless calls cannot run longer than one Bash call from inside a subagent on this host (R1 and its fallback both fail). The 9-minute ceiling is shorter than real blocks (11–36 min in dirigent). +- The trial shows no quality gain over arm C at higher cost. The goal's intent then still stands, but the evidence says no; that is the user's call. +- Plan-usage policy makes nested headless sessions impractical (R11). + +--- + +## Rejected alternatives + +- **Agent-tool forks, or the trained implementer forking itself.** No fork-source parameter; "a fork can't spawn further forks"; fork mode forces background; depth would sit at the 3-layer limit (provenance §5). The goal drops this path. +- **Carry one implementer and resume it with `SendMessage`.** Accumulates context (the 960k drift) and has the subagent-resume routing hazard. +- **MCP server now.** The goal puts it later; a script is quicker to prove and lifts into Dirigent or an MCP wrapper unchanged in contract. +- **`--bare` + API key.** User chose OAuth; `--bare` would also cost the 1 h TTL. +- **Background Bash notifications or `Monitor` as the wait mechanism.** Unverified inside a subagent; polling files with `wait` works for any caller. +- **A long-lived `--input-format stream-json` base process fed per block.** It continues one session; it cannot branch it, so forks would still need `--resume --fork-session`. +- **Registry in the user's home or the OS temp dir.** Survives across runs nobody needs, and the reviewer and a successor Athanor could not find it next to the run log. + +--- + +## Decisions this design made that the goal did not settle + +1. **Skill name** `forks`; helper in **Node** (no jq, no Python here). +2. **State per run**, in a `-forks/` folder beside the run log; raw streams gitignored. +3. **Usage reaches the run log by paste, not by append**: the helper writes the ledger and renders markdown rows; the Athanor pastes them. Deviates from the goal's literal "writes them to the run log" to avoid interleaved writes. +4. **Launch flags supplied once** at `init`/`train`; forks take only base, block and brief. +5. **Defaults**: `dontAsk`, tools `Bash,Read,Edit,Write,Glob,Grep,Skill` (no Agent), `--strict-mcp-config` with no servers, explicit `--effort` (default `high`), `CLAUDE_CODE_PROMPT_CACHE_TTL=1h`, `DISABLE_AUTOUPDATER=1`, binary `$CLAUDE_CODE_EXECPATH` first. +6. **Coldness never triggers a re-train**; pings only while the Athanor is in its turn, capped at the break-even count; numeric rebase and re-train thresholds as starting values. +7. **Trunk notes are written by the base** at training, not the Athanor. +8. **The Athanor's diff read for loosened assertions becomes a mechanical count**; the phase reviewer keeps the real check. +9. **Commit ownership**: sequential forks commit after calibrating; parallel forks never commit, the Athanor commits them by explicit paths after the gate. +10. **The Athanor no longer builds blocks inline.** +11. **The trial needs a Phase 0 preflight** on a scratch repo; failures in R1, R2 or R5 stop the plan. + +--- + +## Amendment 2026-10-16 01:33 — runtime (user) + +The user decided that the helper is **Python, run with uv** (`uv run`, single file with PEP 723 inline metadata, standard library preferred), not Node. uv 0.5.13 is installed at `~/.local/bin/uv`; `python` is not on PATH, so it is always invoked through `uv run`. Follow the conventions in `plugins/project-uv/`. The rest of the design stands. The planner was told mid-run. Source: [session journal](../journals/2026-10-15-21-grotto-work.md). diff --git a/docs/workpad/goals/headless-forks.md b/docs/workpad/goals/headless-forks.md new file mode 100644 index 0000000..90dce88 --- /dev/null +++ b/docs/workpad/goals/headless-forks.md @@ -0,0 +1,53 @@ +# Goal: Trained implementer, forked per block — headless forks inside Spire (version TBD) + +**Set**: 2026-10-16, Igor session [2026-10-15-21-grotto-work](../journals/2026-10-15-21-grotto-work.md) +**Relates to**: [lean-athanor-0.2.1](./lean-athanor-0.2.1.md) — its "Trunk and forks is tested, not assumed" criterion. This goal restores the user's original shape, which 0.2.0 lost (see provenance report below). +**Inputs**: +- [Athanor trunk/fork provenance](../reports/2026-10-15-23-athanor-trunk-fork-provenance.md) +- [Headless fork sessions: mechanics + follow-up](../research/2026-10-16-00-claude-headless-fork-sessions.md) +- [Spire vs. superpowers experiment](../reports/2026-10-15-22-spire-vs-superpowers-plan-and-build-experiment.md), §5 (accounting gaps, inline blocks) + +## Intent (user, 2026-10-15/16) + +The Athanor learns plans, design and implementer reports, not code. For hard work it trains **one** implementer early with background information, then **forks that implementer** once per block or subsection. The forks need only a small "you are continuing" brief. Bookkeeping (cache lifetime, which base is warm, tokens) is deterministic, done by a tool and not by agent attention. Solve it inside Spire now, without waiting for Dirigent. A Dirigent harness or MCP tool can come later and take this over. + +## Success criteria + +### Mechanism +- [ ] Spire ships a small deterministic helper (a script in the skill; MCP is a later option) that owns the `claude -p` command line. It covers: train a base, fork the base with a brief, update the base, report usage, and list or expire bases. +- [ ] Each base is stored with its full launch fingerprint: binary path, full model id, flags, permission mode and tools, cwd, agent file. Every fork reuses that fingerprint exactly, because any difference loses the prompt cache (research F-table). +- [ ] The environment is scrubbed of Desktop-host variables before launch; auth works without `--bare` or with an explicit key (user's choice). +- [ ] Per call, the helper returns the real model (`modelUsage`, not the init event), the tokens per usage column, and the per-fork delta cost. It writes them to the run log, so per-block cost is known, including for work the Athanor would previously have done inline. + +### Bookkeeping +- [ ] The registry records the last cache hit per base. "Keep warm / let expire / re-train" is computed from the cache lifetime (1 h on a subscription within plan usage, otherwise 5 min; a hit resets it) and the base's training cost. It is not left to the agent's judgment. + +### Process +- [ ] The Athanor procedure (athanor.md, implementation SKILL.md) describes the trained-implementer shape. In it, the Athanor holds the plan, design, trunk notes and reports, and does not read code itself. +- [ ] Forks don't see each other's work. The procedure says who keeps the base current: re-basing it with a summary of what changed between blocks, or a delta brief for each fork. It also says when each applies. +- [ ] A project agent file can specialize the trained implementer (`--agent` / `--agents`). +- [ ] The spawning reference gets the verified rows from today's research (Agent-tool fork copies the caller only; forks can't fork; fork mode makes everything background; headless fork mechanics; nested-process depth is per process and fails silently). + +### Proven, not assumed +- [ ] Trial on a real multi-block plan comparing the trained-implementer forks against the current shape on quality, tokens (all four columns, priced) and wall time, with every orchestrator counted, including the root. + +## Open decisions (user) + +- Where the helper lives: the implementation skill (assumed from "inside the skill"), or its own Spire skill. +- Update the base vs. delta brief per fork as the default. +- Auth: OAuth with a scrubbed env vs. `--bare` + API key. +- Whether headless forks replace Agent-tool fork options in athanor.md or sit beside them. +- Plan mode upgrades Haiku to Sonnet silently: accept as a floor, or always pass an explicit model. + +### Decided 2026-10-16 (user, session [2026-10-15-21-grotto-work](../journals/2026-10-15-21-grotto-work.md)) + +- **Location**: its own Spire skill, which the implementation skill references. This makes it easier to lift into Dirigent later. +- **Fork sync**: by default each fork gets a delta brief (what changed since the base). At phase boundaries, or when the delta grows, the Athanor updates the base and forks from the new one. +- **Auth**: OAuth with a scrubbed env (subscription → 1 h cache lifetime). No `--bare`. +- **Agent-tool forks**: replaced. Complex blocks go to forks of the trained implementer, and fresh Agent-tool agents stay for mechanical or distant blocks. The Agent-tool fork path is dropped from athanor.md. +- **Model**: always an explicit full model id. The fingerprint requires it anyway, which also sidesteps plan mode's silent Haiku→Sonnet upgrade. *(Igor's default, not asked; user may override.)* + +## Later + +- Dirigent harness / MCP tool takes over the helper's job ("stable for the constables"). +- Token-count access for agents beyond the helper's per-call returns. diff --git a/docs/workpad/journals/2026-10-15-21-grotto-work.md b/docs/workpad/journals/2026-10-15-21-grotto-work.md index 6435295..71eb8cf 100644 --- a/docs/workpad/journals/2026-10-15-21-grotto-work.md +++ b/docs/workpad/journals/2026-10-15-21-grotto-work.md @@ -62,6 +62,43 @@ - Desktop env: removing 12 vars with `env -u` drops the Desktop tools. `--bare` needs an API key, since it refuses OAuth. The messaging socket is the nested run's own pipe; whether it can reach the host is unverified. - Cache TTL (docs): 1 h for main conversations, incl. `-p` and `--fork-session`, on a subscription within plan usage; otherwise 5 min, which also covers Agent-tool forks and subagents. Configurable via CLAUDE_CODE_PROMPT_CACHE_TTL / CLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL. A hit resets the timer. Forks must run from the same cwd. - Version: two installs (~/.local/bin 2.1.283 vs the Desktop's $CLAUDE_CODE_EXECPATH 2.1.299). Use one binary and a full model id for the base and all forks. +- Committed + pushed a7ce5db (provenance report, fork research, journal), via a Haiku 5.5 subagent. +- User idea (thinking aloud, not decided): a lightweight harness, possibly an MCP tool, that manages prepped headless sessions and forks them. Agent files allow specialized implementers. Plan mode implies at least Sonnet. Bookkeeping (the 1 h TTL) must be deterministic, hence "stable for the constables", something that manages the harnesses. Prepped agents would need only a small "you're continuing" update. The user has already considered building it in Dirigent, along with token-count access for subagents (CC doesn't let an agent read its own tokens), which makes management deterministic and frees agent attention. +- Igor's points raised: + - The cache key is the full launch fingerprint (binary, full model id, flags, cwd, system prompt), so the tool must own the command line. + - Forks don't see each other's work, so the base needs periodic re-basing ("updated by orchestrator", user's 10-05 wording), or each fork needs a delta brief. + - Keep-warm vs re-train is a computable decision. + - Per-call usage from the json output solves the experiment's accounting gaps. + - The env scrub and auth (--bare needs an API key) belong in the tool. +- User: solve it now inside the skill, without Dirigent. Igor wrote goals/headless-forks.md: the user's intent, success criteria (helper script owning `claude -p`, fingerprint, env scrub, usage returns, deterministic TTL bookkeeping, Athanor procedure, rebase vs delta brief, spawning-reference update, trial) and open decisions. Igor assumed "the skill" means the implementation skill; flagged as an open decision. +- 01:23 — Open decisions settled by the user (via questions): own Spire skill; delta brief per fork, rebase per phase; OAuth with a scrubbed env; Agent-tool forks replaced by headless forks (fresh agents stay for mechanical blocks). Igor default: always an explicit full model id. Recorded in the goal's "Decided 2026-10-16" section. +- Dispatched the design (stage 3): general-purpose, model: opus, independent → designs/2026-10-16-01-headless-forks.md. Separation: the designer doesn't plan; the planner will be a fresh `spire:planner`. +- Design back: designs/2026-10-16-01-headless-forks.md (claude-opus-5-5). + - Shape: a `spire:forks` skill that the Athanor preloads. `forks.mjs` is a single dependency-free Node file (Node 24 is present; jq and python are missing). State lives per run next to the run log. The fingerprint is set at train time and checked against each init event. Numeric keep-warm/rebase/re-train rules. Detached supervisor + foreground `wait` ≤540 s. + - Roles: the trained base writes the trunk notes, and the Athanor reads no code and builds nothing inline. Sequential forks test and commit; parallel forks never commit (the Athanor commits by path). + - Decisions the goal didn't settle: name and runtime; usage reaches the run log by paste (one writer); flags only at train time; leaf forks with no Agent tool; `$CLAUDE_CODE_EXECPATH` binary first (the PATH 2.1.283 can't run Sonnet 5.5); starting thresholds; the trial must not be this skill's own build. + - Top risks: R1 detached survival on Windows; R2 cache on 100k+ tool-using bases; R5 usage semantics; several untried flags. Price ratios assumed. +- User accepted the design together with its own decisions. Dispatched `spire:planner` (pin, model omitted) → plans/2026-10-16-02-headless-forks.md. The brief applies today's lesson: open nested CLAUDE.md files with Read and include a Conventions section. Note: the commit-attribution policy changed per the system; no attribution lines from now on. +- 01:33 — User: the helper should use uv. Told the running planner (Python via `uv run`, PEP 723 single file, follow plugins/project-uv). Appended a dated runtime amendment to the design. uv 0.5.13 is present. +- User long-term idea: a dedicated Rust app, distributable as a CLI and absorbable as a Dirigent default feature; a few existing tasks would fit. Dispatched a researcher (opus) for an independent report on the three forms (CLI / Dirigent feature / MCP face) plus candidate tasks → reports/2026-10-16-01-session-harness-app-forms.md. +- User: if the attack finds nothing major, go straight from plans into implementation, record unclear things, iterate afterwards; commit and push at the end. For the Rust report: the main thing is the token MCP and lightweight agent access via Dirigent; bundling ideas welcome (forwarded to the researcher). Bypass mode is on: keep using Read/Write/Edit rather than Bash for files, and tell subagents the same. +- Rust app report back: reports/2026-10-16-01-session-harness-app-forms.md (claude-opus-5-5; it delivered the file but no hand-back message). + - The helper's JSON contract is the seam. + - Dirigent already has most of the Rust parts (dirigent_anth transcript parsing, dirigent_process with Job Objects, taskrunner/logs, the fermata three-layer precedent). + - The forms differ in who owns time. + - Token MCP: the hard part is identity, not counting. It's read-only and can ship first. + - 8 candidate tasks (C1–C8, to confirm) and 4 bundles (A tokens, B stable, C probe, D Dirigent access), plus the researcher's proposals: a TOML profile format, a frozen JSON contract spec, a budget guard, a selftest on CLI update, a name split (stable / ledger). +- Plan back: plans/2026-10-16-01-headless-forks.md (planner adjusted the name; stamp claude-opus-5-5[1m]; no hand-back message, file read directly). + - Python/uv integrated (PEP 723, stdlib, supervisor via sys.executable, extra env scrub of VIRTUAL_ENV, UV_* and the PATH prefix). + - Phases: P0 preflight stop gate (uv, R1 detach, ...), P1 helper core, P2 bookkeeping, P3 skill + procedure text + release, P4 install smoke (needs user), P5 trial (Igor/user). + - Open decisions D1–D5. Igor took the defaults per the user's "proceed unless major": D1 0.3.0, D2 changelog in the release commit body, D3 trial deferred, D4 Igor follow-ups out of scope, D5 refuse at init. **Recorded as unclear things for later iteration.** +- Dispatched `spire:attacker` (pin) → plans/2026-10-16-02-headless-forks-build.md. Brief: conventions pass, precedent search, Read tool hygiene, no attribution lines; the build covers P0–P3. +- Attack back: plans/2026-10-16-02-headless-forks-build.md (claude-opus-5-5[1m]). + - No STOP-level findings. + - Top 5: orphaned forks after the Athanor dies (fix: 20 min orphan kill); `wait` dies at the 120 s Bash default (needs timeout 600000); Windows rename/handle races in the registry; the P0.6 TTL test unpassable as written (now a ratio test, after P0.1); forks could launch forks (env mark + refuse). + - Also: B0 commits the docs on main first, then a worktree `.claude/worktrees/headless-forks`. The harness flagged a "settings-json" pattern in the report: it's only a mention of `.claude/settings.json` (U4), no instruction. + - Open, for the user before Phase 4: U1 test via `--plugin-dir` vs direct update; U2 trial plan + arm C; U3 who merges and pushes; U4 Sonnet/Haiku ids in settings; U5 defaults (20 min orphan kill, $25 Phase 0 spend cap, worktree name). + - **Igor decision for U3 within the user's "commit and push":** push the branch, don't merge to main. Merging installs 0.3.0 for every project via the marketplace. Recorded as unclear for the user. ## Adepts diff --git a/docs/workpad/plans/2026-10-16-01-headless-forks.md b/docs/workpad/plans/2026-10-16-01-headless-forks.md new file mode 100644 index 0000000..23b5b07 --- /dev/null +++ b/docs/workpad/plans/2026-10-16-01-headless-forks.md @@ -0,0 +1,257 @@ +# Plan: Trained implementer, forked per block (`spire:forks`) + +**Date**: 2026-10-16 01:38 +0200 +**Planner**: model id `claude-opus-5-5[1m]` (stage 4; does not attack, build or review) +**Goal**: [goals/headless-forks.md](../goals/headless-forks.md), including "Decided 2026-10-16" +**Design**: [designs/2026-10-16-01-headless-forks.md](../designs/2026-10-16-01-headless-forks.md), plus the coordinator's runtime override of 2026-10-16 (Python under `uv run`, not Node; a dated amendment goes into the design file separately) +**Evidence**: [research/2026-10-16-00-claude-headless-fork-sessions.md](../research/2026-10-16-00-claude-headless-fork-sessions.md) (claim ids such as `3a`, `F4c`), [reports/2026-10-15-23-athanor-trunk-fork-provenance.md](../reports/2026-10-15-23-athanor-trunk-fork-provenance.md) +**Filename**: the brief named `-02-`; the hour at writing was 01, so the workpad rule (`plugins/spire/skills/workpad/SKILL.md:72`) gives `2026-10-16-01-headless-forks.md`. +**Next**: attack (stage 4b), which writes the build plan. This plan is not the build plan. + +The design stands except for the runtime. Section numbers like "design §5" refer to the design file. Where this plan adds something the design does not have, it says **Planner addition** and gives the reason. + +--- + +## 1. Conventions this plan relies on or could break + +| Rule | Where | How the plan meets it | +|---|---|---| +| Gitmoji prefix on every commit, format ` ` | `CLAUDE.md:7`, `docs/workpad/specs/commit-conventions.md:3-5` | Per-step prefixes in §7. | +| `🤖` for agent files: CLAUDE.md, skills, AGENTS.md | `commit-conventions.md:91` | Everything under `plugins/spire/skills/` and `agents/` is committed as `🤖`, including the skill's script and tests (they live inside a skill). Workpad reports use `📝`. | +| Each plugin is self-contained | `CLAUDE.md:3` | Helper, templates and tests all stay inside `plugins/spire/skills/forks/`. Nothing goes to the repo root. No dependency on `plugins/project-uv/` at runtime. | +| Work on Spire uses Spire; workpad is authoritative for placement | `CLAUDE.md:11`, `:15` | Docs go to `docs/workpad/`. This build runs on the **installed 0.2.2** Athanor shape, because the tool under test cannot build itself (design §10). | +| Workpad naming `YYYY-MM-DD-hh-` for plans and reports; dated subfolder for a cascade | `plugins/spire/skills/workpad/SKILL.md:72`, `:80` | Run log `reports/-headless-forks-run.md`. Preflight report `reports/-headless-forks-preflight.md`. The helper's state dir `…-run-forks/` is a cascade folder (design §3.2). | +| Spire README lists components; `plugins/spire/CLAUDE.md` points to it | `plugins/spire/CLAUDE.md:1`, `plugins/spire/README.md:11-21` | Step 3.4 adds the `forks` row. | +| Version lives in **two** manifests, bumped together | `plugins/spire/.claude-plugin/plugin.json:3`, `plugins/spire/.codex-plugin/plugin.json:3`; precedent `7c1ed39`, `f40cbf5` | Step 3.4 bumps both. Marketplace files (`.claude-plugin/marketplace.json`, `.agents/plugins/marketplace.json`) carry no version, so they stay as they are. | +| No CHANGELOG file exists. Release notes live in the release commit body | `git ls-files` (no changelog); `7c1ed39` body | Step 3.4 writes the changelog as the body of the release commit. See open decision D2. | +| LF line endings for `*.md`, `*.py`, `*.json` | `.gitattributes:1-5` | `forks.py`, templates, registry and ledger are written with `\n`. The helper writes its files in binary or `newline="\n"` mode on Windows. | +| `**/*.log` and `**/*.tmp` are gitignored repo-wide | `.gitignore:9-10` | **Planner addition:** evidence and gate output under the state dir's `logs/` uses `.txt`, so it is not hidden silently. Temp-then-rename files may use `.tmp`. | +| uv conventions: PEP 723 scripts run as `uv run path/to/script.py`; `SCRIPTS_UV` names the uv binary; probes are stdlib-only | `plugins/project-uv/skills/project-uv/SKILL.md:21`, `:31`; `scripts/probe.sh:6-11`; `scripts/probe.py:4` | `forks.py` is a single file with a PEP 723 block and stdlib only. It is invoked as `uv run /forks.py …`. `$SCRIPTS_UV` is honoured as described in §3.1. | +| Skill shell injection plus a pre-approving `allowed-tools`, a pattern its own comment calls novel | `plugins/spire/skills/adepts/SKILL.md:4`, `:7-12` | `forks/SKILL.md` follows it with `Bash(uv:*)`. Phase 4 verifies it, and §6 marks it **Assumed**. | +| Reliquary already allows `Bash(uv:*)`; `rm`/`mv`/`git push` are `ask`; `gh`/`ssh` are denied | `.claude/settings.json:32`, `:53-54` | Under `dontAsk`, forks get `rm`/`mv` denied. Denials surface as `needs_attention` (design §3.5). Briefs tell forks to use `git rm`/`git mv`. | +| Project `availableModels` lists only Opus ids and aliases, no Sonnet or Haiku full id; `effortLevel: medium`; `model: claude-opus-5-5[1m]` | `.claude/settings.json:56-69` | Headless runs load project settings (no `--bare`, `5a`). P0.5 tests whether a full id outside the list is refused. See D5. | +| Athanor: goals are read-only; stage by explicit paths; no PRs; no force-push | `plugins/spire/agents/athanor.md:53`, `:55` | No step edits `goals/`. Igor ticks the goal's boxes. All commits stage explicit paths. | +| Every plan, handoff note and review stamps its exact model id | `plugins/spire/skills/implementation/SKILL.md:68` | Header above. The helper stamps the real model from `modelUsage` (design §6). | +| One phase per turn, cheap gate between blocks, tests once per phase, fresh phase review | `implementation/SKILL.md:36-58`, `athanor.md:27-40` | Phases in §7. | +| Spawning reference: status vocabulary, and "when a claim fails, fix it here with a new date" | `implementation/references/spawning-claude-code.md:5-12` | Step 3.3 updates rows with dated statuses. Old rows are replaced only where design §9 says so. | +| Model notes are added with a date, not silently dropped | `implementation/references/models.md:3` | Step 3.3 appends dated notes. | +| No `## Build` section in `.claude/spire/project.md` | `.claude/spire/project.md` (absent) | Defaults from the implementation skill apply. This plan names its own gate commands. | +| Root `AGENTS.md` still points at `g4b_ai` | `AGENTS.md:9-15` | Stale and Codex-facing. Out of scope (§5). | + +--- + +## 2. Runtime change: Python under uv (coordinator override, 2026-10-16) + +This replaces design §3.1. Everything else in design §3 stays. + +- **Decision:** the helper is `plugins/spire/skills/forks/forks.py`, a single file, standard library only. *Why:* the user decided it. `python` is not on PATH here, but uv 0.5.13 is (`~/.local/bin/uv`), and it manages CPython 3.9 to 3.13 locally. +- **Decision:** the PEP 723 header has exactly this form. Its exact form is the decision. + + ```python + # /// script + # requires-python = ">=3.10" + # dependencies = [] + # /// + ``` + + *Why:* stdlib-only means `dependencies = []`. The floor is kept low so that any uv-managed Python on this host (3.10 to 3.13) qualifies; 3.9 is past end of life. Nothing needs `tomllib`. Raise the floor only with a stated reason. +- **Invocation contract:** always `uv run [args]`, from any cwd. With inline metadata, uv must ignore any `pyproject.toml` in the cwd. P0.0 verifies this. +- **Decision:** the detached supervisor is spawned as `sys.executable _supervise `, not through `uv run` again. *Why:* it skips a second uv resolution and keeps uv's own wrapper process out from between the supervisor and its launcher, which is one fewer layer to detach from (R1). +- **Planner addition, env scrub for uv:** `uv run` changes the environment the helper sees. The child `claude` (and every fork's Bash) must not inherit the helper's Python environment. On top of design §4, the scrub therefore drops `VIRTUAL_ENV` and every `UV_*` variable that uv injects, and removes the PATH entries that point inside `sys.prefix`. P0.0 measures the exact list. *Why:* otherwise a fork running `python` or `uv run pytest` in a Python project would resolve to the helper's interpreter. +- `where` reports the uv version, the Python version and path, the script path, the resolved `claude` binary and its version, and the scrub check. Design §2 had node in this slot. +- `$SCRIPTS_UV` (project-uv contract) is honoured by the **Athanor's calls**: SKILL.md says to use `$SCRIPTS_UV` if it is set, else `uv`. The injection line uses plain `uv`. **Assumed**, see §6. +- Design R14 ("node may be missing") becomes "uv may be missing, or may be unable to provide Python ≥ 3.10 (offline, no managed Python)". `where` fails loudly, and the Athanor falls back to the fresh-agent path and logs it. + +--- + +## 3. Decisions the builder cannot make alone + +Design §"Decisions this design made" 1–11 stand, with 1 amended by §2 above. On top of those: + +1. **Phase 0 stop rule.** If P0.0 (uv), P0.1 (R1), P0.2 (R2) or P0.3 (R5) fails, the plan stops. The Athanor hands back with `STOP` and the preflight report, and Igor takes it to the user. Nothing in `plugins/` changes before Phase 0 passes. *Why:* design §10 and §11 "what would make this wrong". Adding uv as a stop item follows the coordinator's message. +2. **Phase 0 runs from the Athanor's own Bash, as an Agent-spawned subagent, on this Windows host.** *Why:* R1 is about exactly that configuration. A researcher run from the root would test the wrong process tree. +3. **Phase 0 probes are throwaway** and live in the session scratchpad or a scratch clone, not in `plugins/`. Their results get copied into the preflight report. The Phase 1 builder may reuse their code. +4. **Scratch target for every live test** (P0, the Phase 1/2 smokes, Phase 4) is a `git clone --local` of this repo in the scratchpad. It is never the working checkout. *Why:* forks commit, and P0.2 needs ≥ 100k tokens of real reading, which this repo provides. The clone carries `.claude/settings.json`, so P0.5 tests the real `availableModels` behaviour too. +5. **Tests use a stub `claude`** that replays canned `stream-json` (init, assistant events, result) and can be told to hang, die, or emit denials. No unit test spends tokens. The builder chooses how the stub is made executable on Windows. The helper gains no test-only flags beyond what `--binary` already gives. Time is injected by monkeypatching one clock function in the imported module. +6. **Tests live at `plugins/spire/skills/forks/tests/test_forks.py`**, a PEP 723 script using `unittest`, run as `uv run plugins/spire/skills/forks/tests/test_forks.py`. *Why:* self-contained plugin, and it lifts with the skill into Dirigent. +7. **Planner addition: supervisor liveness.** The supervisor writes a heartbeat (`heartbeat_at` in its job file) at least every 30 s. If `wait` or `list` sees a running job whose heartbeat is older than 120 s and whose supervisor pid is gone, it marks the job `lost` and appends a ledger line with whatever usage was captured. *Why:* the design has no answer for a supervisor killed by a job object or a reboot. Without one, `wait` returns exit 3 forever. +8. **Planner addition: `registry.json` carries `"schema": 1`.** *Why:* the state format is the hand-off point to a later Dirigent or MCP tool. +9. **The release is a minor bump, 0.2.2 → 0.3.0**, in both manifests. Smoke fixes in Phase 4 bump the patch. *Why:* the Athanor's procedure changes incompatibly (no code reads, Agent-tool forks dropped), and the plugin cache is keyed by version. See D1. +10. **Owner models (defaults for the attacker to reshape).** P0: the Athanor itself, `claude-opus-5-5[1m]`. Phases 1–2 (process supervision, Windows, state machine): Opus 5.5. Phase 3 procedure text (athanor.md, implementation SKILL.md, forks SKILL.md): Opus 5.5. Phase 3 reference rows and README: Sonnet is acceptable. + +--- + +## 4. Contracts and invariants + +### 4.1 Command surface (design §3.4, unchanged except `where`) + +`where`, `init`, `train`, `fork`, `rebase`, `continue`, `wait`, `usage`, `list`, `expire`, `kill`, internal `_supervise` and `tick`. Every command except `where` takes `--run `. Exit codes: `0` ok, `2` usage error or refused, `3` `wait` timed out with jobs still running, `4` a job failed. + +**Invariant:** stdout carries exactly one JSON object, or for `usage --format md` exactly one markdown fragment. Diagnostics go to stderr. uv's own output goes to stderr too, and P0.0 confirms it. + +**Path contract (planner addition):** `--run`, `--brief` and `--cwd` accept relative, Windows (`G:\…`, `G:/…`) and Git Bash (`/g/…`) forms. The helper normalises them before storing and hashing. *Why:* the Athanor's Bash is Git Bash, while `forks.py` runs on native Windows Python. The transcript slug is derived from the normalised Windows path (`6d`, `7`). + +### 4.2 Result object (what `wait` returns per finished job) + +This is the contract between Phase 1/2 (helper) and Phase 3 (procedure text). Field names are fixed. Values follow design §6. + +```json +{"job":"j0007","kind":"train|fork|rebase|continue|ping","base":"B1","block":"1.3","phase":1, + "status":"ok|failed|timeout|killed|lost", + "flags":["stalled","needs_attention","usage_mismatch","ttl_downgrade","cold_start"], + "fingerprint":"ok|drift:,", + "session":"","parent_session":"|null", + "models_used":["claude-…"],"tokens":{"input":0,"output":0,"cache_creation":0, + "cache_creation_1h":0,"cache_creation_5m":0,"cache_read":0}, + "cost_delta_usd":0.0,"cache":"hit|partial|miss","base_state":"warm|expiring|cold", + "wall_s":0,"turns":0,"denials":0,"note":"notes/1.3.md|null","raw":"raw/j0007.jsonl"} +``` + +`fork` additionally returns `{job, base, base_state, warm_until, rebase_due, retrain_due, reasons[]}` at launch (design §3.4). The ledger line is design §6's shape plus `status` and `flags` as above. + +### 4.3 State machines + +``` +job: created ─► running ─► ok | failed | timeout | killed | lost + (flags may be set while running: stalled, needs_attention) +base: training ─► active ─► superseded (rebase / re-train) | retired (expire) + warmth (computed, never stored as truth): warm | expiring | cold — design §5 +``` + +**Invariants:** + +- Only `train` and `init` accept launch fields. `fork`, `rebase`, `continue` and `ping` read them from the base record. Overrides are refused with exit 2 (design §3.3). +- A `fork` from a base that is not `active` is refused without `--force`. A cold base is forkable. **Coldness never triggers a re-train** (design §5). +- Registry writes happen under `registry.lock` (create-exclusive, stale after 30 s), written to a temp file and then renamed. The ledger is append-only. Nothing else writes either file. +- At most `parallel_cap` jobs of kind `fork`/`continue` run per run (default 2). Pings and the tick do not count toward it. +- `init.model` is never recorded as the model. `models_used` comes from the `modelUsage` delta (design §6). +- `total_cost_usd` is never summed. Cost is always a delta against the session total before the call (`3d`, `F4e`). +- Every launch re-checks binary version, size and mtime. A change sets a `binary` drift and `rebase_due` (design §5). +- The env scrub is one constant (design §4 plus §2 above). `where` lists host `CLAUDE_CODE_*` variables that are on neither the drop list nor the keep list. + +### 4.4 Commit ownership (design §8.3, unchanged) + +There is one committer at a time. Sequential forks commit their block by explicit paths after their gate passes. Parallel forks never commit; the Athanor commits each block afterwards by the paths its note lists. **Planner addition:** the Athanor commits the run log and the state dir (everything except `raw/` and lock files) by explicit paths at phase close, as `📝`. + +--- + +## 5. Scope fence + +Out of scope: + +- An MCP server, a Dirigent harness, and token-count access beyond the helper's per-call returns (goal "Later"). +- `--bare` and API-key auth (goal decision: OAuth with a scrubbed env). +- Agent-tool forks. They are removed from athanor.md, not kept beside the new path (goal decision). +- Editing `plugins/spire/skills/igor/SKILL.md`. Line 66 ("forking yourself") is flagged by the design. **Planner flag:** line 112 has Igor `TaskStop` an Athanor, which would leave its detached forks running. Both go to Igor as follow-ups, see D4. +- Codex support for `spire:forks`. The skill says it is Claude Code only. `.codex-plugin/plugin.json` gets only the version bump. +- Worktree-per-fork, cross-cwd forks, and auto-tuning of thresholds. The trial tunes them by hand (design §5). +- A CHANGELOG file (see D2), root `AGENTS.md`, other plugins, and the planner, attacker and reviewer agent files. +- Editing `goals/` (Igor's) or the design file (the coordinator is amending it). +- The other criteria of goal `lean-athanor-0.2.1`, beyond what the trial answers. + +--- + +## 6. Assumed + +- **Assumed:** the preloaded skill's `!` injection runs at agent start under Windows and resolves `${CLAUDE_SKILL_DIR}`, and `allowed-tools: Bash(uv:*)` pre-approves the Athanor's later `uv run …/forks.py` calls (design §2, R4). Default if not: the Athanor's brief says to rely on the project allow rule (reliquary has one at `.claude/settings.json:32`), and SKILL.md tells users of other projects to add `Bash(uv run:*)`. This is checked in Phase 4, the first point where the installed plugin exists. It is not a Phase 0 stop item. +- **Assumed:** `$SCRIPTS_UV` is honoured in SKILL.md's instructions, not in the injection line. +- **Assumed:** evidence and gate logs under the state dir are committed, as design §3.2 gitignores only `raw/` and locks, and they use `.txt` (because of `.gitignore:9`). +- **Assumed:** Phase 0's real token spend is acceptable: a ≥ 100k base, about 8 forks, and the TTL test. It is notional under the subscription but draws on plan usage (R11). +- **Assumed:** if P0.6 (TTL refresh) fails, pings default to off (`init --pings off`) and the keep-warm rule ships disabled. If P0.4 (parallel) fails, `parallel_cap` defaults to 1. +- **Assumed:** the build runs on a branch named by Igor's brief, default `headless-forks`, not on `main`. +- **Assumed:** after Phase 3 the user updates the installed plugin (marketplace update, reload) before Phase 4. The plan cannot do this itself. +- Design-level assumptions (`[1m]` in `--model`, `Skill` in `--tools`, `apiKeySource`, `--strict-mcp-config` dropping claude.ai connectors, `--agent` resolution, `DISABLE_AUTOUPDATER`, transcript message usage, message ids kept in fork copies) are tested in P0.5/P0.7, except `DISABLE_AUTOUPDATER`, which stays assumed. + +--- + +## 7. Phases + +Gates used below: + +- **G-unit**: `uv run plugins/spire/skills/forks/tests/test_forks.py` (exit 0). +- **G-where**: `uv run plugins/spire/skills/forks/forks.py where` (exit 0, valid JSON). +- **G-text**: the grep checks listed in Phase 3. + +### Phase 0: Preflight (stop gate) + +Owner: the Athanor, from its own Bash. Target: the scratch clone (decision 4). Output: `reports/-headless-forks-preflight.md` with, per check: command shape, raw numbers, pass or fail, and the Phase 1 default it sets. The raw stream files stay in the scratch dir, and their paths go into the report. Commit the report as `📝` (or `⚗️`). + +| Step | Check | Pass criterion | On fail | +|---|---|---|---| +| P0.0 | **uv.** A PEP 723 probe with the §2 header runs via `uv run ` from the Athanor's Bash. (a) cwd = repo; (b) cwd = a scratch dir holding a `pyproject.toml` with a `[project]` table: uv does not sync or build it. Record the uv version, the Python chosen, first-run and warm latency, whether stdout stays clean, and the env diff between Bash and inside `uv run` (`VIRTUAL_ENV`, `UV_*`, PATH prefix). | Runs in both cwds, stdout holds only the probe's output, and the env diff is recorded. | **STOP** | +| P0.1 | **R1 detach.** From a `uv run` Python process, spawn a detached `sys.executable` supervisor that spawns a short `claude -p` (or a sleeper first, then claude). Windows: try the creation flags with and without breakaway from the job. The supervisor must outlive (a) the uv process and (b) the Bash call, and a later Bash call must see its output growing until it completes. Also test tree-kill. If that fails, run the `--attach` variant under Bash `run_in_background: true` and poll it from later foreground Bash calls for ≥ 15 min. | Either variant survives for ≥ 15 min and completes, and a kill removes the whole tree. Record which variant and which flags. | **STOP** if both fail | +| P0.2 | **R2 large base.** Train a base with real tool use (Read/Grep over the clone) to ≥ 100k context tokens, using the design's default fingerprint: full model id, `--effort`, `dontAsk`, the tool list, strict MCP, `--append-system-prompt-file`, scrubbed env, pinned TTL. Then three **sequential** forks, each reading, editing, gating and committing one small file. | Each fork's **first** assistant event shows `cache_read` ≥ 0.9 × the base's context tokens. Record whether any compaction happened. One re-run is allowed if a probe mistake in the fingerprint is found and named. | **STOP** | +| P0.3 | **R5 usage.** For a fork with ≥ 5 turns: the `stream-json` `result` event has `usage`, `modelUsage`, `total_cost_usd`, `num_turns`, `is_error`, `permission_denials`. Compare the `modelUsage` delta (vs. the base's cumulative), the top-level `usage`, and the sum of per-assistant-event usage. | A reproducible rule gives the four per-call columns (plus the 1h/5m split) and the cost delta. Record the rule; Phase 1 implements exactly it. | **STOP** if the fields are missing or no consistent rule exists | +| P0.4 | **R3 parallel.** Two tool-using forks in parallel on disjoint files, no commits, then the Athanor gates and commits both. | No `index.lock` error, both edits intact, both first requests hit. | `parallel_cap` default 1 | +| P0.5 | **Flags (R8, R9, R15) and models.** Each must be accepted and visible in `init` or behaviour: `--effort` with `CLAUDE_EFFORT` scrubbed (and whether a different effort misses the cache); `--strict-mcp-config` with an empty config drops `mcp__claude_ai_*` tools; `Skill` in `--tools`; `[1m]` in `--model`; `apiKeySource` present (record its value); `--agent ` resolves a project `.claude/agents/.md` together with `--append-system-prompt-file` (a fork quotes one marker string from each); agent `tools:` vs `--tools`; a full Sonnet id **not** in the clone's `availableModels` (`.claude/settings.json:56-67`). | Each item recorded as accepted, refused or ignored, with the default it sets. | Per item: drop the flag or field from the fingerprint and record it. The `availableModels` result feeds D5. | +| P0.6 | **R7 TTL.** Base A: ping at ~50 min, fork at ~65 min. Base B: no ping, fork at ~65 min. Start it first and run it beside P0.2–P0.5. The Athanor keeps polling until it finishes. | A's fork hits and B's misses. | Pings default off | +| P0.7 | **Transcript facts (informational).** Message-level `usage` present in `~/.claude/projects//.jsonl` lines (for `max_base_tokens`); copied base messages keep their ids in fork transcripts (for trial dedupe); `--session-id` combined with `--resume --fork-session` (R6). | Recorded. | `max_base_tokens` falls back to the last result's context estimate; the trial dedupes by content hash | + +**Phase 0 complete when:** the report exists with every row filled and a one-line verdict, `GO` or `STOP`, at the top, and it lists the defaults Phase 1 must use (scrub additions, detach variant and flags, usage rule, parallel cap, pings, fingerprint fields kept or dropped). Phase review: a fresh reviewer checks the report's numbers against the raw files, and Igor makes the GO/STOP call. + +### Phase 1: Helper core (lifecycle, fingerprint, usage) + +Owner: Opus 5.5. Files: `plugins/spire/skills/forks/forks.py`, `preamble.md`, `templates/train.md`, `templates/fork.md`, `tests/test_forks.py` (plus the stub). Commits: `🤖`. + +- **1.1 Skeleton and state.** PEP 723 header, command dispatch, exit codes, the stdout invariant, path normalisation (§4.1), state-dir derivation from `--run`, `registry.json` (`schema: 1`) with lock and atomic write, the state dir's own `.gitignore` (`raw/`, `*.lock`), `mcp-empty.json`, and the `where` and `init` commands. *Done when:* G-where passes on this host, and the unit tests cover path forms, lock contention and stale-lock recovery, init idempotency, and exit 2 on bad input. +- **1.2 Launch and supervise.** Env scrub (design §4 plus §2 plus P0.0's list), the fingerprint (build, hash, launch-time binary re-check, runtime compare against `init` → `drift`), `train`, `fork` (delta header: commits since `base_commit` plus `--shortstat`, and the refusals in design §3.4), `continue`, `_supervise` (the detach variant P0.1 chose, stream parsing, heartbeat, watchdogs `timeout`/`stalled`/`needs_attention`, `lost` per decision 7), results and ledger with the usage rule from P0.3, `wait` (without the tick), `list`, `expire`, `kill` (tree). `preamble.md` and the two templates, per design §2 (stale-snapshot sentence in `fork.md`, R13). *Done when:* the unit tests against the stub cover every job status and flag, the cost delta for fork and continue, the cache verdict, a refusal for each override, the parallel-cap refusal, `lost` detection, and `wait` exit codes 0, 3 and 4. +- **Phase 1 tests and smoke.** G-unit green. Live smoke in the scratch clone: `init` → `train` a small base → two sequential forks that commit → one `continue` → one `kill` of a started fork → `wait` loops → `list`. *Phase 1 complete when:* both forks report `cache: hit` and `fingerprint: ok`, `models_used` equals the requested id, ledger lines match the result objects, `kill` leaves no `claude` process behind, and the smoke's results and ledger are appended to the run log. + +### Phase 2: Bookkeeping (warmth, rebase, usage rendering) + +Owner: Opus 5.5. Same files plus `templates/rebase.md`. Commits: `🤖`. + +- **2.1 Warmth and keep-warm.** TTL inference, `last_hit`, `warm_until`, margin, state (design §5); the tick inside each supervisor (60 s) and inside `wait` (30 s); the ping rule with its activity condition and break-even cap; `ttl_downgrade`; `cold_start`; `init --pings on|off` defaulting per P0.6. *Done when:* clock-injected unit tests cover the warm/expiring/cold boundaries at 1 h and 5 min, ping eligibility (including no pings without activity in the last 10 min and the cap at 19 and 11), and TTL inference from 1h-only, 5m-only and read-only calls. +- **2.2 Rebase and re-train flags.** The `rebase` command (fork → new base with `parent`, old base `superseded`), `rebase_due` (phase boundary via `--phase` change, commit and line thresholds, delta-size estimate, binary drift), `retrain_due` (fingerprint-defining field differs, `max_base_tokens` per P0.7, four rebases in a lineage), thresholds from `init`. *Done when:* unit tests trigger each reason separately, and `fork`'s launch return lists the right `reasons[]`. +- **2.3 Usage rendering.** `usage --block|--phase --format json|md`. The md block row has exactly the columns of the new run-log block table (step 3.3); the phase table gives per-kind totals, cold starts, cache-hit ratio and a priced total. *Done when:* a unit test checks the md against a fixed ledger fixture. +- **Phase 2 tests and smoke.** G-unit green. Live: in the scratch clone, train → fork (phase 1) → fork with `--phase 2` returns `rebase_due` → `rebase --as B2` → fork from B2 hits B2's prefix; let one base sit into `expiring` and observe a ping when pings are on; `usage --phase 1 --format md` renders. *Phase 2 complete when:* all of that is in the run log with the helper's JSON. + +### Phase 3: Skill and procedure text, release + +Commits: `🤖`. The release commit carries the changelog body (step 3.4). + +- **3.1 `forks/SKILL.md`.** Frontmatter per design §2 with `allowed-tools: Bash(uv:*)` and the injection `` !`uv run "${CLAUDE_SKILL_DIR}/forks.py" where` ``. Body: when to use (the Athanor, complex blocks), the command contract (§4.1), the result object (§4.2), the rules the Athanor obeys (flags, not judgment; nothing running at hand-back; never resume or fork a headless session by hand; `$SCRIPTS_UV`), the fallback when `where` fails, and a Claude-Code-only note. Short: the design says "Short". *Done when:* every command and result field named matches `forks.py --help` and §4.2. +- **3.2 `agents/athanor.md`.** All of design §8.4's athanor bullets: `skills:` gains `spire:forks`; intake no longer reads code and trains the base instead (the base writes the trunk notes); step 2 routes complex blocks to headless forks and mechanical or distant ones to fresh agents; no inline building; keepers gain "you do not read code", "nothing running at hand-back" and the commit table (§4.4); "read the fix's diff for loosened assertions" (`athanor.md:34`) becomes the mechanical check of design §8.2 step 5; the deliverable gains usage totals and base state. The Agent-tool fork path is removed. *Done when:* G-text passes (below). +- **3.3 Implementation skill and references.** `implementation/SKILL.md:45-49` rewritten as "trained implementer and forks" (design §8.4), and line 65ff gains "headless forks take a full model id". Line 12's inline rule stays for non-Athanor use. `references/spawning-claude-code.md`: Fork-table rows and the new "Headless sessions (`claude -p`)" section exactly per design §9, each row with research id and date 2026-10-16, plus the models and rules-of-thumb lines, plus two rows from Phase 0 (detach behaviour on Windows from P0.1, uv env injection from P0.0) marked `observed 2026-10-16 (preflight report)`. `references/run-log.md`: the `Forks:` header line, the new block table `Block | Owner (model id from modelUsage) | Session | Commit | in | out | cache write | cache read | cost Δ`, the phase-close usage table, and "Trunk notes (written by the base at training)". `references/models.md`: dated notes per design §8.4. *Done when:* G-text passes, and every new spawning row has a status and a date. +- **3.4 README, version, release.** `plugins/spire/README.md` skills table gains `forks`. Both manifests go to `0.3.0` (D1). Release commit `🤖 Spire 0.3.0: trained implementer forked per block (spire:forks)`, whose body is the changelog, as in `7c1ed39`: new skill, Athanor procedure change, Agent-tool fork dropped, references updated, preflight report path. *Done when:* both manifests show the same version and the README row exists. +- **G-text** (run at phase end): + - `grep -n "fork of yourself\|That reading is your trunk\|read the fix's diff" plugins/spire/agents/athanor.md` returns nothing; + - `grep -n "spire:forks" plugins/spire/agents/athanor.md` hits in the frontmatter; + - `grep -n "a fork of the builder" plugins/spire/skills/implementation/SKILL.md` returns nothing; + - the `version` values in both manifests are equal; + - every command name in `forks/SKILL.md` appears in `forks.py`; + - G-unit and G-where still pass. + + *Phase 3 complete when:* G-text is green and a fresh reviewer has checked the procedure text against the helper's actual JSON and exit codes. The review explicitly covers contract drift between Phases 1–2 and Phase 3. + +### Phase 4: Install and smoke (needs the user to update the plugin) + +Entry: the user has updated and reloaded Spire 0.3.0. Owner: Igor dispatches a fresh 0.3.0 Athanor on a toy plan in the scratch clone: 2 phases, 3 complex blocks in one area plus 1 mechanical block. + +- **4.1** The Athanor's start shows the `where` injection output, and its first `forks` calls run without a permission prompt (the R4 **Assumed** above). Record the outcome in the spawning reference only if it differs from the assumption. +- **4.2** The procedure is followed: the base writes the trunk notes; complex blocks go to forks with delta briefs; the mechanical block goes to a fresh agent; phase 2 starts from a rebased base; the run log carries `usage` rows; `list` shows nothing running at hand-back. The Athanor's transcript shows no `Read` of source files under the clone's `plugins/`. +- *Phase 4 complete when:* 4.1 and 4.2 are recorded in `reports/-headless-forks-smoke.md`. Fixes go in as patch versions (0.3.1 and up), both manifests, `🤖`/`🐛`. + +### Phase 5: Trial (goal "Proven, not assumed"), Igor-owned + +Per design §10 "The trial": arms F and C, sealed inputs, a transcript audit with a positive control, a blind judge, four token columns priced for every agent including the root, wall time from `date` and git, and the arm F mechanism metrics. Output: `reports/-headless-forks-trial.md`. Thresholds in design §5 are retuned from it, as a patch release. Entry needs D3. *Complete when:* the report exists and Igor has put the result in front of the user. Whether the goal box gets ticked is the user's call. + +--- + +## 8. Risks the attacker should test this plan against + +- **R1 under uv.** On Windows, `uv run` may hold its child in a job object, and so may the Bash tool. Breakaway may be refused. P0.1 is designed around this, but a variant that survives the Bash call and then dies at the end of the subagent's turn would be fatal only for hand-backs with running jobs, which the procedure forbids anyway. +- **The env bleed from `uv run`** (§2) also reaches the Athanor's gate commands if they are run through the helper. They are not: the Athanor runs gates in its own Bash. +- **The Phase 3 text and the Phase 1–2 JSON** have different owners. §4.2 is the fixed seam. The Phase 3 review checks it. +- **`availableModels`** (`.claude/settings.json:56-67`) may refuse the owner model a plan names for forks, in this repo and others. D5. +- **The arm C contamination** in the trial (D3). + +--- + +## 9. Open decisions (not settled by the goal, the design or the coordinator) + +- **D1. Version number.** Default `0.3.0` (decision 9). Alternative `0.2.3`. The goal says "version TBD". +- **D2. Changelog form.** No CHANGELOG file exists. The default follows precedent: the release commit body (`7c1ed39`). The alternative is a new `plugins/spire/CHANGELOG.md` starting at 0.3.0. +- **D3. Trial set-up.** (a) Which real multi-block plan; the design says the user picks. (b) How arm C gets the 0.2.2 shape once 0.3.0 is installed, because the preloaded skills would be 0.3.0. Default: run arm C **before** the user installs 0.3.0 (between Phase 3 and Phase 4), and arm F after Phase 4. That means the trial plan must be picked before Phase 4. +- **D4. Igor follow-ups.** `igor/SKILL.md:66` (fork wording) and `:112` (`TaskStop` of an Athanor orphans detached forks; it would need "then run `forks list`/`kill` on its run"). Default: out of this plan, a one-line follow-up for Igor after Phase 4. +- **D5. Models outside `availableModels`.** If P0.5 shows that a full id not in the project's list is refused, should the helper refuse at `init` with a clear message (default), and should the user add full Sonnet or Haiku ids to `.claude/settings.json`? The plan does not touch user or project settings. diff --git a/docs/workpad/plans/2026-10-16-02-headless-forks-build.md b/docs/workpad/plans/2026-10-16-02-headless-forks-build.md new file mode 100644 index 0000000..32006d3 --- /dev/null +++ b/docs/workpad/plans/2026-10-16-02-headless-forks-build.md @@ -0,0 +1,337 @@ +# Build plan: Trained implementer, forked per block (`spire:forks`) + +**Date**: 2026-10-16 01:50 +0200 +**Attacker**: model id `claude-opus-5-5[1m]` (stage 4b; did not write the plan, will not build or review it) +**Plan (as written, unchanged)**: [plans/2026-10-16-01-headless-forks.md](./2026-10-16-01-headless-forks.md). Section refs "plan §N" point there. +**Design**: [designs/2026-10-16-01-headless-forks.md](../designs/2026-10-16-01-headless-forks.md) incl. the 01:33 runtime amendment (Python under uv) +**Goal**: [goals/headless-forks.md](../goals/headless-forks.md) +**Evidence**: [research/2026-10-16-00-claude-headless-fork-sessions.md](../research/2026-10-16-00-claude-headless-fork-sessions.md) (claim ids `1a`…`F5b`) +**Filename**: `-02-` as the coordinator named it (workpad rule 1, explicit instruction). +**Build scope**: Phases 0–3. Phase 4 (install and smoke) and Phase 5 (trial) are post-build steps; they need the user. + +The plan stands except where this document says **Amended** or **New**. Where both say something, this document wins. + +--- + +## STOP-level findings + +**None.** Nothing found blocks the build before it starts. Phase 0 is the stop gate the plan already has, and the findings below go into it or into the contracts. + +These came closest: + +- **A5 (uv and job objects).** On Windows, `uv run` or the Bash tool may hold the helper in a kill-on-close job object that does not allow breakaway. Both of the plan's P0.1 variants would then fail, and the plan would STOP for a reason that has a known workaround. This plan adds that workaround as a rung before STOP. +- **A9 (untracked inputs).** The goal, design and plan are untracked on `main`, so a build branch would start without them. Igor commits them first (pre-build step B0). + +--- + +## 1. What the attack found + +Severity: **Important** changes a contract or the phase structure. **Medium** changes one block. **Minor** is a one-line fix. Each finding names where this plan resolves it. + +### Important + +- **A1. Forks are orphaned when the Athanor dies.** Supervisors are detached by design, so they outlive the orchestrator. The orchestrator can die in several ways: Igor `TaskStop`s it when a new plan invalidates the run (`plugins/spire/skills/igor/SKILL.md:112`), it crashes, it runs out of context, or the user interrupts it. The forks then keep editing and committing on the branch for up to `--max-minutes` (90). The plan's `lost` rule (plan §3.7) covers a dead *supervisor*, not a dead *orchestrator*. D4 defers Igor's text, so this has to be fixed in the helper. **New invariant N1** (§4): an orphan watchdog. +- **A2. The Bash tool's default timeout is 120 s.** Its maximum is 600 s. `wait --timeout 540` is safe only if the Athanor passes `timeout: 600000` on that Bash call. Neither the design nor the plan says so: design §7 calls 600 s the "ceiling", but it is the maximum, not the default. Without the parameter, every `wait` gets killed at 2 min with no JSON. **§4 C2**, G-text check, P0.1(d). +- **A3. Windows file semantics break the registry design.** `os.replace` raises `PermissionError` while another process has the target open. Here that means `wait`/`list` readers and the supervisors that tick every 60 s. The plan also puts a 30 s heartbeat per job into the registry (plan §3.7), which multiplies writes under one lock. Appends from parallel supervisors to `ledger.jsonl` are not atomic on Windows. Two tickers can each decide to ping the same base. **§4 C3.** +- **A4. P0.6's pass criterion cannot pass as written.** Bases A and B share cwd, tools and system prompt, and P0.2–P0.5 run in the same clone. So the shared system and tool prefix stays warm (`F4d`), and B's fork at 65 min reads that part from cache. It shows `partial`, not a miss. The same fact changes the helper's semantics: a fork of a cold base will usually show `partial`. Separately, P0.6 needs a timed probe that runs for 65 min. It cannot be driven by foreground `sleep` (the harness blocks it), so it depends on P0.1's detach mechanism, yet the plan starts P0.6 first. **Amended P0.6** and the ordering in Phase 0, **§4 C5**. +- **A5. P0.1's escape route is too narrow.** If `uv` (or the Bash tool) puts its child in a job object with `KILL_ON_JOB_CLOSE` and without `BREAKAWAY_OK`, `CREATE_BREAKAWAY_FROM_JOB` fails with access denied. The `--attach` fallback then routes its notification to the session root (`spawning-claude-code.md:29`). There is a standard way out of any job: create the process through WMI/CIM (`Win32_Process.Create`), which starts it outside the caller's job. There is a second problem. If `sys.executable` is a venv launcher (`python.exe` that re-spawns the base interpreter), the supervisor gets an extra layer. **Amended P0.1** ladder, **§4 C6**. +- **A6. Forks can launch forks.** Headless runs load project settings (`5a`). Reliquary allows `Bash(uv:*)` (`.claude/settings.json:32`), and forks have the `Skill` tool, so a fork can load `spire:forks` and run `uv run …/forks.py fork`. The deny rule `Bash(claude:*)` does not cover that, and it does not cover `"$CLAUDE_CODE_EXECPATH" -p` either. **New invariant N3** (§4). +- **A7. The successor Athanor would re-train.** When an Athanor's context is spent, Igor starts a fresh one "from the run log and trunk notes" (`igor/SKILL.md:112`). The new intake says "first phase: `forks init`, `train`" (design §8.2 step 1). It has no rule for an existing state dir, so a successor would init a second time and train a second base. **Block 3.1** adds an adoption rule. +- **A8. The release path is missing a step.** The marketplace is a git source (`.claude/settings.json:2-8`), and `spire` resolves to `./plugins/spire` inside it (`.claude-plugin/marketplace.json`). So 0.3.0 reaches anyone's plugin cache only after the branch is merged and pushed to the remote's default branch. Installing it also switches the Athanor procedure in **every** project at once, before the smoke and the trial. **Post-build steps**, decisions **U1** and **U3**. +- **A9. The build inputs are untracked.** `goals/headless-forks.md`, the design and the plan are `??` in git status, and the session journal is modified. A worktree or branch made now starts without them. **Pre-build step B0.** + +### Medium + +- **A10. `wait` exit codes are ambiguous.** It is unclear what `wait --all` returns when one job failed and others are still running. The Athanor loops "until exit 0 or 4" (design §7) and could stop looping with jobs running. **§4 C4.** +- **A11. Parallel mode would need a markdown parser.** "Refuses `parallel` mode unless the brief names its file set" (design §3.4) means parsing the brief. **Amended:** `--paths` is required for parallel. The helper stores it, the result echoes it, and the Athanor commits exactly those paths (plan §4.4 becomes mechanical). +- **A12. The helper's git calls can collide with a fork's commit.** The delta header, `--shortstat` and the rebase thresholds call git, and those calls can take `index.lock` opportunistically while a fork commits. **§4 C7:** every helper git call uses `git --no-optional-locks`. +- **A13. Consumers the plan's 3.2/3.3 lists miss:** + - `athanor.md:23` "otherwise orient via direct file reads". It must say plans and docs, not code. + - `athanor.md:60` "a fail-first test inside a block, a look one layer up before fixing" are code-level actions by the Athanor. They become things it writes into briefs. + - `athanor.md:7-10` the frontmatter description ("blocks in the foreground") is what Igor reads to route. + - `implementation/SKILL.md:40` "Each block commits and adds a short handoff note to the run log". Forks write `notes/.md` and the Athanor appends them. + - `models.md:49` "a trunk read at intake of about 100k–250k". That read is now the base's: add a dated note, keep the old line. + - **Block 3.1 / 3.2.** +- **A14. D5 ("refuse at init") needs a rule the plan does not give.** `availableModels` can live in user, project and local settings, and their merge is unknown. Reliquary lists the alias `sonnet` (`.claude/settings.json:64`), and nobody has tested whether that admits `claude-sonnet-5-5`. **Amended P0.5**, **§4 C8.** + +### Minor + +- **A15. Cut `$SCRIPTS_UV` from the Athanor's calls.** `"$SCRIPTS_UV" run …` does not match a `Bash(uv:*)` allow rule, so it would prompt or be denied behind the closed door, and it is unset on this host. Calls use plain `uv`. `where` reports `SCRIPTS_UV` when it is set, and SKILL.md mentions it only for hosts without `uv` on PATH. +- **A16. Add a `.python-version` case to P0.0.** The Athanor will build in Python projects that carry one. Add a cwd with `.python-version` pinning 3.9 and check that the script's `requires-python >= 3.10` wins. +- **A17. R4 can be partly tested before release.** Precedent: an `!` injection at agent preload already runs on this host. Igor preloads `spire:adepts` (`agents/igor.md:15`), and the journals record its output, "Roster empty at boot" (2026-09-21, -09-29, -10-05, -10-15). That line is `roster.sh:18`'s own output, so the injection ran. What is untested: an **Agent-tool subagent's** preload, `uv`, and whether `allowed-tools` pre-approves the call. **New P0.8**, a headless proxy in the scratch clone. It is informational, not a stop item. +- **A18. Smoke and test details.** + - Check that `kill` "leaves no `claude` process behind" by the recorded PIDs. Checking by name would find the host, which is `claude.exe` too. + - Generate the stub `claude` wrapper at test runtime in a temp dir. `.gitattributes` has no rule for `.cmd`/`.bat`, so nothing of the kind gets committed. +- **A19. PIDs are reused on Windows.** `lost` and orphan detection compare PID plus process start time. +- **A20. G-text checks only one direction.** Add the reverse (every command in `forks.py`'s dispatch is named in SKILL.md), plus checks for the `600000` rule and the `SPIRE_FORKS_CHILD` guard. +- **A21. Fix-pass commits land after the release commit.** The release commit is the last block of Phase 3, but the Phase 3 review and fix pass come after it. That is fine: 0.3.0 is unpublished until it is pushed, and fixes before the push stay 0.3.0 with no version change. +- **A22. Pre-existing drift, not in scope.** + - The README skills table already lacks `adepts`, `curator`, `init`, `project` and `zathras` (`README.md:13-21`). This build adds only `forks`. + - The agent and skill lists in `.claude/spire/project.md:31-32` are stale (no planner/attacker/reviewer; `forks` is missing after this build). That file is project config. Both are follow-ups. +- **A23. The design cites project config keys that do not exist.** It refers to an assertion pattern "from project config" (design §8.2 step 5) and "unless project config says parallel test runs are safe" (§8.3). Neither key exists in the `## Build` template. The defaults go into the procedure text (pattern `assert`; parallel tests off). Adding the keys is a follow-up for `spire:init`/`spire:project`. + +--- + +## 2. Conventions pass + +| Rule | Governing source | Plan §1 | Finding / action | +|---|---|---|---| +| Gitmoji ` ` | `CLAUDE.md:7`, `specs/commit-conventions.md:3-5` | yes | ok | +| `🤖` for agent files incl. skills | `commit-conventions.md:91` | yes | ok. Precedent is split for a new skill with a script: `1c4ec76` used `✨` (adepts + `roster.sh`), the newer `f1de89e` used `🤖` (orbit scripts). Keep `🤖`; `✨` is acceptable for 1.1's first commit. Either way it is not a finding. | +| **No `Co-Authored-By` or other attribution lines** | user policy (coordinator, 2026-10-16); `.claude/settings.json:70` `includeCoAuthoredBy: false`; precedent `7c1ed39` body has none | **missing** | Added to every block below. It applies to commits made by forks and by fresh agents too. | +| Plugin self-contained | `CLAUDE.md:3` | yes | ok | +| Workpad placement and naming; cascade subfolder | `plugins/spire/skills/workpad/SKILL.md:72`, `:80` | yes | ok | +| README lists components | `plugins/spire/CLAUDE.md:1` | yes | Add the `forks` row only (A22). | +| Two manifests carry the version; marketplace files carry none | `.claude-plugin/plugin.json:3`, `.codex-plugin/plugin.json:3`; checked `.claude-plugin/marketplace.json`, `.agents/plugins/marketplace.json` | yes | ok | +| Codex loads `./skills/` wholesale | `.codex-plugin/plugin.json:10` | partly | `forks` reaches Codex automatically and there is no per-skill opt-out. The "Claude Code only" line in SKILL.md is required, not optional. | +| LF endings | `.gitattributes:1-7` | yes | No rule for `.cmd`/`.bat`, so stubs are generated at test runtime (A18). | +| `**/*.log`, `**/*.tmp` ignored | `.gitignore:9-10` | yes | `.txt` for logs (plan). | +| `__pycache__/` ignored | `.gitignore:13` | no | Covers the test import of `forks.py`. No action. | +| Worktrees under `.claude/worktrees/` (ignored) | `.gitignore:7`; Igor: worktree for code runs, `igor/SKILL.md:179` | no | The build runs in `.claude/worktrees/headless-forks`, branch `headless-forks` (B0). | +| Push branches, never open PRs; push only if instructed | `igor/SKILL.md:177`, `athanor.md:55` | partly | The build does not push. Merge and push are post-build (U3). | +| Goals read-only for the Athanor; stage explicit paths | `athanor.md:53`, `:55` | yes | ok | +| Stamp exact model id | `implementation/SKILL.md:68` | yes | ok | +| Spawning statuses: verified / observed / assumed / unknown, dated | `spawning-claude-code.md:5-12` | yes | New rows use only these four. The current `:61` "unverified" is outside that vocabulary, and the design's replacement row fixes it. | +| Model notes dated, not dropped | `models.md:3` | yes | ok (A13: `:49` gets a dated note). | +| uv: PEP 723, `uv run path/to/script.py`, `SCRIPTS_UV` | `project-uv/SKILL.md:21`, `:31` | yes | A15 cuts `$SCRIPTS_UV` from the calls. | +| Skill `!` injection + `allowed-tools` | `adepts/SKILL.md:4`, `:7-12` | yes | Precedent at preload (A17); P0.8. | +| Separation: builder ≠ reviewer; fresh review per phase | `implementation/SKILL.md:22`, `:53` | yes | ok | + +--- + +## 3. Precedent search on the risks + +**Dropped** (the repo or this host already does the risky thing): + +| Risk | Precedent | +|---|---| +| `${CLAUDE_SKILL_DIR}` (Windows path) inside an `!` injection under Git Bash | `adepts/SKILL.md:12` with `roster.sh`; it ran at Igor's preload in four journals (A17). | +| Nested `claude -p` from a subagent's Bash works at all, including cwd in the session scratchpad | research T1–T7, D1–D3, E1, V1 (all from a subagent in this scratchpad). | +| Running the Desktop-bundled binary via `$CLAUDE_CODE_EXECPATH` from outside its package | research `F5a`, V1. | +| uv-managed Python ≥ 3.10 available offline | `uv python list --only-installed`: 3.10.16, 3.11.11, 3.12.x and 3.13.1 under `%APPDATA%\uv\python` (checked 2026-10-16). | +| `__pycache__` or `.log` files leaking into or hiding from git | `.gitignore:13`, `:9`. | +| Absolute user paths in committed workpad files | the research doc already carries them. | + +**Reduced**: + +| Risk | Precedent | What remains | +|---|---|---| +| R4 injection + `allowed-tools` | main-thread preload works (A17); reliquary has a project allow rule for `uv` (`settings.json:32`), so pre-approval matters only in other projects | Subagent preload and pre-approval: P0.8 proxy, then Phase 4. | +| D3(b) arm C once 0.3.0 is installed | the plugin cache keeps old versions: `~/.claude/plugins/cache/reliquary/spire/` holds 0.1.0, 0.1.1, 0.2.0 and 0.2.2 | Whether `--plugin-dir` on the 0.2.2 cache dir (or a worktree at `a7ce5db`) runs cleanly beside an installed 0.3.0 is **assumed** (U2). | +| R3 parallel tool-using forks | index-lock races between parallel children happened in the 2026-10-15 experiment (design §8.3, M5), so the risk is real | The mitigation (parallel forks never commit) stays; P0.4 tests it. | + +**Kept** (no precedent): R1, R2, R5, R7, the Windows file semantics (A3), contract drift at the Phase 1–2/3 seam, orphans (A1), R11 plan usage, R12 binary drift. See §8. + +--- + +## 4. Contract amendments (on top of plan §4) + +The exact form matters for these, so they are stated as invariants. Everything else in plan §4 stands. + +- **C1. Phase 1 values for Phase 2 fields.** In Phase 1, `base_state` and `warm_until` are computed with the configured TTL (default 3600 s), with no inference and no pings. `rebase_due` and `retrain_due` are `false` with `reasons: []`. Phase 2 replaces the computation, not the fields. +- **C2. The `wait` call.** SKILL.md states: *call `wait` from Bash with `timeout: 600000`; never omit it.* `wait --timeout` keeps its default of 540 and its maximum of 570. +- **C3. State files on Windows.** + - (a) Every registry write is temp-then-`os.replace`, retried on `PermissionError` (at least 20 tries, 50 ms apart). Readers retry the same way. + - (b) Per-job runtime state (supervisor PID plus start time, claude PID plus start time, `heartbeat_at`, `turns`, `last_event_at`) lives in `jobs/.json`. Its only writer is that job's supervisor. The registry holds config, bases and the job index. + - (c) Ledger appends happen under `registry.lock`. + - (d) A ping is decided and marked `ping_in_flight` on the base under the lock. Other tickers skip a base that is in flight. +- **C4. `wait` exit codes.** + - Exit `3` while the wait condition is unmet. `done[]` may already hold finished results, failed ones included. + - Exit `0` or `4` only when the condition is met: `0` if every job in the condition ended `ok`, else `4`. + - Every result is returned by at least one `wait`. `list` always shows all jobs. +- **C5. Cache verdict and cold start.** + - The verdict is set from the **first** assistant event's `cache_read` against the base's context tokens: `hit` ≥ 0.9, `partial` > 0, `miss` = 0. + - `cold_start` is set when the base was `cold` at launch, **or** when the verdict is not `hit` and the fingerprint is `ok`. + - SKILL.md says that a cold base usually shows `partial`, because the shared system prefix stays warm (`F4d`). +- **C6. How the supervisor is spawned.** The interpreter is `sys._base_executable` when it differs from `sys.executable` (a venv launcher), else `sys.executable`, with `-I`. Stdio goes to files under `raw/`, and no inherited handles remain. The creation method is whichever rung of P0.1 passed. It is one constant in `forks.py`, and `where` reports it. +- **C7. Git calls.** Every git call from the helper is `git --no-optional-locks …`. `where` reports the resolved `git`. +- **C8. Models outside `availableModels` (D5).** + - `init` applies the rule P0.5 observed, against the `availableModels` lists it finds in `~/.claude/settings.json`, `/.claude/settings.json` and `/.claude/settings.local.json`. Managed settings are out of scope. + - If the id is refused, `init` exits 2. The message names the id and the files that hold lists, and says that the user must add the id. + - At runtime, a refusal in `train` ends as status `failed`, reason `model_refused`. No settings file is ever written. +- **N1. Orphan watchdog.** + - The helper records `last_orchestrator_call_at` on the run whenever `wait`, `fork`, `continue`, `train`, `rebase` or `list` is called. + - A supervisor whose job is still running kills its `claude` tree when no such call has happened for `orphan_after` seconds (`init` default 1200, more than 2 × 570). The job's status becomes `killed`, with flag `orphaned`, and a ledger line is written. + - *Why:* A1. The procedure keeps the Athanor in a `wait` loop whenever a fork runs, so a silence of 20 min means it is gone. +- **N2. Parallel forks need `--paths`.** `fork --mode parallel` requires `--paths

…`. The paths are stored in the job and echoed as `paths` in the result. Sequential forks may pass `--paths` too. +- **N3. No forks inside forks.** The child env gains `SPIRE_FORKS_CHILD=1` (on the keep list). Every launching command (`train`, `fork`, `rebase`, `continue`, `tick` pings) exits 2 when that variable is set. `where`, `list`, `usage` and `wait` stay allowed. +- **Usage md columns** (2.3 and `run-log.md`, fixed here): `Block | Owner (model id from modelUsage) | Session | Commit | in | out | cache write | cache read | cost Δ`. `Commit` lists the commits between the fork's launch HEAD and its end, made by a sequential fork. Parallel forks show `—`, and the Athanor fills in the commit. + +--- + +## 5. Decisions applied + +The coordinator's defaults: + +- **D1** = 0.3.0, both manifests. +- **D2** = the release commit body, no CHANGELOG file. +- **D3** = trial deferred; it needs the user; Phase 5 is out of this build. +- **D4** = the Igor follow-ups are out of this plan and recorded below. N1 makes Igor's `TaskStop` non-fatal in the meantime. +- **D5** = refuse at init with a clear message, no settings changes (C8). + +The attacker's defaults, which the build takes unless the user overrides them: + +- N1 with a 20 min threshold. +- N2, N3. +- A15: `$SCRIPTS_UV` cut from the calls. +- Worktree `.claude/worktrees/headless-forks` on branch `headless-forks`. +- Live smokes use `claude-opus-5-5` (it is in the clone's `availableModels`). +- The Phase 0 spend trigger in §6. + +--- + +## 6. Build + +Gates (run from the worktree root): + +- **G-unit**: `uv run plugins/spire/skills/forks/tests/test_forks.py`, exit 0. +- **G-where**: `uv run plugins/spire/skills/forks/forks.py where`, exit 0. Stdout is exactly one JSON object; uv's own output goes to stderr. +- **G-text**: the list in Phase 3. + +Standing rules for every block: + +- Commits use gitmoji, explicit paths, and **no `Co-Authored-By` or other attribution lines**. +- No push, no PR. +- `goals/` is read-only. +- Each handoff note stamps its model id. +- Every live test targets a `git clone --local` in the session scratchpad, never the worktree (plan §3.4). +- Owners are fresh Agent-tool agents dispatched by the 0.2.2 Athanor. Unpinned agents get an alias, and each stamps its id (`spawning-claude-code.md:47-52`). + +### Pre-build (Igor, before dispatch): B0 + +1. Commit on `main` by explicit paths: the goal, the design, the plan, this build plan, and the journal. Message: `📝 Plan headless forks (spire:forks): goal, design, plan, build plan`. The other untracked report (`reports/2026-10-16-01-session-harness-app-forms.md`) is Igor's call. +2. Run `git worktree add .claude/worktrees/headless-forks -b headless-forks main`. +3. Dispatch the installed **0.2.2** Athanor (`model` omitted) with this build plan. The worktree path is its checkout, and the run log is `docs/workpad/reports/-headless-forks-run.md` inside it. + +### Phase 0: Preflight (STOP gate) + +**Owner**: the Athanor itself, from its own Bash (`claude-opus-5-5[1m]`). This is one block, because the process tree is what is under test (plan §3.2). Probes are throwaway and live in the scratchpad; nothing in `plugins/` changes. + +**Order** (amended, A4): P0.0, then P0.1, then start P0.6 as a detached timed probe using P0.1's mechanism. Then P0.2–P0.5, P0.7 and P0.8 while it runs, then collect P0.6. + +**Changes to plan §7 Phase 0** (rows not named here stand as written): + +- **P0.0** adds: + - (c) A cwd with `.python-version` = `3.9` (A16). + - Record `sys.executable` vs `sys._base_executable` (C6), and the resolved `git` and `claude`. +- **P0.1** ladder (A5). Each rung is tried only if the one before it failed: + 1. `Popen` with `DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_BREAKAWAY_FROM_JOB` from inside `uv run`. + 2. The same without breakaway. + 3. Creation through WMI/CIM: `powershell -NoProfile -Command "Invoke-CimMethod -ClassName Win32_Process -MethodName Create …"`, capturing the returned PID. + 4. `--attach` under Bash `run_in_background: true`, polled from later foreground calls. + + Each rung must survive four things: (a) the uv process exiting; (b) the Bash call ending; (c) ≥ 15 min, with output growing in later calls; and (d) a Bash call that the tool kills on timeout (call with `timeout: 5000`). A tree-kill must also leave none of the recorded PIDs alive. Record the rung and its flags. **STOP** only if all four rungs fail. +- **P0.5** adds three model cases for C8: + - a listed full id (`claude-opus-5-5`); + - an unlisted full id whose alias is listed (`claude-sonnet-5-5` vs `sonnet`); + - an unlisted id of an unlisted family if one exists, else skip. + + Record the rule. +- **P0.6** (amended, A4). Bases A and B are trained with different training content and the same fingerprint, and run as a detached timed probe. Pass when A's fork after its 50 min ping shows first-event `cache_read` ≥ 0.9 × A's context, **and** B's fork at 65 min shows < 0.5 × B's context. A `partial` on B is the expected result. On fail: pings default off (plan §6). +- **P0.8 (new, informational, A17).** In the scratch clone: + - Remove `Bash(uv:*)` from the clone's `.claude/settings.json` (scratch only). + - Add `.claude/skills/forkprobe/SKILL.md` with `allowed-tools: Bash(uv:*)` and `` !`uv run "${CLAUDE_SKILL_DIR}/probe.py" where` ``. + - Add `.claude/agents/forkprobe.md`, which preloads it. + - Run a headless `claude -p --tools "Task,Bash" --permission-mode default` that dispatches `forkprobe` as an Agent-tool subagent. The subagent quotes the injected text, then runs `uv run …/probe.py ping`. + + Record three things: whether the injection ran in the subagent, whether the later call was pre-approved (no entry in `permission_denials`), and the wall time of the injection. On fail, SKILL.md says that projects need a `Bash(uv run:*)` allow rule (plan §6 default). +- **Spend trigger (new).** Sum the per-call `total_cost_usd` deltas (client estimates). If the sum passes **$25** before P0.2 is complete, stop probing and hand back the partial report as `STOP (budget)` for the user. + +**Phase 0 done when**: `reports/-headless-forks-preflight.md` exists with every row filled. A one-line `GO` or `STOP` heads it, and it lists the defaults Phase 1 must use: scrub additions, the P0.1 rung and flags, the usage rule, the parallel cap, pings, the fingerprint fields kept or dropped, the `availableModels` rule, the injection outcome. Commit it as `📝`. The Athanor hands back. + +**Review and decision**: a fresh `reviewer` checks the report's numbers against the raw files in the scratchpad. Igor makes the GO/STOP call. GO continues without the user; STOP goes to the user. + +### Phase 1: Helper core + +| Block | Owner | Scope | Gate | May stay broken | +|---|---|---|---|---| +| **1.1** Skeleton, state, `where`, `init` | fresh, `opus` | plan §7 1.1 plus: C3(a); `jobs/` dir; C7; C8 in `init`; the `orphan_after` config (N1). `where` reports uv, Python (both executables), the script, `claude`, `git`, the spawn rung, `SCRIPTS_UV` if set, and unknown `CLAUDE_CODE_*` vars. | tests the block owns pass via `uv run …/tests/test_forks.py`; G-where | Launching commands exit 2 `not implemented`. | +| **1.2** Launch and supervise | fresh, `opus` | plan §7 1.2 plus C1, C3(b–d), C4, C5, C6, N1, N2, N3, A19. The stub `claude` is generated at test runtime (A18). It must answer `--version`, replay init/assistant/result, hang, die, emit denials, and refuse a model. Tests also cover: replace while a reader holds the file open (C3a); exit codes in the mixed case (C4); the orphan kill under an injected clock (N1); refusal under `SPIRE_FORKS_CHILD` (N3); `parallel` without `--paths` refused (N2). | G-unit | Warmth inference, pings, `rebase`, `usage --format md` (Phase 2). | +| **1.3** Phase tests, smoke, fallout | the Athanor runs the smoke from its own Bash; any fallout goes to a fresh `opus` agent | G-unit, then the live smoke of plan §7 ("Phase 1 tests and smoke") in the scratch clone with `claude-opus-5-5`. Every `wait` is called with Bash `timeout: 600000`. The `kill` check uses recorded PIDs (A18). | G-unit + the smoke criteria of plan §7 | none | + +Commits: `🤖` (1.1 may use `✨`). **Fallbacks**: + +- If G-unit is still red after 1.3's fallout agent plus one more fresh fix agent, hand back red and do not start Phase 2. +- If a smoke fork shows a verdict other than `hit` while P0.2 passed, it is a fingerprint bug. One fix, then re-smoke. If it still misses, hand back red with the raw stream paths. +- If the Athanor judges its context has passed about 600k (`models.md:19`, a guesstimate), hand back for a successor started from the run log. + +### Phase 2: Bookkeeping + +| Block | Owner | Scope | Gate | May stay broken | +|---|---|---|---|---| +| **2.1** Warmth and keep-warm | fresh, `opus` | plan §7 2.1, ping dedupe per C3(d); the cold-start rule per C5 | G-unit | `rebase`, `usage md` | +| **2.2** Rebase and re-train flags | fresh, `opus` | plan §7 2.2. A missing binary (the Desktop path is versioned, `F5a`) gives `retrain_due` with reason `binary_missing`, and launches refuse. | G-unit | `usage md` | +| **2.3** Usage rendering | fresh, `sonnet` | plan §7 2.3 with the fixed column contract (§4) | G-unit | none | +| **2.4** Phase tests, smoke, fallout | the Athanor, plus a fresh `opus` for fallout | the plan §7 "Phase 2 tests and smoke", same Bash rules as 1.3 | G-unit + the smoke criteria | none | + +Commits `🤖`. The fallbacks are the same as Phase 1. + +### Phase 3: Skill text, procedure text, release + +| Block | Owner | Scope | Gate | May stay broken | +|---|---|---|---|---| +| **3.1** Procedure text | fresh, `opus` | `forks/SKILL.md` per plan §7 3.1 (C2 rule, C4 and C5 semantics, N1–N3, the A15 cut, the Claude-Code-only line); `agents/athanor.md` per plan §7 3.2 **plus** A7 (a `Forks:` line in the run log → `list` and adopt, never re-`init`), A13 (`:7-10`, `:23`, `:60`), A23 defaults; `implementation/SKILL.md` per plan §7 3.3 plus A13 (`:40`); `references/run-log.md` (with the fixed columns) | G-text items 1–4, 6, 7 | Versions still at 0.2.2. | +| **3.2** Reference rows | fresh, `sonnet` | `spawning-claude-code.md` per design §9 and plan §7 3.3, including the P0.0 and P0.1 rows marked `observed 2026-10-16 (preflight report)`, and the P0.8 row; `models.md` dated notes, including the A13 note on `:49` | every new row has one of the four statuses and a date | Versions still at 0.2.2. | +| **3.3** README, version, release | fresh, `sonnet` | the README `forks` row; both manifests at `0.3.0`; G-text in full; release commit `🤖 Spire 0.3.0: trained implementer forked per block (spire:forks)` with the changelog as its body (precedent `7c1ed39`): new skill, Athanor procedure change, Agent-tool fork path dropped, references updated, preflight report path, the known follow-ups. No attribution lines. | G-text in full, G-unit, G-where | none | + +**G-text**: + +1. `grep -n "fork of yourself\|That reading is your trunk\|read the fix's diff\|the code the phase touches" plugins/spire/agents/athanor.md` returns nothing. +2. `spire:forks` appears in athanor.md's frontmatter `skills:`. +3. `grep -n "a fork of the builder" plugins/spire/skills/implementation/SKILL.md` returns nothing. +4. `forks/SKILL.md` contains `600000`, `SPIRE_FORKS_CHILD` and `--paths`. +5. The `version` values in both manifests are equal and read `0.3.0`. +6. Every command name in SKILL.md is in `forks.py`'s dispatch, **and** every name in the dispatch (except `_supervise`) is in SKILL.md. +7. G-unit and G-where still pass. + +**Phase 3 review**: a fresh reviewer checks the procedure text against the helper's real JSON and exit codes (run G-where and a stubbed `wait`). This is the Phase 1–2/3 seam. Fix-pass commits that land after the release commit stay 0.3.0 (A21). + +--- + +## 7. Post-build (outside this build; the user is needed) + +1. **Merge and push** `headless-forks` into the remote's default branch (U3). Until then nobody can install 0.3.0 (A8). +2. **Phase 4**, plan §7 unchanged, with these entry options (U1): (a) smoke 0.3.0 first with `--plugin-dir /plugins/spire` in the scratch clone, with `spire@reliquary` disabled in the clone's `settings.local.json` (**assumed** workable), then update the marketplace; or (b) update the marketplace directly. Option (b) switches every project's Athanor before any smoke. +3. **Phase 5**, the trial, plan §7 unchanged. Entry needs U2. +4. Igor ticks goal boxes; the user decides on the "Proven" box. + +--- + +## 8. Risks for the build + +These shape the build. They are not the phase reviewer's scope. + +| Risk | Where it bites | Handling | +|---|---|---| +| R1 detach under the uv and Bash job objects | P0.1, then every launch | Four-rung ladder; STOP only if all fail. | +| R2 large-base cache | P0.2 | STOP rule as planned. | +| R5 usage-field rule | P0.3 → 1.2 | 1.2 implements exactly the recorded rule. | +| R7 TTL refresh | P0.6 | Amended criterion; pings off on fail. | +| Windows file semantics | 1.1, 1.2, 2.1 | C3 plus its unit tests. | +| Contract drift between the Phase 1–2 JSON and the Phase 3 text | 3.1 | §4 fixes the seam; G-text 6; the Phase 3 review. | +| Orphaned forks | after the build, wherever 0.3.0 runs | N1; D4 follow-up. | +| R11 plan usage | Phase 0, smokes | Spend trigger; smokes on small bases. | +| R12 binary drift or disappearance (Desktop update) | any run | Version/size/mtime check; `binary_missing` → `retrain_due`. | +| Shipping the procedure change before the trial | post-build install | U1. | +| R4 injection and pre-approval in subagents | 0.3.0 in other projects | P0.8 proxy; fallback text in SKILL.md. | + +--- + +## 9. Open decisions for the user + +None of these blocks the build (Phases 0–3). U1–U3 must be settled before Phase 4. + +- **U1. How 0.3.0 reaches installs.** Smoke via `--plugin-dir` before the marketplace update (default), or update directly. Updating changes the Athanor in every project before the smoke and the trial. +- **U2. The trial (D3).** (a) Which real multi-block plan. (b) Arm C after install: `--plugin-dir` on the cached 0.2.2 (`~/.claude/plugins/cache/reliquary/spire/0.2.2`, still present) or a worktree at `a7ce5db`; or run arm C before installing 0.3.0. +- **U3. Merge and push.** Who merges `headless-forks` and pushes, and when. +- **U4. `availableModels`.** If P0.5 shows that unlisted full ids are refused, do you want full Sonnet or Haiku ids added to `.claude/settings.json`? Without them, reliquary's forks are Opus-only. This build does not touch settings. +- **U5. Defaults you may override**: the orphan threshold of 20 min (N1), the Phase 0 spend trigger of $25, the worktree and branch name. + +--- + +## 10. Follow-ups (recorded, not built) + +- D4: `igor/SKILL.md:66` fork wording; `:112` `TaskStop` of an Athanor should be followed by `forks list/kill` on its run (N1 covers it after 20 min). +- `## Build` keys for the assertion pattern, safe parallel tests, and the default fork owner model (full id), in `spire:init` / `spire:project` (A23). +- README skills table drift; `.claude/spire/project.md:31-32` lists (A22). +- Root `AGENTS.md` still points at `g4b_ai` (plan §5). diff --git a/docs/workpad/reports/2026-10-16-01-session-harness-app-forms.md b/docs/workpad/reports/2026-10-16-01-session-harness-app-forms.md new file mode 100644 index 0000000..3870c47 --- /dev/null +++ b/docs/workpad/reports/2026-10-16-01-session-harness-app-forms.md @@ -0,0 +1,435 @@ +# Session harness app: three forms, candidate tasks, risks + +**Date**: 2026-10-16 +**Author**: researcher, model id `claude-opus-5-5` +**Status**: findings and options for the user. No verdict. +**Asked by**: Igor, session [2026-10-15-21-grotto-work](../journals/2026-10-15-21-grotto-work.md) (entry "User long-term idea: a dedicated Rust app") +**Inputs**: [goals/headless-forks.md](../goals/headless-forks.md), [designs/2026-10-16-01-headless-forks.md](../designs/2026-10-16-01-headless-forks.md) (claim ids such as `F4a` and risk ids such as `R1` refer to it and to the research), [research/2026-10-16-00-claude-headless-fork-sessions.md](../research/2026-10-16-00-claude-headless-fork-sessions.md), [research/2026-10-15-14-claude-code-tasks-and-pcvelz.md](../research/2026-10-15-14-claude-code-tasks-and-pcvelz.md), [reports/dirigent/](./dirigent/). I also took a read-only look at the Dirigent repo (`G:\dev\projects\dirigent`, HEAD `b95d026f`, 2026-10-15) to see which crates already exist. Web: one search, on the Rust MCP SDK. + +Working name in this report: **stable**. It comes from the user's "stable for the constables", meaning something that keeps the harnesses. A headless session is a *constable*. A trained base is a *mount* kept in the stable. These are placeholders, not proposals. + +--- + +## 0. Summary + +- The near-term Spire helper (`spire:forks`) already fixes the app's core: one owner of the `claude -p` command line, a launch fingerprint, an env scrub, a registry and ledger, numeric keep-warm and rebase rules, and per-call usage. A Rust app is that core with a longer lifetime and more callers. The **JSON contract of the helper is the natural seam**: keep it and all three forms can sit behind it. +- Dirigent already has most of the parts in Rust: + - `dirigent_anth` parses Claude Code JSONL transcripts, dedups streamed messages, links subagents and reads plan-usage gauges. + - `dirigent_process` manages process lifetimes, using Windows Job Objects. + - `dirigent_taskrunner` and `dirigent_logs` run child processes and capture their output. + - `dirigent_fermata` is a precedent for "library + CLI + harness adapter" in three layers. + + So the gap between "standalone CLI" and "Dirigent feature" is smaller than it looks, provided the core crate does not depend on Dirigent. +- The three forms differ mainly in **who owns time**: + - The CLI is per call. Bookkeeping ticks only while someone is calling it, which is how the Spire helper works today. + - Dirigent is a long-running server with a watchdog, so it can keep bases warm, enforce TTLs and push signals. + - MCP is a face only. It inherits the time model of whatever sits behind it. +- **The token MCP** (§6), the user's main target: agents and orchestrators read their own and their children's token counts. Apart from headless `result` events, Claude Code exposes almost none of this live, so the data must come from transcripts (`~/.claude/projects/…`, subagents under `subagents/agent-.jsonl`). Identity, meaning "which session am I", is the hard part, not counting. It is read-only and can ship first. +- **Lightweight agent access** (§7): launch / resume / fork / message / status / wait / stop / list over stored launch profiles, as CLI and MCP, and as a Dirigent connector later. These are turn-based headless sessions, not live TUI sessions. +- I found eight workpad tasks that fit, all marked **candidate, confirm** (§8). Bundling ideas, including my own proposals, are in §12. Two fit strongly: orchestrator token introspection (lean-athanor "Later") and the experiment's transcript-based token accounting. Plan-usage and spend-limit awareness also fits strongly. It comes from the Dirigent reports. +- The main risks: + - Claude Code flag and event churn: the app pins against a moving CLI. + - OAuth: the 1 h TTL requires OAuth, and OAuth requires not using `--bare`, which means inheriting discovery. + - Cache-TTL rules: docs-only, account-dependent, and they drop silently to 5 min over the plan limit. + - Detached-process survival on Windows: an untested design risk (R1) that the forms solve differently. + +--- + +## 1. What the app is responsible for (all forms) + +These come from the goal and the design. They are the same in every form. + +| Responsibility | Source | Notes | +|---|---|---| +| Train a base session | goal §Mechanism; design §3.4 `train` | `claude -p --session-id ` with the full fingerprint | +| Fork per task with a brief | design `fork` | `--resume --fork-session`; flags re-passed identically (`4b`, `4c`) | +| Rebase (absorb a delta into a new base) | design `rebase` | lineage: `parent`, `superseded` | +| Continue a fork (one more turn) | design `continue` | `--resume` without `--fork-session` | +| Keep-warm / expire / re-train bookkeeping | goal §Bookkeeping; design §5 | TTL 1 h or 5 min (`F4a`); a hit resets it (`F4c`); pings below the break-even count | +| Launch fingerprint, checked for drift | design §3.3 | binary path + version + size + mtime, full model id, effort, mode, tools, MCP, agent hash, preamble hash, cwd, pinned env | +| Env scrub | design §4; `F3a`, `F3b` | 12 Desktop vars plus model, TTL and auth overrides | +| Usage and token accounting | design §6; `3d`, `F4e` | per-call deltas from cumulative `modelUsage`/`total_cost_usd`; real model from `modelUsage`, not `init.model` (`F2a`) | +| Token-count access for agents | goal "Later"; journal 2026-10-16 | "CC doesn't let an agent read its own tokens". The app can answer for sessions it launched, and for others by transcript | +| Supervision | design §3.5 | stall and timeout watchdogs, kill-tree, `permission_denials` → `needs_attention` | + +Two responsibilities appear only in a longer-lived app, so the Spire helper does not have them: + +- **Cross-run bases.** The design keeps state per run because nothing outlives the TTL. A daemon could keep a project base warm across runs, but only while pinging, so this is a cost question, not a mechanism question. +- **Plan-usage awareness.** It predicts the drop to 5 min (`F4a`, R11) and the spend-limit kills (Dirigent retrospective: "a kill cost 3 h"). `dirigent_anth` already reads plan gauges (`anth_usage.rs`: `UsageGauge { percent_used, resets }`, read via a PTY). + +--- + +## 2. Form 1: standalone Rust CLI + +### Responsibilities +Everything in §1, run per invocation, with no daemon required. Its job is to be what `forks.py` (the uv helper) is now, as a single binary that needs no runtime (no Node, no uv, no Python). + +### Command surface (sketch) +Keep the helper's verbs and exit codes (design §3.4: `0` ok, `2` refused, `3` wait timed out, `4` job failed). Add verbs for the cross-run, accounting and probe cases. + +``` +stable where # binary, versions, scrub check (same JSON as helper) +stable init --state

[launch fields] # --run kept as an alias that derives +stable train --state --base B1 --brief f.md +stable fork --state --base B1 --block 1.3 --brief f.md [--mode parallel] +stable rebase --state --base B1 --as B2 --brief f.md +stable continue --state --job j0007 --brief f.md +stable wait --state --all|--any| --timeout 540 +stable usage --state [--phase n] [--block id] --format json|md +stable list | expire | kill | tick # as helper + +# new in the app +stable tokens --session |--transcript [--dedup-by message-id] [--price ] + # four columns + priced; works on any CC transcript, not only launched ones +stable context --session # current context size of a session ("own tokens") +stable plan usage # plan gauges → predicted TTL bucket +stable probe # run the same prompt under N launch variants, record init + usage +stable daemon [--state …] # optional: run the tick loop + pings; same state files +``` + +`stable tokens` and `stable context` cover the "token-count access for agents" wish. An agent calls them through Bash with its own session id. Its session id is in `CLAUDE_CODE_SESSION_ID` (`F3a` lists that var as inherited, but whether it is the *agent's* id or the host's is unverified). `stable probe` is the hook for the harness-behaviour candidate (§8, C4). The token MCP (§6) and agent access (§7) extend this surface. + +### State and storage +- Keep the helper's layout per state dir (design §3.2): `registry.json`, `ledger.jsonl`, `briefs/`, `notes/`, `results/`, `logs/`, `raw/` (gitignored). The Spire convention puts it beside the run log. The CLI should accept any directory, so non-Spire callers are not bound to the workpad. +- Optional user-level index (`%LOCALAPPDATA%\stable\index.jsonl` or the XDG equivalent). It lists the known state dirs so `stable list --all` and the daemon can find bases across runs. It is an index only; the per-run files stay the source. +- Locking: the same `open(wx)` lockfile plus write-temp-then-rename that the helper uses. In Rust: `fs2`/`fd-lock` or a plain `create_new` lockfile. The helper's files are the contract, so a Rust CLI and the Python helper could even read each other's state dirs during a migration. That only works if the file schemas are frozen, which is a decision, not a given. + +### Migration from the Spire helper +- **Option A: keep the JSON contract byte-stable.** `SKILL.md` switches from `uv run forks.py …` to `stable …` when `stable where` succeeds, and falls back to the helper otherwise. The Athanor procedure does not change. This costs a written schema: stdout objects, exit codes and the ledger line (design §6) as a versioned spec with a `"schema": 1` field. Today the contract lives in the design and the code. +- **Option B: contract v2.** Rename `--run` to `--state`, add fields (plan bucket, transcript-derived context size). The skill text changes once. Worth it only if v1 shows real defects in the trial. +- **Option C: helper as shim.** `forks.py` becomes a ~30-line wrapper that execs `stable` when present. A single skill text, two runtimes. + +In every option the trial results (design §10) should be in before the contract freezes, because design §5 thresholds and R5 (usage semantics) may change fields. + +### Packaging and distribution +- `cargo install`, GitHub/Gitea release binaries (Windows x64, Linux x64/arm64, macOS arm64), optionally `cargo-dist` for installers. A Reliquary plugin could ship a skill that only *documents* `stable` and checks for it. Shipping binaries inside a plugin is possible but heavy, and the marketplace is text-first. +- Single static binary is the main gain over the helper: design R14 ("Node may be missing") and the uv amendment's "python not on PATH" both go away. + +### Windows concerns +- **Detached survival (R1).** Claude Code's Bash tool may run its children inside a Job Object with kill-on-close. That is how `dirigent_process/src/windows.rs` treats its own children (`JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`), and it is the obvious mechanism behind R1. A Rust supervisor can try `CREATE_BREAKAWAY_FROM_JOB | DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP`. Breakaway fails if the job lacks `JOB_OBJECT_LIMIT_BREAKAWAY_OK`. If it fails, the remaining options are the helper's `--attach` fallback or a daemon that was started outside the Bash tool (form 1 `daemon`, or form 2). +- **Kill-tree.** Use Job Objects of its own: assign each `claude.exe` to a job, kill the job. This is cleaner than `taskkill /T /F`. +- **Paths.** The transcript dir slug is derived from the Windows path (`C--Users-…`, `7`, `6d`). Normalise cwd with `dunce` (Dirigent already uses it in `dirigent_tools`). +- **Two binaries** (`F5a`): the PATH install and the Desktop's `$CLAUDE_CODE_EXECPATH` under `AppData\Local\Packages\Claude_…`. The Desktop path changes on every Desktop update (it contains the version and a hash), so a long-lived registry must treat it as drift, never as a stable key. +- Git Bash quoting no longer matters, because the scrub is an env map passed to `CreateProcessW`. +- Console windows: use `CREATE_NO_WINDOW` for children. + +### What it needs from Claude Code +- `-p`, `--resume`, `--fork-session`, `--session-id`, `--output-format stream-json --verbose`, `--model` (full id), `--effort`, `--permission-mode`, `--tools`, `--allowedTools`/`--disallowedTools`, `--strict-mcp-config --mcp-config`, `--append-system-prompt-file`, `--agent`/`--agents`, `--max-turns`, `--max-budget-usd` (`5c`). +- Events: the `system/init` fields (`session_id`, `model`, `tools`, `permissionMode`, `plugins`, `agents`; `apiKeySource` assumed), assistant events (`message.model`, per-message `usage`), and the `result` event (`usage`, `modelUsage`, `total_cost_usd`, `num_turns`, `is_error`, `permission_denials`, `subagent_stats`) (`1a`). +- Env: `CLAUDE_CODE_PROMPT_CACHE_TTL` (≥ 2.1.242, `F4b`), `DISABLE_AUTOUPDATER` (assumed honoured), and the scrub list. +- Transcripts: `~/.claude/projects//.jsonl`, 30-day retention (`6d`), for context size and for accounting of sessions it did not launch. +- `/usage` output for plan gauges. It is only reachable through a PTY today, the way `dirigent_anth` does it, which makes it fragile. + +--- + +## 3. Form 2: Dirigent built-in feature + +### Responsibilities +Everything in §1, plus what a long-running server makes possible: +- **Owned time.** The watchdog-supervised server runs the keep-warm tick continuously. TTL expiry and stall detection happen even when no agent is in its turn. Pings stay a policy choice: the design pings only while the Athanor is active, and a server could widen that. +- **Signals into running work** (the lean-athanor "Later" wording). Dirigent can push "fork j0007 stalled", "plan at 92 %, TTL bucket will drop" or "base B1 cold in 6 min" to the orchestrator, the UI or Matrix. Polling (`wait`) stays the fallback. +- **Archival for free.** Fork transcripts go through `dirigent_archivist` like any Claude session, and `dirigent_anth` already does dedup, lineage and subagent links. Per-fork usage then lands in the session-token-stats pipeline (Dirigent `goals/session-token-stats.md` wants `tokens_cache_read`/`tokens_cache_creation`/`cost` per session already). +- **UI.** A "stable" view showing bases, lineage, warm/expiring/cold, forks running, priced ledger. This matches the Dirigent brainstorm's "Integrations → claude" facade (`brainstorm/rework_claude_concept.md`) and "token metering & caps" (`brainstorm/dirigent_as_acp.md:22, :191`). +- **Policy.** `dirigent_fermata` is a harness-agnostic policy gate with a Claude hook adapter. Forks launched by the stable could get fermata hooks by default, which narrows R16 (forks with full Bash under `dontAsk`). + +### API surface (sketch) +- **Rust**: a `StableService` in the server, wrapping the core crate (§5), registered like the `TaskRunner`. +- **REST** under the existing versioned API (`dirigent_rest`, `/api/v1`): + - `POST /stable/bases` (train) + - `POST /stable/bases/{b}/forks` + - `POST /stable/bases/{b}/rebase` + - `POST /stable/jobs/{j}/continue` + - `GET /stable/jobs/{j}`, plus SSE `GET /stable/events` for signals + - `GET /stable/usage?run=…&phase=…` + - `GET /sessions/{id}/tokens` for any archived session + - `DELETE /stable/bases/{b}` (expire) + - `DELETE /stable/jobs/{j}` (kill) +- **CLI**: the same `stable` verbs as form 1, as a thin client that talks to the server when it is up. With the helper's JSON on stdout, a Spire skill cannot tell the difference. +- **ACP**: Dirigent is an ACP orchestrator. A headless fork could also be offered as an ACP session to Dirigent's clients. That is a larger step, and nothing in the inputs asks for it. + +### State and storage +- Dirigent's data dir (`DIRIGENT_DATA_DIR`) holds the registry and ledger, or a small table next to the archivist. Fork transcripts are archived by the archivist as they are now. +- Spire still wants the run-log-adjacent copy (design §3.2, "the reviewer and a successor Athanor find it"). Options: Dirigent writes or mirrors the per-run state dir as an export, or the run log carries only `Forks: dirigent://stable/run/` and the reviewer reads usage through the CLI. The second makes workpad artefacts depend on a running server, which matters for Codex and for other machines. + +### Migration from the Spire helper +- Same choice as form 1 (stable v1 contract, v2, or shim). The extra step: the skill's `where` detects a running Dirigent (named pipe `\\.\pipe\dirigent` on Windows, socket on macOS, per `dirigent_watchdog`) and routes there, otherwise to a local `stable` binary, otherwise to the uv helper. +- The design chose "its own Spire skill… easier to lift into Dirigent later" (goal, Decided). The lift then means: SKILL.md stays, and only the backend changes. + +### Packaging and distribution +Ships inside Dirigent's existing installers and watchdog. No separate distribution, but users without Dirigent get nothing unless form 1 also exists. The user said "absorbable as a default feature", which implies form 1 exists and Dirigent embeds the same crate. + +### Windows concerns +- R1 goes away when the server is the parent: the server is started by the watchdog, outside any Claude Code job object. This is the cleanest answer to R1 among the three forms. +- Dirigent already lists 13 Windows "path/lock" failures in `dirigent_archivist` tests (handoff-lean-finish §rules). Transcript reading while Claude Code writes the file needs share-mode-tolerant opens (`FILE_SHARE_READ|WRITE`). Tail-reading the live JSONL is a known Windows trap. +- The server runs as the user, so OAuth credentials and `~/.claude` are the same as in the interactive sessions. A service account would break auth. + +### What it needs from Claude Code +Same as form 1, plus: +- live transcript tailing (for signals mid-fork, not only at `result`); +- a stable transcript schema (the archivist depends on it already); +- possibly hook events: Claude Code hooks posting to Dirigent would give "turn ended / tool denied" signals without stream parsing. Fermata already has a Claude hook adapter. + +--- + +## 4. Form 3: MCP server (a face over form 1 or 2) + +### Responsibilities +Only a face. It exposes the §1 verbs as MCP tools so an agent can call them without Bash and without a `Bash(node:*)`/`Bash(uv:*)` permission rule (design R4). It owns nothing that forms 1 or 2 don't. + +### Tool surface (sketch) +`stable_train`, `stable_fork`, `stable_rebase`, `stable_continue`, `stable_wait` (bounded by the MCP call timeout), `stable_usage`, `stable_list`, `stable_expire`, `stable_kill`, and the read-only tools `session_tokens` and `session_context`. Inputs and outputs are the helper's JSON objects, so the contract carries over directly. Resources could expose `stable://run//ledger` and the registry. + +### State and storage +None of its own. Over form 1 it is a stdio MCP process that calls the core crate in-process and writes the same state dirs. Over form 2 it is an HTTP/SSE MCP endpoint served by Dirigent, or a stdio shim that proxies to Dirigent's REST, like `dirigate bridge` does for ACP. + +### Migration from the Spire helper +SKILL.md would name MCP tools instead of Bash commands, so the procedure text changes, though the JSON does not. The design rejected MCP "now" because a script is quicker to prove and "lifts into Dirigent or an MCP wrapper unchanged in contract" (design §Rejected). + +### Packaging and distribution +- A Reliquary plugin can declare an MCP server in its manifest. Codex supports `mcpServers` in its plugin format too (`reports/2026-10-02-16-codex-plugin-icons-and-metadata.md:55`). This is the one form that reaches Codex and other agents uniformly (`research/2026-05-11-skills-ecosystem.md:109`: "MCP server: universal"). +- Rust SDK: `rmcp`, the official Rust implementation (tokio; stdio, HTTP and other transports; tool macros). Docs.rs lists versions up to 3.1.0 against different spec revisions, so pin the version ([docs.rs rmcp 3.1.0](https://docs.rs/crate/rmcp/3.1.0/source/README.md), [Rust SDK overview](https://docsearch.algolia.com/mcp/docs/repo/modelcontextprotocol/rust-sdk)). I found no `rmcp` in Dirigent's `Cargo.lock`, so it would be a new dependency there. + +### Windows concerns +- A stdio MCP server is a child of the Claude Code session, so it lives in that session's process tree and possibly its job object. Forks it spawns face R1 again unless they break away or delegate to a daemon or Dirigent. +- MCP tool-call timeouts limit `stable_wait`, so the poll loop stays. Long blocks (11–36 min in Dirigent) still need repeated waits. + +### What it needs from Claude Code +Same as the backend. One Claude Code-specific interaction: the forks run with `--strict-mcp-config` and an empty MCP config (design §3.3), so the stable's own MCP server is **not** visible inside forks. That is intended: forks are leaves. It also means "token-count access for agents" through MCP covers only the agent that has the server configured (the orchestrator), not the forks. Forks would need the CLI or a deliberate MCP config that contains only the stable's read-only tools. Adding it changes the fingerprint, so it has to be in the base from training. + +A question to verify: MCP tools are part of the tool list, so connecting or disconnecting the stable's MCP server in the *orchestrator's* session changes that session's cache prefix. That is harmless for forks but costs the orchestrator a cache miss once. + +--- + +## 5. What is shared: a core crate + +A layering that would let all three forms reuse one implementation. It mirrors `dirigent_fermata`'s "core / harness / bin" rule: nothing inner imports anything outer. + +| Layer | Contents | Existing Dirigent parts that could be reused or extracted | +|---|---|---| +| `stable_core` (sync, no tokio, no Dirigent deps) | Fingerprint type + canonical hash; env-scrub tables; TTL/warm/rebase/re-train rules (design §5) as pure functions over registry + ledger; ledger and registry schemas (serde) with `schema` version; usage-delta arithmetic (`3d`, `F4e`); price table type | none directly; the rules are new | +| `stable_claude` (Claude Code adapter) | Command-line builder per verb; `stream-json` event types; transcript reader for context size and accounting | `dirigent_anth` (transcript parse, streaming dedup, lineage, subagent links, `anth_usage` plan gauges). Depending on it is possible if `dirigent_anth` stays free of Dirigent-only deps (its `dirigent_config` dep is already behind an optional feature) | +| `stable_proc` | Spawn with explicit env, Job Object kill-tree, breakaway attempt, stall/timeout watchdog, lockfiles | `dirigent_process` (Job Objects on Windows), `dirigent_taskrunner` (spawn/monitor), `dirigent_logs` (capture) | +| faces | `bin/stable` (clap; the only place with stdout JSON + exit codes); `stable_mcp` (rmcp); Dirigent `StableService` + REST routes + UI | `dirigate` for the bridge/shim pattern; `dirigent_rest` | + +A harness trait in the adapter layer, the way fermata has `HarnessAdapter`, would keep a door open for Codex (`dirigent_codex` exists) without designing for it now. Whether Codex has a fork-session equivalent was not checked. + +Where the crate lives: three options. +1. Inside the Dirigent workspace, published as standalone crates the way fermata ships a `fermata` binary. +2. Its own repo, which Dirigent depends on. +3. Inside Reliquary. That is unlikely, since Reliquary is a text-first plugin marketplace. + +Option 1 gets the reuse cheaply. Option 2 keeps form 1 independent of Dirigent's release cadence. + +--- + +## 6. The token MCP (the user's main target) + +The user's main target, relayed by the coordinator after the first draft: agents and orchestrators read their own token counts and their children's. This section treats that as its own product. It is read-only and needs no fork machinery, so it can ship before or without §2–§4. + +### What Claude Code exposes vs. what must be read from transcripts + +| Data | Exposed live by Claude Code | Where it really lives | Status | +|---|---|---|---| +| Per-call tokens, real model, cost for a **headless** run | yes: `result` event (`usage`, `modelUsage`, `total_cost_usd`), cumulative across resume (`1a`, `3d`) | stdout of the launcher | verified (research) | +| Per-message usage for **any** session | no API. The agent cannot ask the harness for it, which is the user's point (journal, 2026-10-16) | transcript lines `~/.claude/projects//.jsonl`; assistant messages carry `message.usage` and `message.model` | message-level `usage` presence assumed (design §5); `dirigent_anth` parses these files today | +| Agent-tool **subagent** usage | only in the completion result the parent sees (design §6: "the Athanor copies what it sees") | `/subagents/agent-.jsonl` plus optional `agent-.meta.json` (`dirigent_anth/src/subagent.rs:19–20`), linked structurally to the spawning call (`resolve_subagent_links`) | verified in Dirigent's parser; layout version-dependent | +| Headless children (`claude -p`) | invisible to Agent tracking (`6c`) | their own transcripts under the *child's* cwd slug; lineage only if the launcher records it (the stable's registry does) | verified (`6d`) | +| Current context size | no tool for the agent itself. From memory, unverified: the status-line command receives a JSON payload with context-window fields in recent CLI versions, and hooks receive `session_id` and `transcript_path` on stdin | last assistant message's `usage` (input + cache read + cache creation) in the transcript | to verify | +| Plan usage / limits | `/usage` in the TUI only | `dirigent_anth::anth_usage` reads it through a PTY (`UsageGauge { percent_used, resets }`) | fragile, works today | +| Cache TTL bucket actually used | per message: `usage.cache_creation.ephemeral_1h_input_tokens` / `_5m_` | transcript or `result` | verified in `result` (`3c`) | +| Prices | not exposed except as client-side `total_cost_usd` | a price table kept as data | ratios assumed (design §5) | + +Conclusion: apart from headless `result` events, **everything comes from transcripts**. The token MCP is in substance a transcript reader with lineage, dedup and pricing. That is `dirigent_anth` plus a small amount of new code. + +### How an agent finds itself + +The hard part is identity, not counting. Options, roughly in order of reliability: + +1. **The launcher tells it.** The stable puts `STABLE_SESSION=` (and the parent's id) into the env, or the brief names the id. This only covers sessions the stable launched. +2. **A hook tells it.** A `SessionStart` hook (the input includes `session_id` and `transcript_path`, from memory) writes a small "who am I" file or injects the id as context. This works for interactive root sessions too. Cost: one hook in user or project settings, and its output changes the prompt prefix once. +3. **The env.** `CLAUDE_CODE_SESSION_ID` is inherited (`F3a`), but it may be the *host's* id, not the subagent's (§11). Agent-tool subagents share the parent's session id and are told apart by agent id, so a subagent asking "me" needs `(session, agent_id)`. +4. **Heuristic.** Newest transcript in the cwd slug touched in the last N seconds. Wrong under parallel sessions in one cwd, which is exactly the Spire case. Use only as a labelled fallback. + +The MCP call should therefore accept an explicit `session` (+ optional `agent`). "Self" works only where 1 or 2 is in place, and the response says which method resolved it (`resolved_by: env|hook|launcher|heuristic`). + +### Surface (MCP tools; the CLI mirrors them as `stable tokens …`) + +``` +tokens_self {agent?} -> {session, agent?, resolved_by, model(s), in, out, cache_write_1h, cache_write_5m, cache_read, context_now, cost_est, as_of} +tokens_session {session, agent?, since?} -> same shape +tokens_children {session, agent?, depth?: 1|all} -> [{child: agent_id | headless session, kind: subagent|fork|headless, model, tokens…, status?}], totals +tokens_tree {session} -> nested tree with rolled-up totals (base history counted once, dedup by message id) +context_now {session, agent?} -> {tokens, model, window, pct, source: transcript|result} +plan_usage {} -> gauges + predicted TTL bucket (1h|5m) +price {tokens…, model, table?} -> priced row (table id stamped) +``` + +Rules for the shape: +- always four columns plus the 1 h/5 min split; +- the real model from messages, never from `init`; +- forks counted once (a fork transcript contains the copied base history, design §10 trial note); +- a timestamp, because transcripts lag by up to a message. + +### Data source and caching +- Read-only over `~/.claude/projects/`. Incremental: remember a byte offset per file and parse only appended lines. Windows needs share-tolerant opens (§3). +- Lineage from four places: (a) Claude Code's own subagent links, (b) the stable registry for headless forks, (c) the `parentUuid` tree, (d) optionally the env/hook breadcrumbs above. +- Cache the per-file aggregates in a small index under the app's data dir. In Dirigent this already exists conceptually as session-token-stats (Dirigent `goals/session-token-stats.md`). + +### Where it runs +- **Standalone stdio MCP** (rmcp) over `stable_core` + `dirigent_anth`. It can be configured in the orchestrator's MCP config only. Forks run `--strict-mcp-config` and would get it only if their base was trained with it (fingerprint), see §4. +- **Dirigent-hosted** MCP endpoint: same tools, answered from the archivist's pre-computed stats, plus push signals. +- **CLI** `stable tokens self|session|children|tree` for agents without MCP (Codex, forks). It costs one Bash permission rule. + +### Caveat +Reading one's own tokens costs tokens. A `tokens_self` call returns about 200 tokens, and calling it every turn adds up. Its usefulness for decisions ("hand off now", "rebase now") is an open question for a trial, not a given. + +--- + +## 7. Lightweight agent access (launch, resume, fork, message, status) + +The user's wider aim: a lightweight way to reach agents through Dirigent later. A minimal verb set that generalises the fork helper beyond the Athanor: + +| Verb | Meaning | Claude Code mechanism | Notes | +|---|---|---|---| +| `launch` | new session from a profile (fingerprint) + prompt | `claude -p --session-id` | profile = named, stored launch fingerprint (model id, tools, mode, agent file, cwd, env pins) | +| `resume` | one more turn in an existing session | `--resume ` | refuses if a process already holds that id (interleave hazard, `2c`) | +| `fork` | branch a session | `--resume --fork-session` | profile taken from the source session, not the caller | +| `message` | send a turn and return without waiting | `resume` with detached supervision | a mailbox per session. Messages are queued and sent one at a time, never concurrently, because concurrent turns on one id interleave | +| `status` | running/idle/stalled, last event, turns, tokens, warm state | stable registry + transcript tail | also covers sessions the stable did not launch (read-only: last activity, tokens) | +| `wait` | block until done or timeout | file/registry poll or SSE | bounded per call (Bash 10 min, MCP timeout) | +| `stop` | kill the process tree | Job Object / process group | | +| `list` | sessions and profiles, filterable by run, project, parent | registry + transcript index | | + +**CLI**: `stable agent launch --profile impl-opus --prompt f.md`, `stable agent message --prompt f.md`, `stable agent status `, and so on. JSON on stdout, the helper's exit codes. + +**MCP**: `agent_launch`, `agent_resume`, `agent_fork`, `agent_message`, `agent_status`, `agent_wait`, `agent_stop`, `agent_list`. Same JSON. + +**Through Dirigent**: +- A headless Claude session becomes one more connector type next to ACP connectors. +- The same verbs map to REST plus SSE. +- Messages from the UI or from Matrix (`dirigent_matrix`) go into the mailbox. +- Transcripts are archived as usual. +- The ACP bridge (`dirigate`) is the precedent for exposing a session over a second protocol. + +Limits to state up front: +- These are **turn-based** headless sessions, not live interactive ones. Messaging a *running* interactive TUI session is not possible this way. `--input-format stream-json` on one long-lived process could keep a session open for streamed turns, but it cannot branch (design §Rejected). +- Agent-tool subagents inside a session are not addressable from outside. You can read them, not message them. +- Every `resume`/`message` re-passes the full profile. That is the fingerprint rule applied to every verb, and it is why a profile store is central. + +--- + +## 8. Workpad tasks that would fit (all **candidates, confirm**) + +| # | Task | Path | Why it fits | +|---|---|---|---| +| C1 | Orchestrator introspection of subagent token use / signals into running work | `goals/lean-athanor-0.2.1.md:28` ("Later"); also `goals/lean-athanor.md:44`, `plugins/spire/skills/implementation/references/models.md` "Future work"; `journals/2026-10-05-21-athanor.md:20` ("dirigent will work on orchestrator token introspection / signals") | This is the "token-count access for agents" and "signals" responsibility directly. `stable tokens/context` plus form 2's SSE signals. Context sizes stop being guesstimates for sessions the stable launched or can read. | +| C2 | Headless-forks goal, "Later" items | `goals/headless-forks.md:52–53` | Explicitly says a "Dirigent harness / MCP tool takes over the helper's job ('stable for the constables')" and asks for "token-count access for agents beyond the helper's per-call returns". This is the app's charter. | +| C3 | Transcript-based token accounting for experiments and builds | `reports/2026-10-15-22-spire-vs-superpowers-plan-and-build-experiment.md` §Time and tokens (l.287–305), proposals l.402, l.414; open thread in `journals/2026-10-15-21-grotto-work.md:98` ("Was the superpowers SDD controller's transcript summed…?") | The report recomputed four-column totals from transcripts, deduplicated by message id, and could not price them. It flags that harness-reported numbers are "not sums". It asks to record tokens for inline blocks and to include Igor-dispatched reviewers. `stable tokens --dedup-by message-id --price` is that job as a deterministic tool. | +| C4 | Harness behaviour across modes | `goals/harness-behavior-investigation.md` (open since 2026-09-18) | Questions 2, 4 and 5 (system-prompt structure per mode, context pressure, model selection and fallback) are measurable by launching the same prompt under controlled variants and recording `init` + `modelUsage`. Variants: Desktop env, scrubbed, `--bare`, plan vs. dontAsk. The research already found one such effect (plan mode silently upgrades Haiku to Sonnet, `F2a`). `stable probe` would be the rig. It covers headless/CLI variants only, not remote-control UI behaviour. | +| C5 | "Read vs. cat" nested-CLAUDE.md harness factor | experiment report §6 (l.443ff.); journal l.30 (auto-mode prompt carries the same nudge) | Another measurable harness behaviour: a probe run with a known nested CLAUDE.md, comparing transcripts for whether it loaded. Same rig as C4. Weaker fit: a one-off experiment, not ongoing bookkeeping. | +| C6 | Worktree tooling gaps | experiment report l.315–320 | Slug derived from the ticket key (a second worktree refused), no `--base`, a confirmation that silently did not appear because stdin was closed, shared-venv worktrees with no `.venv`. These are gaps in the *team's worktree tool*, which is not Claude Code. Fit is **partial**: the stable cares about cwd as a fingerprint field (`F4d`: forks must run in the base's cwd), and parallel tool-using forks may want per-fork worktrees (research caveat, design R3). A `stable fork --worktree` that creates and records a worktree would touch this. Fixing the worktree tool itself is out of scope. | +| C7 | Test-runner locks / "one test runner per machine" | experiment report l.320 (rule stale since per-worktree test DBs); design §8.3 (parallel forks never run shared-state tests); Dirigent's 13 Windows path/lock test failures (`reports/dirigent/2026-10-15-chatgpt-importer-handoff-lean-finish.md`, rules) | A stable that runs parallel forks needs a machine-level resource lock ("only one fork runs the suite at a time" or "these N worktrees are isolated") as policy data in its registry, not in agent prose. Fit is **partial**: the lock primitive fits, but the per-project knowledge of what is safe belongs in project config (`spire:project` `## Build`). The 13 Dirigent failures are a separate bug ticket, not this app. | +| C8 | Spend-limit and plan-usage awareness | `reports/dirigent/2026-10-14-chatgpt-importer-retrospective.md` §5 "Spend" ("Check the API budget before a long serial run. A kill cost 3 h."); `research/2026-10-05-21-athanor-usage-dirigent.md` §2.2 (six agents killed by HTTP 429, partial state left); design R11 | Plan gauges (`dirigent_anth::anth_usage`) plus per-call ledger give a deterministic preflight ("refuse parallel forks above X %"), a TTL-bucket prediction and a clean "stop before kill" state. Strong fit for form 2. | + +Related items outside this repo, found during the Dirigent look, are context only and not on the list: Dirigent `goals/session-token-stats.md` (pre-computed four-column stats per session) and `brainstorm/dirigent_as_acp.md:191` ("token metering & caps… per connector or per session"). + +I looked and did not find any test-runner-lock task as its own goal or plan in this workpad. C7 is assembled from mentions. `brainstorm/` has nothing on tokens or harnesses. Its `2026-11-15-implementation_thoughts.md` is about the design-to-plan stages, and its date is in the future relative to today. + +--- + +## 9. Sequencing options + +| Option | Order | For | Against | +|---|---|---|---| +| S1 Helper first, Rust later | uv helper → trial → freeze contract v1 → `stable_core` + CLI → Dirigent embeds → MCP face | The trial decides whether forks pay at all (design "what would make this wrong") before any Rust is written. The contract is proven by use. | Two implementations of the rules for a while. The Python state-file schemas become a compatibility burden. | +| S2 Rust CLI replaces the helper before the trial | skip or shorten the uv helper; trial on `stable` | One implementation. It fixes R1 and R14 with a real supervisor. | Builds a Rust app before knowing whether the mechanism pays. The uv decision (design amendment 01:33) was just made. | +| S3 Dirigent first | build the feature in Dirigent; CLI as its client; standalone later | The server solves R1 and owns time. Archivist/anth reuse is immediate. Signals arrive early. | Users without Dirigent (other machines, Codex hosts) get nothing. It couples Spire runs to a running server. | +| S4 Accounting first | `stable tokens/context` (read-only, transcript-based) as the first Rust deliverable, independent of forks | Lowest risk: no process launching. It serves C1, C3 and C8 right away, works on past sessions, and reuses `dirigent_anth` almost as is. | Does not touch the fork mechanism, which is the goal's core. | +| S5 MCP early | MCP face over the uv helper | Removes the Bash permission question (R4) for the Athanor. Reaches Codex. | The design rejected it for now. Wait timeouts remain, and the face over a Python helper would be rewritten later. | + +S4 is independent and can combine with any of the others. + +--- + +## 10. Risks + +### Claude Code CLI churn +- Flags and defaults move fast. In the inputs alone: + - Tasks were removed by model at 2.1.233, then moved to an allowlist at 2.1.268. + - Fork mode became the interactive default at 2.1.232. + - Cost became cumulative on resume at 2.1.277. + - Cache-TTL settings arrived at 2.1.242. + - Model aliases change meaning by CLI version (`F2c`). + - Two installs on one machine were 16 patch versions apart (`F5a`). +- A Rust app compiled against one event shape breaks silently when `result` or `init` fields change. Mitigations: + - Parse leniently, as `dirigent_anth` already does ("handles unknown fields"). + - Record `binary.version` per call (design already does). + - Keep a minimum/maximum tested-version table with a `where` warning. + - Run a small contract test (T1–T7 of the research) on each new CLI version, possibly as `stable selftest`. +- `stream-json` field parity with `json` is still an assumption (R5), and `apiKeySource` is assumed. +- The Agent SDK (TypeScript and Python, from memory; I did not check for a Rust SDK) spawns the same CLI. It does not insulate against churn, so a Rust app calling the CLI directly is in the same position as the SDKs. + +### OAuth +- The 1 h TTL needs a subscription within plan usage (`F4a`). `--bare` refuses OAuth (`F3d`). So the app must run without `--bare`, which means discovery stays on: plugins, skills, hooks, CLAUDE.md, claude.ai connectors (`5a`, `F3b`). That drift surface has to be pinned by flags (`--strict-mcp-config`) and detected by `init` comparison (R9). +- OAuth tokens refresh under the user's `~/.claude`. A daemon or server must run as the same user and must not race the interactive CLI's refresh. Whether concurrent refreshes from many headless processes are safe is untested. +- Plan policy: many parallel headless sessions on one subscription may hit usage limits faster, or fall under terms about automated use. Whether subscription OAuth may be used for unattended orchestration at scale was not checked here. **Open, for the user.** +- If the user ever switches to an API key: 5 min TTL, keep-warm economics change (break-even 11 pings instead of 19, design §5), and billing becomes real cost, not notional. + +### Cache-TTL rules +- Docs-only: hit-resets-timer (`F4c`), cross-cwd behaviour (`F4d`), and which prefix a ping refreshes (design §5 "Assumed"). Preflight 5 in the design is the only wall-time test planned. +- Silent downgrade: over the plan limit the main bucket drops to 5 min with no error. Only the `ephemeral_5m` vs `ephemeral_1h` split in `usage.cache_creation` reveals it (design `ttl_downgrade`). A long-lived daemon pinging on a stale 1 h assumption would waste money. The rules must re-derive TTL from each call's usage, as the design does, not from config. +- Claude Code upgrades change the system prompt and so invalidate every base (`F5b`; prompt-caching docs "Upgrading Claude Code"). A cross-run base store (forms 1 daemon / 2) must expire all bases on binary drift. A Desktop auto-update can trigger that mid-run. +- Pricing ratios (read 0.1, 1 h write 2.0, 5 min write 1.25) are assumed and not re-checked (design §5). The price table must be data, not code. + +### Other +- **R1 detached survival** decides the CLI's architecture on Windows (§2). Form 2 avoids it. +- **Agent-visible token counts** may change agent behaviour. An agent that can read its own context size may start self-truncating or rushing. Whether exposing it helps is a question for the trial, not an assumption. +- **Safety**: forks with full Bash under `dontAsk` (R16). A Dirigent-hosted stable could attach fermata by default. A standalone CLI cannot assume it. +- **Two writers**: the design keeps the run log single-writer (paste, not append). Form 2 pushing into workpad files would reopen that question. + +--- + +## 11. Gaps and contradictions + +- **Runtime of the near-term helper**: the design body specifies Node (`forks.mjs`, §3.1, Decision 1). The 01:33 amendment and this brief say Python via uv (`forks.py`, PEP 723). The amendment wins, but the design body was not rewritten. The plan (`plans/2026-10-16-02-headless-forks.md`, named in the journal) is not in the repo yet, so I could not check which helper commands survived planning. This report assumes the design §3.4 verb set. +- §6 relies on some things from memory that I did not verify: status-line and hook payload fields (`session_id`, `transcript_path`, context-window fields). The docs pages for hooks and status line should be checked before the token MCP's identity design is fixed. +- `CLAUDE_CODE_SESSION_ID` as the way for an agent to learn its own session id is unverified: it may be the host's id, not the subagent's. Agent-tool subagents write their own transcripts under the parent session (`dirigent_anth` "Sub-Agent Loading"), so "own tokens" for an Agent-tool subagent probably needs the parent id plus the agent id. +- I did not check: + - whether Codex has a fork/resume-by-id equivalent; + - OAuth concurrency or terms-of-use for unattended headless runs; + - whether a Rust Claude Agent SDK exists; + - `JOB_OBJECT_LIMIT_BREAKAWAY_OK` on Claude Code's Bash tool job. +- Dirigent facts come from crate `CLAUDE.md` heads and a few greps, not a full read. "Reusable as is" for `dirigent_anth`/`dirigent_process` is a reading of their stated dependencies, not a build test. +- Web budget used: 1 search, 0 page fetches. + +--- + +## 12. Ideas for bundling + +Labels: **[from workpad]** means a bundle of the §8 candidates, grouped by shared mechanism. **[proposal, researcher]** means my own idea; nothing in the inputs asks for it. Every item is for the user to accept, reshape or drop. + +### Bundle A: "tokens" (read-only, ships first) +- **[from workpad]** C1 orchestrator introspection, C2's "token-count access for agents", C3 transcript-based accounting (dedup by message id, priced, reviewers included), C8 plan-usage awareness. Dirigent side: `goals/session-token-stats.md` (pre-computed four-column stats). +- Shared mechanism: transcript reader + lineage + price table (`dirigent_anth` + `stable_core`). No process launching, so neither R1 nor OAuth risk applies. +- Deliverable: the token MCP (§6) and `stable tokens` CLI. +- **[proposal, researcher]** First consumer: rerun the 2026-10-15 experiment's accounting with it and settle the journal's open thread ("was the SDD controller summed?"). The result can be checked against the hand-made numbers, which also tests the tool. + +### Bundle B: "stable" (fork mechanism) +- **[from workpad]** C2 (the helper's job, taken over), C6 partial (`fork --worktree`), C7 partial (resource locks for parallel forks). +- Shared mechanism: launch profiles, fingerprint, supervisor, registry. +- **[proposal, researcher]** Gate it on the uv helper's trial (design §10). If forks do not pay, Bundle B shrinks to the agent-access verbs of §7 without keep-warm logic. + +### Bundle C: "probe" (harness measurement rig) +- **[from workpad]** C4 harness-behaviour investigation, C5 Read-vs-cat nested CLAUDE.md. +- **[proposal, researcher]** `stable probe `: N launch variants × one prompt. It records `init`, real model, tool list, prompt-prefix hash and tokens, and diffs them. It reuses Bundle B's launcher, so it needs B's process layer but none of its bookkeeping. It would also catch CLI churn: run the same probe spec after each Claude Code update as a contract test (§10). + +### Bundle D: "agent access via Dirigent" +- **[from workpad, Dirigent]** `brainstorm/rework_claude_concept.md` "Integrations → claude" facade; `brainstorm/dirigent_as_acp.md` token metering and caps. +- **[proposal, researcher]** A headless-Claude connector in Dirigent that implements the §7 verbs. Tokens come from Bundle A, launching from Bundle B. Messages arrive from UI, REST or Matrix into a per-session mailbox. Fermata hooks attach to every launched profile by default. + +### Further proposals (researcher, unprompted) +- **[proposal, researcher]** A **profile file format**, a TOML launch fingerprint (`profiles/impl-opus.toml`) that is checked into a project. Spire's `spire:project` `## Build` section could name profiles instead of models. It is shared by the helper, the CLI, Dirigent and MCP, and it is the smallest artefact that makes all forms agree. +- **[proposal, researcher]** A **frozen JSON contract spec** under `docs/workpad/specs/` covering the helper's stdout objects, ledger line and exit codes, with `"schema": 1`. Without it, "keep the contract stable" across forms is only an intent. +- **[proposal, researcher]** **Budget guard** as a cross-cutting rule: refuse launches above a plan-gauge threshold, and write a clean "stopped for budget" state instead of being killed by a 429 (C8; Dirigent's 3 h loss). +- **[proposal, researcher]** **Self-test on CLI update**: `stable selftest` replays the research's T1–T7 (tiny, ~$0.13) and refuses to launch on a CLI version whose event shape or fork semantics changed. +- **[proposal, researcher]** Name split, if wanted: *stable* for the app and fork manager, *ledger* or *tally* for the token MCP. The token part is useful without the rest, and a separate name makes it easier to ship alone.