📝 Headless-forks post-mortem and Spire guidance text audit

This commit is contained in:
2026-10-16 23:03:06 +02:00
parent dff19572ab
commit 5bd945ea60
4 changed files with 536 additions and 0 deletions
@@ -0,0 +1,251 @@
# Post-mortem: the headless-forks / gremlins build
- **Date**: 2026-10-16
- **Author model**: claude-opus-5-5 (one agent, read-only apart from this file, 30-minute timebox)
- **Verdict**: **a failure.** The code works, but the build missed the brief by a factor of 4–5 in size and took most of a day. Most of the excess came from planning and process, not typing.
- **Sources**: journal `docs/workpad/journals/2026-10-15-21-grotto-work.md` (cited as J:line), the goal, design, plan and build plan on `main`, the code verdict `reports/2026-10-16-22-gremlins-code-verdict.md` (V:line), the brainstorm `brainstorm/2026-10-16-22-grotto-plan-structure.md` (B:line), and on the branch worktree `.claude/worktrees/headless-forks/docs/workpad/`: the run log `reports/2026-10-16-08-headless-forks-run.md` (RL:line), the preflight (PF:line), the reviews, re-checks and digests, plus `git log headless-forks`.
---
## 1. Outcome
**Asked for.** "Spire ships a **small deterministic helper** (a script in the skill …) that owns the `claude -p` command line. It covers: train a base, fork the base with a brief, update the base, report usage, and list or expire bases" (goal:17). The intent was to keep bookkeeping out of agent attention (goal:12).
**Delivered.** Branch `headless-forks` at `ab1d945`, 51 commits, not merged, Phase 3 not reviewed, block 3.8 left as WIP (J:196).
| Artifact | Lines |
|---|---|
| `gremlins.py` | 4,155 |
| `tests/test_gremlins.py` | 4,235 (about 180 tests) |
| SKILL.md, preamble, templates | 189 |
| Process documents (design 430, plan 257, build plan 337, three contract digests 869, run log 812, preflight 187, five reviews and checks 989, platform handoff 339, trunk notes 53) | about 4,270 |
That is about 12,700 lines of code, tests and process documents for a tool the verdict sizes at **600–800 lines plus about 500 lines of tests** (V:97-109). The verdict: "careful, consistent and well tested, and it works … still over-built by about four to five times", with "roughly two thirds" of it policing detached processes (V:16). A live happy path on Sonnet 5.5 ran clean for $0.09, with a cache hit on the fork (V:140-151).
**Time.** From the goal (01:23, J:73) to the stop (22:20, `ab1d945`) was **about 21 h of wall clock**. The build itself, from B0 (08:15) to the stop, took **about 14 h**.
| Bucket | Hours | Evidence |
|---|---|---|
| Push approval stall before the build (B0) | ~6.4 | J:102, J:104 |
| Unexplained gap in Phase 0 (10:13 → 12:16) | ~2.0 | RL:18, J:116 (probably an approval prompt; the Athanor guessed "host suspend") |
| Unexplained gap in block 1.2 (12:31 → 15:42) | ~3.2 | RL:105, J:120 |
| **Total waits, mostly approvals** | **~11.6** | J:130 says "~11 h" |
| Active build time, 08:15–22:20 minus the in-build gaps | **~8.9** | below |
Even with every approval wait removed, the build took **about 9 active hours** for a "small helper". That is the failure. The waits only made it look worse.
---
## 2. Timeline
Times are 2026-10-16 (+0200) unless marked. They come from commits and from the run log's `date` stamps.
| Time | Step | Duration | Notes |
|---|---|---|---|
| 10-15 21:23 | Session start; grotto orientation, experiment synthesis | — | not this build |
| 00:44–01:16 | Headless-fork research (local tests, $0.48) | ~0.5 h | J:52-64; this is where the mechanism was proven |
| 01:23 | Goal written, decisions settled | — | J:73-74 |
| ~01:25–01:50 | Design → plan → attack, three Opus agents | ~0.5 h | J:75-101 |
| 01:50 → 08:15 | B0 push waiting for approval | **6.4 h** | J:104 |
| 08:16–12:17 | **Phase 0** preflight | 4.0 h | 1 h is the TTL test by design (J:107); ~2 h is a gap (RL:18) |
| 12:18–12:29 | 1.1 skeleton, state, `where`, `init` | 8 min | already **1,124 lines** of code |
| 12:31–15:49 | 1.2 launch and supervise | 3.3 h | about 3.2 h of it is the clock gap; code grows to 2,403 |
| 15:50–15:56 | 1.3 smoke | 6 min | |
| 15:57–16:40 | Phase 1 review → fix → re-check | 43 min | 3 Important, 16 Minor; +245 code lines |
| 16:42–17:44 | **Phase 2** (2.1 15 min, 2.2 22 min, 2.3 8 min, smoke 8 min) | 1.0 h | code goes 2,648 → 3,640 |
| 17:44–18:22 | Phase 2 review → fix → re-check | 38 min | 2 Important, 14 Minor; +147 code lines |
| 18:23–18:44 | Phase 3 digest (684 lines) → pre-build check → amendment | 21 min | the check step was added by Igor (J:160) |
| 18:44–20:24 | **3.1 rename** | **1 h 40** | a sed job took the time of an audit (J:172-176) |
| 20:26–20:39 | 3.2 platform seam and gate | 13 min | +274 code lines |
| 20:41–21:43 | 3.3 N3, N4, kill flake | 62 min | the flake did not reproduce (RL:627) |
| 21:45–21:57 | 3.4 WSL check, 3.4a test fixes | 12 min | |
| 21:59–22:18 | 3.5–3.7 text, references, handoff | 19 min | handoff report is 339 lines |
| 22:20 | 3.8 stopped by the user; WIP commit | — | J:196 |
**Where the active ~9 h went:**
- About 2 h on Phase 0 probing, and 1 h waiting on the TTL test. This was the most valuable part.
- About 1 h writing the core in Phase 1. Block 1.2 is hidden inside a 3.3 h span.
- About 1 h on Phase 2 features, most of them heuristics (V:74-91).
- About 1 h 40 on review, fix and re-check loops, plus digest checks.
- About 3 h on Phase 3: the rename alone took 1 h 40; the seam over a supervisor that should not exist; one hour chasing a flake.
Each block owner was a fresh Opus 1M agent using **187k–351k tokens** (RL:66-67, RL:261-263, RL:236, RL:482). The Athanor reached about 515k tokens by the end of Phase 2 (J:149).
**Code growth** (`git show` on each commit; `gremlins.py` / tests):
| Commit | Point | Code | Tests |
|---|---|---|---|
| `d36ac2f` | 1.1 skeleton | 1,124 | 673 |
| `009a252` | 1.2 supervisor | 2,403 | 1,587 |
| `7b9b748` | P1 fixes | 2,648 | 1,976 |
| `5fda2e2` | 2.2 rebase/re-train | 3,420 | 3,020 |
| `4b92b51` | 2.3 usage | 3,640 | 3,378 |
| `d2d5104` | P2 fixes | 3,787 | 3,630 |
| `e9f1b06` | 3.2 seam and gate | 4,070 | 3,944 |
| `62eef3b` | 3.3 N3/N4 | 4,155 | 4,222 |
Two facts stand out. The skeleton alone, before any launch code, was already larger than the whole minimal version. And by the end of Phase 1 the code was about 3× the verdict's target, before a single review fix landed.
---
## 3. Root causes, ranked
Each candidate the user named is tested against the evidence. They are ranked by how much of the excess size and time they explain. Approval waits are ranked last on purpose: they cost hours but no lines.
### RC1. Phase 0 removed the design's main premise, and nobody re-planned. **Supported; the biggest single driver.**
- The design's shape: "Launch returns at once (detached supervisor)" (design:255). Its fallback was `--attach` under `run_in_background`, with the line "**The Athanor never relies on the background notification**" (design:149). Risk R1 was that detached children would die when the Bash call ended (design:373).
- Phase 0 found that a Bash call past its timeout "is **not killed**; it is moved to the background … and its completion notification arrived in this subagent" (PF:71). That falsifies the reason for detaching.
- What happened next: to keep pass criterion (d), "a Bash call that the tool kills on timeout" (build plan:223), the Athanor manufactured the kill with `TaskStop` (RL:25). The test was kept alive, and nobody asked whether it still mattered. The preflight says "no fallback beyond the ladder was needed" (PF:50). Igor made rung 3 (PowerShell/CIM hidden detach) the permanent default (RL:58). The open question "does the Phase 3 text want a line on it?" (RL:52) was routed as a documentation item (RL:507).
- The cost: two thirds of the code (V:16, V:60-72), meaning detach, supervisor, `SupClock`, orphan watchdog, heartbeats and reaping, the pid-reuse-safe tree kill, `RegistryLock`, and most of the platform seam. The seam then put an interface over machinery that should not have existed (470 lines for one tested platform, V:26, V:72).
- Igor noticed only at 22:11, after the user asked "an operating system?" (J:191-193).
### RC2. The goal and plan specified a large tool; "small" was a label. **Supported.**
- The goal's own criteria include computed keep-warm, expire and re-train rules (goal:23), rebase-or-delta policy (goal:27), agent-file specialisation, the fingerprint, and the env scrub (goal:17-29). Igor admits this: "small helper was a label, not a requirement" (J:194).
- The design and plan then turned each criterion into machinery:
- 13 commands (plan:88);
- a ping rule with an activity window and a break-even cap "at 19 and 11" (plan:204);
- five rebase and re-train thresholds plus binary drift (plan:205, design:210-216);
- `lost` detection and three watchdogs (plan:197).
- The verdict's "heuristics nobody asked for" (V:74) is half true. The **user** didn't ask for them. **Igor's goal and the design** did. The failure is that nobody turned the user's sentence into a size or a scope cap.
- The attack (build plan) found no STOP-level issues (build plan:18). It attacked correctness and harness risks, not size. So it *added* contracts (orphan kill, C1–C20) on top.
### RC3. No design or size criterion anywhere, and a review loop that only adds. **Supported. Weaker in lines than RC1/RC2, strong as a pattern.**
- No document sets a line budget, a module budget or a "simplest thing" test. Not the goal, the design, the plan, the build plan, the reviewer brief or the digest check.
- Reviews hunted bugs and found real ones (section 4), and every fix added code and tests:
- P1 fixes: +245 / +389 lines (`7b9b748`);
- P2 fixes: +147 / +252 (`d2d5104`);
- 3.3's N3/N4 fixes: +85 / +278.
- Both Phase 2 documents carry a "**Built but not asked for (on record only)**" section: about 25 unrequested additions, among them `--warm-margin-s`, `ping_missed`, `rebase_conflict` and six new job fields (phase2-review:212-228; phase2-recheck:135-146). None was cut. The section header says outright that recording replaces removing.
- The code now references 49 distinct review-finding IDs about 100 times (V:44). It reads "like layers of review fixes", and several mechanisms are "a patch on a patch" (V:66-67).
- In lines, review fixes account for about 480 of 4,155 (about 12 %). The loop didn't create the bulk. It **locked it in**: each pass hardened over-built code with more tests (now longer than the code) instead of asking whether it should exist.
### RC4. Plans without code structure; architecture invented per phase in self-written digests. **Supported.**
- The plan fixes a command surface, a result JSON and two state machines (plan:86-130). It does not fix modules, a data model or layering. "One dependency-free file" was the design's only structural statement (J:77).
- The Athanor filled the gap with its own contract digests (Phase 1 121 lines, Phase 2 64, Phase 3 **684**). Phase 3's digest alone is about as long as the whole minimal tool.
- **Every Important finding** in Phases 1 and 2 traced back to points these digests left undefined (J:123, J:150-154). Igor named the pattern himself (J:154).
- The result is a single 4,155-line file with a 150-line `_conclude` god function that mixes classification, eight flag rules, two near-duplicate dicts and the base state machine (V:78-84).
- The brainstorm draws the lesson directly: the plan needs "at least the structure of the code" (B:9).
- One caution: the experiment's "code in the plan adds nothing" result was measured on correctness and merge choice, not design quality (B:43; experiment report:13, :356). It doesn't contradict this root cause.
### RC5. Opus 1M everywhere, no budgets or timeboxes, and an orchestrator that explained instead of checking. **Supported.**
- Every block owner was Opus 1M except 2.3 (Sonnet) (RL:66-67, RL:261-263). No brief carried a time or token budget until the user forced one at 22:11 (J:177).
- The rename went to a block "called mechanical" with "a 10-part rename map and audit-grade checks, with no model downgrade and no timebox" (J:175).
- Its run-log note diffs 191/200 assert lines and classifies every remaining "fork" hit by line (RL:540-559).
- It took 1 h 40 against an estimated 10–15 min (J:174).
- The 3.3 flake hunt started 16 busy-loop processes and ran 10 instrumented reproductions on a pre-change copy, all to classify a flake that never came back (RL:627).
- Igor's admissions:
- he rationalised the rename ("touches every file") before checking (J:172);
- his "$25 spending cap" wording was misleading (J:109);
- he accepted the Athanor's "host suspend" guess for a gap that was probably an approval prompt (J:114, J:116);
- he passed the user's lean instructions on "late or diluted" (J:183). "Implement first, test at the end" arrived after the last Phase 2 code change (J:148).
- Process steps Igor added on his own:
- a full review, fix and re-check loop per phase;
- the pre-build digest check (J:160, J:181).
- The digest check did pay off (J:169), but it was another cold-start Opus read of 684 + 4k lines.
### RC6. Every block was a cold-start agent re-reading thousands of lines. **Supported as a cost driver; partly as a size driver.**
- Each block owner started fresh and read the plan, build plan, digests, run log and code. That is 5–6k lines by Phase 3 (B:8), at 187k–351k tokens per block.
- The successor boot section lists **six document groups** to read before touching code (RL:503).
- Size effect: nobody held the whole code (J:189), and by design the Athanor reads no code (goal:26). So no single agent was ever in a position to say "this whole stack is unnecessary". The irony is that this build lacked the very trained-and-forked implementer it was building (B:8).
### RC7. No prototype first. **Supported, and linked to RC1.**
- Phase 0 probed mechanisms (detach rungs, cache ratio, TTL, usage fields) with throwaway scripts. It never ran an end-to-end happy path of the product shape.
- A roughly 300-line `train → spawn → usage` script run under `run_in_background` would have exercised the completion notice directly. It would have made RC1 obvious at about 10:00, not 22:11.
- The verdict's minimal version is that prototype, written after the fact (V:97-132). The user says so: "it would have helped to have a prototype first, and then use it to write the real code" (B:26).
### RC8. Mid-build scope additions on top of over-built code. **Partly supported; a consequence, not a cause.**
- The rename, the platform seam, the path matrix, the WSL check and the cross-platform handoff were all added during the build, by user decision (goal:50-71, J:129-145). Together they were most of Phase 3 (about 3 h, about 515 code lines).
- They were reasonable requests against the code as it stood. The seam was asked for because the code had "12 scattered `if IS_WINDOWS` branches" (J:143), and those exist mostly because of the supervisor (RC1). This is listed for completeness, not as blame. The user wondered whether their own "chaotic" input caused this (B:18). The evidence points at the process.
### RC9. Approval waits. **Real, about 11.6 h, but not the main issue.**
- They are explained by the user being away while not in auto mode (J:116). They added wall clock and no lines.
- Removing them still leaves about 9 active hours and a 4–5× over-build.
**What the evidence does not support.** The single-file stdlib constraint (J:188, hypothesis 3) as a cause of size. It explains why everything sits in one file. It doesn't explain why there is so much.
---
## 4. What worked
- **The mechanism is verified and works.** `claude -p --resume <base> --fork-session` with an identical fingerprint gets cache hits:
- 0.9999 on a small base in the smoke (RL:154);
- about 1.0 on a 182k-token base (J:112);
- a clean live run in the verdict (V:147).
- **The cache and usage accounting is real know-how**: `StreamState`, `usage_from_result` with the 1h/5m split, cost deltas against the session's cumulative total, `cache_verdict`, `infer_ttl`, `base_warmth` (V:48-50). The smoke reconciled every ledger line against the raw `total_cost_usd` differences (RL:157).
- **Phase 0 findings that are expensive to re-derive:**
- the 1 h TTL, and that a hit refreshes it (J:107);
- unlisted models are substituted silently, not refused (J:112);
- plan mode upgrades Haiku;
- CIM detach is needed if one does detach;
- **the Bash tool backgrounds rather than kills, and the notice returns** (PF:71). This last one is a graduation candidate (J:246), and it is the finding that should have reshaped the build.
- **Env scrub constants**: an empirically gathered list of Desktop and SDK variables (V:52).
- **The reviews caught real bugs**, each fixed with a test that fails on the old code (J:127-128, J:156):
- I2: the pid-reuse tree walk could have killed `dwm`/`winlogon` on this host (phase1-review:22);
- I1: errors misreported as `model_refused`;
- Phase 2 I1: `delta_tokens` would have forced a rebase before every fork.
- **The digest check paid off at once**: one BLOCKER (gate tests that could never pass, and whose closed case passed for the wrong reason), plus 12 ADJUSTs (digest-check:9-26, J:169). The *point* is right: check the architecture before the build. The *vehicle* was a symptom of RC4.
- **The run log and journal made this post-mortem possible.** Every gap, admission and deviation is timestamped. Audit-grade logging was too expensive per block (RC5), but as a record it is excellent.
- **Discipline held**: no push or merge by the Athanor, no attribution lines, no leftover processes after any phase (RL:49, RL:194), and the visible-window bug was fixed and verified in 26 checks (J:111).
---
## 5. What the code is worth
The user's view, and this report agrees: the prototype **holds a lot of wisdom and logic and should not be downplayed; it was just too much** (B:25). Treat it as a prototype to learn from:
- **Keep as a reference:** `StreamState`, `usage_from_result`, `cache_verdict`/`infer_ttl`/`base_warmth`, `api_error_reason`, the `SCRUB_*` lists and `scrubbed_env`, `claude_argv` and the fork argv, `atomic_write_bytes`/`_retry`, the templates and preamble, and the ledger plus `usage --format md` (V:46-56). That is about 600 lines of distilled facts.
- **Keep as evidence:** the tests that pin real harness behaviour, such as the P0.2 seq-1 numbers in `RuleTests` (RL:106), the path matrix, and the WSL findings (RL:658-669, zombie liveness, PATH interop).
- **Keep as a negative example:** the supervisor stack and the heuristics layer, as a worked case of what correctness-only review and an unchallenged premise produce.
- **Don't delete it from history.** The user's rule is: commit before throwing out (B:24). The branch is the record (J:196).
The next step is the user's call (J:197, J:229): a minimal rewrite over blocking calls run with `run_in_background`, or straight to the Rust app. One check comes first: does `TaskStop` on a backgrounded Git Bash call kill the `claude.exe` grandchild (V:134)? PF:71 suggests it does ("kills the process tree by parent links"). A one-line confirmation on the target shape is still worth running.
---
## 6. Lessons: options for grotto and Spire
These are options, not verdicts. Each one names the root cause it addresses.
### Grotto (design → plan → handoff)
1. **The plan fixes the code structure** (RC4). The plan includes modules, the data model, the main interfaces, and where the special cases live. Options: a short skeleton in the plan (signatures plus the data model, no bodies), or a "structure" section that the attacker must attack. Then no builder writes a contract digest; a gap goes back to the planner.
2. **A size budget per artifact** (RC2, RC3). The goal or plan states an expected size: "helper ≤ 800 lines, tests ≤ 600". Crossing 1.5× is a STOP that goes to Igor. Options: hard cap vs. checkpoint; lines vs. commands or flags.
3. **Prototype first** (RC7, RC1). Before the full plan, one agent writes the happy path (about 300 lines, one timebox) on the real harness, and the plan is written from it. The prototype can be thrown away after it is committed.
4. **A premise list with re-plan triggers** (RC1). The design lists the premises its shape rests on (for example "R1: Bash kills long calls, so we detach"). Every preflight result is checked against that list, and a falsified premise forces a re-plan, not a documentation note. Option: the preflight report gets a mandatory "premises still standing?" table.
5. **Goals separate the user's words from Igor's criteria** (RC2). "Small" in the user's sentence becomes a measurable criterion. Criteria Igor added are marked as such, so the user can strike them.
6. **The attacker gets a simplicity pass** (RC2, RC3): "what is the smallest design that meets the goal, and which contracts can be dropped?", next to the correctness and conventions passes.
### Spire (Athanor, reviewer, Igor, implementation skill)
7. **The reviewer gets a design and size criterion** (RC3). Add findings classes such as "remove", "merge" and "simplify". "Built but not asked for" becomes a cut list by default, with keeping as the exception that must be argued. Option: a separate, cheap "subtractive review" at phase end that may only propose deletions.
8. **Fix passes have a line budget** (RC3). A fix that adds more than N lines, or a new flag or field, goes back to Igor as a design question.
9. **Briefs carry a model, a time budget and a token budget** (RC5). Mechanical blocks get a cheap model, a scripted transform (sed plus one test run) and a timebox, and the review checks them; they don't self-audit (J:176). Option: a default table in the implementation skill (mechanical → Sonnet or Haiku, 15 min; core → Opus, 60 min).
10. **Use the trained implementer for the build itself** (RC6). Once the helper exists, train one base on plan, structure and code, and fork it per block, instead of 5–6k-line cold starts. Until then: hand-off notes over full re-reads, and a fixed boot set of at most two documents.
11. **Igor checks before explaining, and adds no process step without asking** (RC5). For any anomaly (a long block, a clock gap, a cost), Igor reads the evidence first (run log, `ps`, the commit) and states it as unverified if he doesn't. New steps (digest checks, extra reviews) are proposed to the user with their cost.
12. **One review per build or per risk, not per phase by default** (RC3, RC5). The review, fix and re-check triple ran twice and cost about 1 h 20. Option: a review per phase only where the plan marks the phase as high-risk; otherwise one review at the end.
13. **The cheap gate excludes slow full suites** (J:146): already sent mid-build; accrete it into the implementation skill.
14. **A fresh Athanor per phase, booted from a short state section** (J:149). Keep this, but cap the boot reading (see 10).
### Measure next time
- Active time vs. wait time per block. Have the run log record "waiting on approval" when a prompt is pending, if the harness exposes it. Otherwise flag gaps over 10 min explicitly as unknown.
- Lines added per block and per fix pass, next to the token counts already logged.
---
## Gaps in this post-mortem
- The block 1.2 span (12:31–15:49) can't be split into work and waiting. The subagent's harness duration of 11,929 s covers both (RL:67).
- The Athanor's and Igor's own token use is unknown (RL:13, RL:193), so the full token cost of the build can't be given.
- The front-end time (design, plan, attack) is estimated from journal order. The journal has few timestamps there (J:73-101).
- Phase 3 was never reviewed, so its share of real bugs vs. over-build isn't assessed beyond the verdict.