📝 Gremlins MVP build report

2026-10-16 23:34:18 +02:00
parent 79df026d8c
commit 0493f03c98
1 changed files with 97 additions and 0 deletions
@@ -0,0 +1,97 @@
 # Gremlins MVP: build report
 - Date: 2026-10-16/17 · Orchestrator model: claude-opus-5-5 · Gremlin model: claude-opus-5-5 (all calls)
 - Branch `gremlins-mvp` (worktree `.claude/worktrees/gremlins-mvp`), from `main` at `b8affd8`. Not pushed, not merged.
 - Spec: [`2026-10-17-00-gremlins-mvp-spec.md`](../plans/2026-10-17-00-gremlins-mvp-spec.md). Tool: the prototype at `.claude/worktrees/headless-forks/plugins/spire/skills/gremlins/gremlins.py`, unmodified.
 - Wall time: about 21 minutes, 23:12:58 to 23:33:35, against a 90-minute timebox. Fallback: **not needed**; the prototype did not misbehave once.
 ## How it ran
 One base (B1) was trained on the spec, the verdict and the prototype source. It wrote no code. It wrote 70 lines of trunk notes, which pinned every cross-module seam: signatures, return shapes, the state-dir layout, the ledger and record keys, and its own decisions where the spec was open. Then eight gremlins were forked from B1, at most two running at a time (`spawn --mode parallel --paths …`, then `wait --any` / `--all` with Bash timeout 600000):
 1. Wave 1: `stream` + `launch`, then `store` + `call`, each pair started as soon as a slot freed up. Committed in `aff29d6`.
 2. Wave 2: `cli` + `entry`. `entry` is `gremlins.py` plus SKILL.md. Committed in `9209e7d` and `fd75777`.
 3. Wave 3: `tests-unit` (started in `entry`'s freed slot, since it does not depend on cli), then `tests-cli`. Committed in `d662790`.
 4. Integration fix: `79df026`.
 Every gremlin finished `ok` on its first attempt. No gremlin needed a follow-up turn, and none touched another gremlin's files. Each one wrote a note that listed every function with its box.
 ## Gremlin metrics (from the prototype's results and ledger)
 | Gremlin | Module | Wall s | Cost delta $ | Cache | Cache read | Lines written |
 |---|---|---|---|---|---|---|
 | base B1 (train) | trunk notes only | 407.8 | 1.9725 | n/a | 1,085,902 | 0 code (70 notes) |
 | stream | `grm/stream.py` (+`__init__.py`) | 104.3 | 0.5175 | hit | 924,301 | 215 |
 | launch | `grm/launch.py` | 66.7 | 0.4196 | hit | 783,210 | 182 |
 | store | `grm/store.py` | 62.2 | 0.3815 | hit | 903,383 | 116 |
 | call | `grm/call.py` | 80.0 | 0.4198 | hit | 908,838 | 86 |
 | cli | `grm/cli.py` | 230.2 | 1.2211 | hit | 1,667,879 | 359 |
 | entry | `gremlins.py`, `SKILL.md` | 52.2 | 0.3836 | hit | 916,241 | 14 + 54 |
 | tests-unit | `tests/test_{store,launch,stream,call}.py` | 80.0 | 0.5441 | hit | 834,361 | 218 |
 | tests-cli | `tests/fake_claude.py`, `tests/test_cli.py` | 197.3 | 1.1388 | hit | 1,982,799 | 275 |
 - Base cost: $1.9725. The eight spawns together cost $5.0260. Build total (prototype `usage`): **$6.9985**, 9 calls, 0 cold starts, 8 of 8 cache hits.
 - The live smoke of the MVP cost another $0.2204 (4 calls). That brings the grand total to about **$7.22**.
 ## Final line counts against the spec's estimates
 | Module | Lines | Estimate | Reason for the difference |
 |---|---|---|---|
 | `grm/store.py` | 116 | 70 | docstrings; `init_state` and `SUBDIRS` added (trunk decision) |
 | `grm/launch.py` | 182 | 110 | the kept scrub constants and the tool lists; `binary_version` added |
 | `grm/stream.py` | 215 | 170 | `iso`/`parse_iso` moved here; `TOKEN_KEYS`; `mismatch` kept (4 lines, not used) |
 | `grm/call.py` | 86 | 60 | feeder and reader threads; the child is killed if `on_start` raises; a second signal escalates to `kill()` |
 | `grm/cli.py` | 359 | 200 | `delta_header` verbatim (~25); JSON argparse errors (~30); explicit job, ledger and base dicts (~40); status; docstrings |
 | `gremlins.py` | 14 | (in cli) | |
 | **Code total** | **972** | **610** | The verdict estimated 600–800. No failure path was dropped, and nothing outside spec §4 was built. |
 | `tests/` | 493 | about 300 | per-file boilerplate; the fake has 7 scenarios; B4 needs a subprocess and per-platform signalling; one extra test (below) |
 | `SKILL.md` | 54 | not in the spec | see Deviations |
 ## Test and smoke results
 - **Unit tests:** `uv run --no-project --python 3.12 python -m unittest discover -s tests`, run from the skill dir: **24 tests, OK, 7.0 s.** That is 14 module tests and 10 CLI tests, which cover every §6 case from B1 to B8.
 - **Live smoke** (MVP; real `claude.exe` 2.1.299; scratch git repo; model `claude-opus-5-5`):
  - `init`: `run.json` was written with the fingerprint, the argv and `fp_hash`, and a frozen preamble copy was made.
  - `train B1`: `ok`, 5.0 s, $0.0969, session id taken from `--session-id`.
  - `spawn s1`: `ok`, **`cache: hit`** (69,651 tokens read, 1,664 written), 36.5 s, cost delta $0.0431, `base_state: warm`. The note was written and the repo was left untouched.
  - The ledger had **two lines**, each with `model`, `models_used`, six token columns, `cost_delta_usd`, `cost_cumulative_usd` and `hit_at`.
  - `status`: B1 `warm`, `ttl_s` 3600, `ttl_left_s` 3566.
 - **Parallel test (B3):** two `spawn`s, p1 and p2, ran as separate Bash `run_in_background` calls. Both started at 23:30:33 and ended at 23:30:44 and 23:30:45. Both completion notices arrived with the result JSON and exit 0, and both reported `cache: hit`. The run ended with two new job files, two new whole ledger lines (four in total) and two notes.
 - **Processes afterwards:** `Get-CimInstance Win32_Process` shows no `claude -p`, `python` or `uv` process left. The only `claude.exe` processes are the Desktop app and its own sessions. The prototype's `list` showed nothing running, and `expire --all` retired its base B1, so no keep-warm ping can fire later.
 - **Not run live:** the B4 `TaskStop` on a backgrounded `spawn` (spec §5 step 4), a B7 `rebase`, and the B8 gate on WSL. These paths are covered only by the fake-claude tests: CTRL_BREAK to the helper subprocess, and rebase argv and session id.
 ## Deviations from the spec, each with its reason
 1. **`SKILL.md` was added**, although it is not in the module table. Reason: B3 and B4 only work if the orchestrator is told to run `spawn` via `run_in_background` and to stop it with `TaskStop`, and the skill file is where it is told.
 2. **`templates/rebase.md` was reworded** ("a fork … becomes the new base `${new_base}` (phase …)" became "base `${base}` itself, resumed"). cli stopped passing `new_base` and `phase`. Reason: §3 said to use the templates as is, but the old text contradicts B7, which resumes without a fork and keeps the same base. The cli gremlin flagged it, and the fix is in `79df026`.
 3. **`spawn.md`:** `${commit_rule}` became the fixed line "Commit only if the brief below says so", and `${mode}` was removed. This is per spec §3; the fixed line is so that the rule is still visible.
 4. **Trunk-note decisions where the spec was open:**
   - `build_fingerprint` takes `mcp_config` and returns `{fingerprint, argv, fp_hash}`.
   - The argv builders take the stored run dict, so that a test's fake `argv` is honoured.
   - `resume_argv` has no `--session-id`, because claude refuses it on a resume without a fork.
   - The fingerprint adds `allowed_tools` and `disallowed_tools`, because `dontAsk` denies every tool that is not allowed.
   - `resolve_binary` returns a str, not the prototype's tuple.
 5. **`models_used` is ordered by usage, largest first**, not by name. Reason: by name, a haiku side-call could become `model` (= `models_used[0]`).
 6. **The ledger line adds `hit_at`.** Reason: B6 derives warmth from the ledger, and the line's `ts` is the end of the call, not the time the cache was read.
 7. **Job ids:**
   - train is `<B>-train`; it can be re-run only while there is no base file.
   - rebase is `<B>-rebase<n>`; a failed rebase re-run overwrites its own job and raw files.
   - spawn takes `--job J`, and an existing J is refused.
 8. **The base's `cost_cumulative_usd` is stored unrounded.** Ledger and printed costs are still rounded once, at 6 decimals. Reason: a rounded basis made the deltas drift by 1e-6 (0.020001 instead of 0.02).
 9. **`StreamState` prefers the result's `session_id` over the init event's.** Reason: spec §7, B7.
 10. **Exit codes are 0 / 1 / 2:** 0 ok, 1 a call that ended `error` or `stopped`, 2 refused. `status` refuses a run with no `run.json`.
 11. **No `SPIRE_GREMLIN_CHILD` refusal.** Reason: no box needs it, and it would break the helper's own tests when they run inside a gremlin.
 12. **One extra test**, `test_call.test_tee_and_parse`. Reason: it is the only check that the prompt goes in on stdin as UTF-8 and that the stream is teed to the raw file and parsed.
 ## Observations on the dogfooding
 - The prototype raised `rebase_due: phase_boundary` on every wave-2 and wave-3 spawn, because `--phase` was used as a wave label. The flag is advisory, so it was ignored. The delta header gave the forks the new commits instead.
 - The trunk notes were what made the parallel work fit together: four modules written blind against each other imported cleanly on the first try. The only integration fix was the template wording.
 - Each fork read 0.8 to 2.0 M tokens from cache, at about $0.38 to $1.22 per module. The base's $1.97 was spent once.
 ## Open items
 - Live B4 check: a `TaskStop` on a backgrounded MVP `spawn`.
 - Live B7 check: does a plain `--resume` report a cumulative `total_cost_usd`? The cost basis assumes it does.
 - The B8 gate on WSL.
 - Cross-process ledger appends on Windows are not strictly atomic (CRT seek + write). This is accepted per the trunk notes. The live parallel run gave whole lines.