diff --git a/docs/workpad/reports/2026-10-17-gremlins-mvp-build.md b/docs/workpad/reports/2026-10-17-gremlins-mvp-build.md new file mode 100644 index 0000000..5719c02 --- /dev/null +++ b/docs/workpad/reports/2026-10-17-gremlins-mvp-build.md @@ -0,0 +1,97 @@ +# Gremlins MVP: build report + +- Date: 2026-10-16/17 · Orchestrator model: claude-opus-5-5 · Gremlin model: claude-opus-5-5 (all calls) +- Branch `gremlins-mvp` (worktree `.claude/worktrees/gremlins-mvp`), from `main` at `b8affd8`. Not pushed, not merged. +- Spec: [`2026-10-17-00-gremlins-mvp-spec.md`](../plans/2026-10-17-00-gremlins-mvp-spec.md). Tool: the prototype at `.claude/worktrees/headless-forks/plugins/spire/skills/gremlins/gremlins.py`, unmodified. +- Wall time: about 21 minutes, 23:12:58 to 23:33:35, against a 90-minute timebox. Fallback: **not needed**; the prototype did not misbehave once. + +## How it ran + +One base (B1) was trained on the spec, the verdict and the prototype source. It wrote no code. It wrote 70 lines of trunk notes, which pinned every cross-module seam: signatures, return shapes, the state-dir layout, the ledger and record keys, and its own decisions where the spec was open. Then eight gremlins were forked from B1, at most two running at a time (`spawn --mode parallel --paths …`, then `wait --any` / `--all` with Bash timeout 600000): + +1. Wave 1: `stream` + `launch`, then `store` + `call`, each pair started as soon as a slot freed up. Committed in `aff29d6`. +2. Wave 2: `cli` + `entry`. `entry` is `gremlins.py` plus SKILL.md. Committed in `9209e7d` and `fd75777`. +3. Wave 3: `tests-unit` (started in `entry`'s freed slot, since it does not depend on cli), then `tests-cli`. Committed in `d662790`. +4. Integration fix: `79df026`. + +Every gremlin finished `ok` on its first attempt. No gremlin needed a follow-up turn, and none touched another gremlin's files. Each one wrote a note that listed every function with its box. + +## Gremlin metrics (from the prototype's results and ledger) + +| Gremlin | Module | Wall s | Cost delta $ | Cache | Cache read | Lines written | +|---|---|---|---|---|---|---| +| base B1 (train) | trunk notes only | 407.8 | 1.9725 | n/a | 1,085,902 | 0 code (70 notes) | +| stream | `grm/stream.py` (+`__init__.py`) | 104.3 | 0.5175 | hit | 924,301 | 215 | +| launch | `grm/launch.py` | 66.7 | 0.4196 | hit | 783,210 | 182 | +| store | `grm/store.py` | 62.2 | 0.3815 | hit | 903,383 | 116 | +| call | `grm/call.py` | 80.0 | 0.4198 | hit | 908,838 | 86 | +| cli | `grm/cli.py` | 230.2 | 1.2211 | hit | 1,667,879 | 359 | +| entry | `gremlins.py`, `SKILL.md` | 52.2 | 0.3836 | hit | 916,241 | 14 + 54 | +| tests-unit | `tests/test_{store,launch,stream,call}.py` | 80.0 | 0.5441 | hit | 834,361 | 218 | +| tests-cli | `tests/fake_claude.py`, `tests/test_cli.py` | 197.3 | 1.1388 | hit | 1,982,799 | 275 | + +- Base cost: $1.9725. The eight spawns together cost $5.0260. Build total (prototype `usage`): **$6.9985**, 9 calls, 0 cold starts, 8 of 8 cache hits. +- The live smoke of the MVP cost another $0.2204 (4 calls). That brings the grand total to about **$7.22**. + +## Final line counts against the spec's estimates + +| Module | Lines | Estimate | Reason for the difference | +|---|---|---|---| +| `grm/store.py` | 116 | 70 | docstrings; `init_state` and `SUBDIRS` added (trunk decision) | +| `grm/launch.py` | 182 | 110 | the kept scrub constants and the tool lists; `binary_version` added | +| `grm/stream.py` | 215 | 170 | `iso`/`parse_iso` moved here; `TOKEN_KEYS`; `mismatch` kept (4 lines, not used) | +| `grm/call.py` | 86 | 60 | feeder and reader threads; the child is killed if `on_start` raises; a second signal escalates to `kill()` | +| `grm/cli.py` | 359 | 200 | `delta_header` verbatim (~25); JSON argparse errors (~30); explicit job, ledger and base dicts (~40); status; docstrings | +| `gremlins.py` | 14 | (in cli) | | +| **Code total** | **972** | **610** | The verdict estimated 600–800. No failure path was dropped, and nothing outside spec §4 was built. | +| `tests/` | 493 | about 300 | per-file boilerplate; the fake has 7 scenarios; B4 needs a subprocess and per-platform signalling; one extra test (below) | +| `SKILL.md` | 54 | not in the spec | see Deviations | + +## Test and smoke results + +- **Unit tests:** `uv run --no-project --python 3.12 python -m unittest discover -s tests`, run from the skill dir: **24 tests, OK, 7.0 s.** That is 14 module tests and 10 CLI tests, which cover every §6 case from B1 to B8. +- **Live smoke** (MVP; real `claude.exe` 2.1.299; scratch git repo; model `claude-opus-5-5`): + - `init`: `run.json` was written with the fingerprint, the argv and `fp_hash`, and a frozen preamble copy was made. + - `train B1`: `ok`, 5.0 s, $0.0969, session id taken from `--session-id`. + - `spawn s1`: `ok`, **`cache: hit`** (69,651 tokens read, 1,664 written), 36.5 s, cost delta $0.0431, `base_state: warm`. The note was written and the repo was left untouched. + - The ledger had **two lines**, each with `model`, `models_used`, six token columns, `cost_delta_usd`, `cost_cumulative_usd` and `hit_at`. + - `status`: B1 `warm`, `ttl_s` 3600, `ttl_left_s` 3566. +- **Parallel test (B3):** two `spawn`s, p1 and p2, ran as separate Bash `run_in_background` calls. Both started at 23:30:33 and ended at 23:30:44 and 23:30:45. Both completion notices arrived with the result JSON and exit 0, and both reported `cache: hit`. The run ended with two new job files, two new whole ledger lines (four in total) and two notes. +- **Processes afterwards:** `Get-CimInstance Win32_Process` shows no `claude -p`, `python` or `uv` process left. The only `claude.exe` processes are the Desktop app and its own sessions. The prototype's `list` showed nothing running, and `expire --all` retired its base B1, so no keep-warm ping can fire later. +- **Not run live:** the B4 `TaskStop` on a backgrounded `spawn` (spec §5 step 4), a B7 `rebase`, and the B8 gate on WSL. These paths are covered only by the fake-claude tests: CTRL_BREAK to the helper subprocess, and rebase argv and session id. + +## Deviations from the spec, each with its reason + +1. **`SKILL.md` was added**, although it is not in the module table. Reason: B3 and B4 only work if the orchestrator is told to run `spawn` via `run_in_background` and to stop it with `TaskStop`, and the skill file is where it is told. +2. **`templates/rebase.md` was reworded** ("a fork … becomes the new base `${new_base}` (phase …)" became "base `${base}` itself, resumed"). cli stopped passing `new_base` and `phase`. Reason: §3 said to use the templates as is, but the old text contradicts B7, which resumes without a fork and keeps the same base. The cli gremlin flagged it, and the fix is in `79df026`. +3. **`spawn.md`:** `${commit_rule}` became the fixed line "Commit only if the brief below says so", and `${mode}` was removed. This is per spec §3; the fixed line is so that the rule is still visible. +4. **Trunk-note decisions where the spec was open:** + - `build_fingerprint` takes `mcp_config` and returns `{fingerprint, argv, fp_hash}`. + - The argv builders take the stored run dict, so that a test's fake `argv` is honoured. + - `resume_argv` has no `--session-id`, because claude refuses it on a resume without a fork. + - The fingerprint adds `allowed_tools` and `disallowed_tools`, because `dontAsk` denies every tool that is not allowed. + - `resolve_binary` returns a str, not the prototype's tuple. +5. **`models_used` is ordered by usage, largest first**, not by name. Reason: by name, a haiku side-call could become `model` (= `models_used[0]`). +6. **The ledger line adds `hit_at`.** Reason: B6 derives warmth from the ledger, and the line's `ts` is the end of the call, not the time the cache was read. +7. **Job ids:** + - train is `-train`; it can be re-run only while there is no base file. + - rebase is `-rebase`; a failed rebase re-run overwrites its own job and raw files. + - spawn takes `--job J`, and an existing J is refused. +8. **The base's `cost_cumulative_usd` is stored unrounded.** Ledger and printed costs are still rounded once, at 6 decimals. Reason: a rounded basis made the deltas drift by 1e-6 (0.020001 instead of 0.02). +9. **`StreamState` prefers the result's `session_id` over the init event's.** Reason: spec §7, B7. +10. **Exit codes are 0 / 1 / 2:** 0 ok, 1 a call that ended `error` or `stopped`, 2 refused. `status` refuses a run with no `run.json`. +11. **No `SPIRE_GREMLIN_CHILD` refusal.** Reason: no box needs it, and it would break the helper's own tests when they run inside a gremlin. +12. **One extra test**, `test_call.test_tee_and_parse`. Reason: it is the only check that the prompt goes in on stdin as UTF-8 and that the stream is teed to the raw file and parsed. + +## Observations on the dogfooding + +- The prototype raised `rebase_due: phase_boundary` on every wave-2 and wave-3 spawn, because `--phase` was used as a wave label. The flag is advisory, so it was ignored. The delta header gave the forks the new commits instead. +- The trunk notes were what made the parallel work fit together: four modules written blind against each other imported cleanly on the first try. The only integration fix was the template wording. +- Each fork read 0.8 to 2.0 M tokens from cache, at about $0.38 to $1.22 per module. The base's $1.97 was spent once. + +## Open items + +- Live B4 check: a `TaskStop` on a backgrounded MVP `spawn`. +- Live B7 check: does a plain `--resume` report a cumulative `total_cost_usd`? The cost basis assumes it does. +- The B8 gate on WSL. +- Cross-process ledger appends on Windows are not strictly atomic (CRT seek + write). This is accepted per the trunk notes. The live parallel run gave whole lines.