📝 Gremlins MVP spec

docs/workpad/plans/2026-10-17-00-gremlins-mvp-spec.md

@@ -0,0 +1,140 @@
+# Gremlins MVP: implementation spec
+
+- Date: 2026-10-17 · Author model: claude-opus-5-5 · Timebox: 20 min, one agent
+- Inputs: [code verdict](../reports/2026-10-16-22-gremlins-code-verdict.md) (§3 keep, §5 minimal), [post-mortem](../reports/2026-10-16-23-headless-forks-postmortem.md) RC1–RC4, [brainstorm](../brainstorm/2026-10-16-22-grotto-plan-structure.md) "Steering sentences cut both ways"
+- Prototype: `.claude/worktrees/headless-forks/plugins/spire/skills/gremlins/gremlins.py` @ `ab1d945` (cited as `P:<line>`)
+
+**How to read this.** Every module, function and CLI command names the acceptance box (B1–B8) it serves. Anything that serves no box carries a one-line reason, or it is out. The builder does the same in the hand-back: code that traces to no box gets a one-line reason, and a missing failure path that a box names counts as a gap. Size figures are **estimates to justify deviations against, not caps**. Going over is fine with a reason, and so is coming in under, as long as no box's failure path was dropped to get there.
+
+## Acceptance boxes
+
+| | Box |
+|---|---|
+| B1 | `train` creates a base with one blocking `claude -p` call and returns its session id. The launch fingerprint (binary, full model id, flags, cwd, appended prompt) is stored once. |
+| B2 | `spawn` forks the base (`--resume <base> --fork-session`) with a brief, as one blocking call that reuses the fingerprint verbatim, so the prompt cache hits. |
+| B3 | Several gremlins run at once. The orchestrator runs each `spawn` through Bash `run_in_background` and gets the completion notice. The helper has no detach or supervisor code. |
+| B4 | `TaskStop` on the Bash call stops a gremlin. Verified on Windows 2.1.299, so there is no `kill` command (see §7 for the orphaned in-flight command). |
+| B5 | Usage per call comes from the result JSON: the real model from `modelUsage`, four token columns, and the cost delta against the base. One appended JSON line per call in a ledger. |
+| B6 | Warmth is reported only (last cache hit, TTL left). No pings. |
+| B7 | `rebase` resumes the base without forking and passes a short summary of the changes. Optional, tiny. |
+| B8 | No platform-specific process code: Windows Git Bash, WSL, Linux, macOS. The env scrub stays. |
+
+## 1. Ecosystem inventory: what already exists, so the helper must not rebuild it
+
+| Provider | Gives us | So the helper does NOT build |
+|---|---|---|
+| Harness | Bash `run_in_background` with a completion notice that carries stdout. A call that passes its timeout moves to the background and keeps running (post-mortem RC1). `TaskStop`. The orchestrator's own task list. | detach, supervisor, `wait`, heartbeats, the orphan watchdog, reaping of lost jobs, a "running jobs" poller (B3, B4) |
+| `claude -p` | `--session-id <uuid>` (id known before launch), `--resume … --fork-session`, `--append-system-prompt-file`, `--strict-mcp-config`, `--output-format stream-json --verbose`. Its events: `system/init`; `assistant` with per-request `usage`; `result` with `session_id`, `usage`, `modelUsage`, `total_cost_usd`, `is_error`, `api_error_status`. Session storage. Price maths. | price tables, transcript reading, cost computation, session bookkeeping beyond the ids (B1, B2, B5) |
+| Python stdlib | `subprocess.Popen` plus `terminate()` (portable), `signal`, `json`, `uuid`, `argparse`, `string.Template`, `os.replace`, `shutil.which` | ctypes, CIM, Toolhelp32, pid-reuse guards, process-tree walks, a lock manager (B8) |
+| uv | an interpreter for a PEP 723 script with no dependencies (`uv run --script`) | interpreter discovery (`supervisor_interpreter`) |
+| git | `log` / `diff --shortstat` for the change summary | its own change tracking (B7) |
+
+Concurrency without locks (B3): each job writes only its own `jobs/<J>.json`. Only `train` and `rebase` write `bases/<B>.json`. The ledger is append-only, one `write()` per line. Warmth is **derived** from the ledger (B6), so spawns never write base files.
+
+## 2. Module structure (`plugins/spire/skills/gremlins/`)
+
+Entry point: `gremlins.py`, a PEP 723 script with `dependencies = []` that puts its own directory on `sys.path` and imports the package `grm/`. Run it as `uv run --script gremlins.py <cmd> …`. The templates and `preamble.md` stay where they are.
+
+| Module | Purpose | Public functions | Boxes | Est. lines |
+|---|---|---|---|---|
+| `grm/store.py` | state-dir layout, atomic JSON, ledger append, run/base/job records | `state_dir(run) -> Path`; `read_json(p) -> dict\|None`; `write_json(p, obj)`; `ledger_append(run, line: dict)`; `ledger_lines(run) -> list[dict]` | B1, B3, B5 | 70 |
+| `grm/launch.py` | fingerprint, env scrub, binary lookup, argv | `resolve_binary(explicit=None) -> str\|None`; `scrubbed_env(binary, environ=None) -> dict`; `build_fingerprint(*, binary, model, effort, cwd, agent, preamble_path) -> dict`; `train_argv(fp, sid) -> list`; `fork_argv(fp, base_sid, sid) -> list`; `resume_argv(fp, base_sid) -> list` | B1, B2, B7, B8 | 110 (40 are the kept scrub constants) |
+| `grm/stream.py` | stream reader and accounting (pure functions, no IO) | `StreamState.feed(ev, t)`; `classify(ss, exit_code, stopped) -> (status, reason)`; `usage_from_result(result, before_mu, cost_before) -> dict`; `api_error_reason(res)`; `cache_verdict(first_usage, base_ctx)`; `infer_ttl(tokens, prev)`; `warmth(last_hit_ts, ttl_s, now) -> dict` | B2, B5, B6 | 170 |
+| `grm/call.py` | the only subprocess code: run one `claude -p` in the foreground, tee the stream to `raw/<J>.jsonl`, forward a stop | `run_claude(argv, prompt: str, env, cwd, raw_path, on_start) -> (StreamState, exit_code, stopped: bool)` | B1–B4, B8 | 60 |
+| `grm/cli.py` | argparse plus the commands; each command is one record → call → record sequence | `main(argv) -> int`; `cmd_init/train/spawn/rebase/status(args) -> dict` | all | 200 |
+| `tests/` | see §6 | | | about 300 |
+
+**Total: about 610 lines of code** (the verdict estimated 600–800).
+
+`call.py` details:
+- The prompt goes in on stdin, which avoids the Windows argv length limit.
+- `on_start(child_pid)` records the pid in the job file.
+- A handler for SIGTERM and SIGINT, plus `getattr(signal, "SIGBREAK", None)`, calls `child.terminate()` and sets `stopped`. Reason for the `getattr`: it is the one portable way to see Ctrl-Break on Windows, and it is not process code.
+- It uses stream-json rather than `--output-format json`. Reason: the first `assistant` usage is the only way to prove the B2 cache hit, and a stopped call (B4) leaves a partial trace behind.
+
+### Data model
+
+- **`run.json`** (B1; written once by `init`, never rewritten):
+
+  ```
+  {run, created, fingerprint: {binary, binary_version, model: "claude-opus-5-5[1m]", effort,
+   permission_mode, tools, agent, cwd, append_prompt: {path, sha256}, mcp_config, env_pinned},
+   argv: [...full fingerprinted argv...], fp_hash}
+  ```
+
+  Spawn reuses `argv` **as stored** (B2) and only appends `--resume/--fork-session/--session-id`.
+- **`bases/<B>.json`** (B1, B7):
+
+  ```
+  {base, session_id, train_job, commit, trained_at, context_tokens, model_usage, cost_cumulative_usd, ttl_s, rebases}
+  ```
+
+- **`jobs/<J>.json`** (B3, B4):
+
+  ```
+  {job, kind: train|spawn|rebase, base, session_id, status: running|ok|error|stopped, reason,
+   started, ended, child_pid, raw, note, cache, result_excerpt}
+  ```
+
+- **`ledger.jsonl`**, one line per finished or stopped call (B5):
+
+  ```
+  {ts, run, job, kind, base, session_id, model, models_used,
+   tokens: {input, output, cache_creation, cache_read, cache_creation_1h, cache_creation_5m},
+   cost_delta_usd, cost_cumulative_usd, cache, status, wall_s}
+  ```
+
+  `model` is `models_used[0]` from the `modelUsage` delta. The 1h/5m split stays because `infer_ttl` needs it (B6).
+- **Warmth** (B6): the last ledger line for base B whose `tokens.cache_read > 0` gives `last_hit`. The ttl is `infer_ttl` over that line's tokens. Output: `{base_state: warm|expiring|cold, warm_until, ttl_left_s}`.
+
+## 3. Reuse from the prototype (copy, then strip review-ID comments)
+
+| Into | Prototype source | Change |
+|---|---|---|
+| `stream.py` | `StreamState` P:1459–1522, `context_tokens` P:1525, `usage_from_result` P:1573–1601 (+`MU_FIELDS` P:158), `_API_ERROR_TEXT`/`api_error_reason` P:1604–1637, `cache_verdict` P:1640 (+`CACHE_HIT_RATIO` P:153), `infer_ttl` P:1650, `wrote_only_5m` P:1663, `base_warmth` P:1675–1690 (+`TTL_1H/5M` P:137) | `base_warmth` becomes `warmth(last_hit, ttl, now)`, margin `max(120, ttl/10)`, with no `warm_margin_s`. Round costs once, at 6 decimals (verdict §6 rounding finding). |
+| `stream.py` `classify` | the if-chain inside `_conclude` P:1771–1790 | rewrite it as one `match` over `(stopped, exit_code, ss.result, api_error_reason(...))` |
+| `launch.py` | `ENV_PINNED_LAUNCH`, `SCRUB_*` P:86–109, `scrub_class` / `_venv_prefixes` / `_under` / `scrubbed_env` P:1235–1282, `resolve_binary(_via)` P:1309–1335, `claude_argv` P:2652–2665, `canonical_json` / `_sha256_*` P:240–250 | drop `scrub_report`/`scrub_unknown` (only `where` uses them); `norm_path` becomes `os.path.abspath` |
+| `store.py` | `_retry`, `atomic_write_bytes`, `write_json_atomic`, `read_bytes_retry` P:829–866; `ledger_append` P:1046–1059 | the ledger append drops its `RegistryLock` precondition and per-job dedupe: one call writes once |
+| `cli.py` | `render_template` P:1738; `delta_header` P:2677–2697 + `git` P:1214 | used as is |
+| files | `templates/{train,spawn,rebase}.md`, `preamble.md` | remove `${commit_rule}` and `${mode}` from `spawn.md`. Reason: the default rule contradicted no-commit briefs (verdict §3), so the brief now carries commit instructions. |
+
+Do not port: `Platform`/`WindowsPlatform`/`PosixPlatform`, `RegistryLock`, `Supervisor`, `SupClock`, `reap_lost`, `kill_tree`/`process_tree`, ping and `tick`, `due_flags` and the drift checks, `wait`, `where`, `model_admission`, `--method`.
+
+## 4. CLI surface (`G = uv run --script gremlins.py`; every command prints one JSON object on stdout)
+
+- `G init --run R --model M [--effort E] [--cwd D] [--agent A] [--binary P]` writes `run.json` with the fingerprint and the rendered preamble (B1). It refuses to overwrite an existing run.
+- `G train --run R --base B --brief F` makes one blocking call with `--session-id`, then writes `bases/B.json` and prints `{status, session_id, cost, …}` (B1).
+- `G spawn --run R --base B --job J --brief F` makes one blocking fork with the spawn template plus `delta_header` and prints `{status, cache, note, cost, …}` (B2, B3, B5). Reason for the delta: a fork knows only the base's commit, and it reuses B7's function.
+- `G rebase --run R --base B --brief F` resumes the base without a fork; the prompt is the rebase template + `delta_header` + the brief. Afterwards it updates `bases/B.json` from the result. It warns when jobs of B are still marked `running` (B7).
+- `G status --run R [--job J]` shows bases with their warmth, plus the job records (B6; B3 for reading a result again).
+- There is no `kill` command. B4 was verified on Windows with 2.1.299: `TaskStop` on the backgrounded Bash call kills `claude.exe` and its bash wrappers.
+- Out of the MVP, each with its reason: `continue` (no box), `ping` (B6 says no), `usage` aggregation (no box; the ledger lines are greppable), `list`/`wait`/`expire` (the harness covers them).
+
+## 5. Build order (sequencing, not a ranking of value)
+
+1. `store` + `launch` + `stream` (ported) + `call` + `init`/`train`/`spawn`. **Gate:** a live run of `init → train → spawn` shows `cache: hit` and two ledger lines with real model, tokens and cost delta. Nothing else gets built before this gate passes.
+2. B3: two `spawn`s via `run_in_background` at once; both notices arrive, two job files, two ledger lines.
+3. B6: `status` with warmth.
+4. B4: the stop-signal path in `call.py`, then one live `TaskStop` on a backgrounded `spawn` to confirm the job ends without a ledger line and nothing beyond an in-flight command survives.
+5. B7: `rebase`.
+6. B8: run step 1's gate on one non-Windows host (WSL is enough).
+
+## 6. Tests (stdlib `unittest`; a fake `claude` = `[sys.executable, tests/fake_claude.py]` written into `run.json["argv"]`)
+
+The tests cover the happy paths and the failure paths each box names. Add no other tests without a one-line reason.
+
+- **Happy (B1, B2, B5):** `train` stores the base with the session id from `--session-id`. `spawn`'s argv starts with the stored `argv` byte for byte, then `--resume <base> --fork-session`. The ledger line has `model`, four token columns and `cost_delta = total − base cumulative`.
+- **B1 failures:** the binary is missing (error before launch, no job file); a non-zero exit or no `result` event gives `status: error` and no base file; an API error maps through `api_error_reason`.
+- **B2 failures:** unknown base gives an error with no launch; first-usage `cache_read` of 0 gives `cache: miss`, which is reported, not an error.
+- **B3:** two `spawn`s in parallel threads give two job files and two whole ledger lines.
+- **B4:** SIGTERM to a helper running a fake that sleeps leaves the child gone, the job `stopped`, and the partial raw file kept.
+- **B5:** `modelUsage` with two models gives both in `models_used`; a missing `total_cost_usd` gives `cost_delta: null`.
+- **B6:** no hit gives `cold`; 5m-only writes give `ttl_s` 300; a hit at `now − ttl + 60` gives `expiring`.
+- **B7:** the rebase argv has `--resume` and no `--fork-session`; the base record takes the result's `session_id`.
+- **B8:** the scrub drops `SCRUB_DROP` and `UV_*`, sets the pinned vars, and removes venv PATH entries.
+
+## 7. Open risks per box
+
+- **B4 (verified, with a known leftover).** `TaskStop` kills `claude.exe` and its bash wrappers. A command the gremlin had in flight (in the check, a `sleep 120` run through the gremlin's Bash) **survives as an orphan and runs to its own end**. That is accepted for the MVP. A later sweep is an open item, outside MVP scope; a full tree kill would need a Job Object, and B8 forbids that. On a hard kill, the helper's signal handler may never run, so the job file can stay `running` with no ledger line. The orchestrator knows which calls it stopped, so `status` reports the record as written. Reason the handler stays anyway: on POSIX and with Ctrl-C it does run, and it costs about 10 lines.
+- **B7.** Resuming without a fork may give the session a new id in `-p` mode. Take `session_id` from the result; never assume it stays the same. A `spawn` started during a `rebase` forks a session that is still being written. The helper only warns; keeping them apart is the orchestrator's job.
+- **B2.** A cache hit needs more than the same argv: the same env pins, the same cwd, and a system prompt that does not change (for example, the date rolls over at midnight). The helper reports `cache`; it does not prevent misses.