📝 Gremlins MVP spec

docs/workpad/brainstorm/2026-10-16-22-grotto-plan-structure.md

@@ -27,6 +27,16 @@
 - The user wants the whole build bundled into a report that honestly calls it a failure. The approval waits cost time, but they are not the main issue.
 - **Guidance hasn't found its sweet spot yet.** Open question: how big are the skills in text, how much of it is filler, and is each skill truly written for an agent, or does it regurgitate knowledge the model already has?
 
+## Steering sentences cut both ways (user, 2026-10-16)
+
+- "Keep it small", "happy path first" and "delete what isn't needed" are themselves dangerous: they can fire the other way, into cut corners, missing error handling and deleted tests or needed code.
+- Igor's take on safer framing (options, not decisions):
+  - Tie size to requirements, not adjectives. Each function or module traces to an acceptance box, and anything that doesn't gets a one-line reason in the hand-back.
+  - A size *estimate* in the plan that the builder must explain deviations from. A budget to justify against, not a cap to obey.
+  - "Happy path first" as **sequencing**: the happy path runs end to end before edge cases get built. It's not a value judgement against edge cases.
+  - Reviews report "unrequested surface" as a finding category with the decision left to Igor and the user. Reviewers don't delete on their own.
+  - Pair every push with its counterweight in the same sentence, e.g. "cover the failure paths the boxes name; add no others without a reason".
+
 ## Answer: how much code can Opus 5.5 produce in one go?
 
 *(Igor, 2026-10-16, from the claude-api skill's model table)*

docs/workpad/journals/2026-10-15-21-grotto-work.md

@@ -231,6 +231,9 @@
   - Steering sentences match today's behaviour exactly: reviewer.md:22 "Answer every … flag … one by one", "Hunt …", run-log handoff fields with no length cap, attacker.md:25 asking for more scope.
   - No size, happy-path or delete guidance anywhere.
   - Stale text includes the athanor.md:32 Agent-tool fork and the zathras "Always pass model".
+- User: the MVP box list is good. Asked whether to use the current prototype to help implement it. Warned that "small / happy path first / delete what isn't needed" can backfire. Recorded the safer framing options in the brainstorm (trace to boxes, a size estimate to justify against, happy path as sequencing, unrequested surface as a finding for the user to decide).
+- User decided: build the MVP by **dogfooding** the prototype's happy path. Train a base on prototype + verdict + MVP spec, spawn one gremlin per module, fall back to a plain agent if the tool misbehaves, and don't fix the prototype. Dispatched in parallel: an MVP spec writer (opus, 20 min, ≤150 lines, the module structure fixed, the safer framing) → plans/2026-10-17-00-gremlins-mvp-spec.md; the TaskStop/claude.exe check (sonnet, 10 min).
+- TaskStop check (sonnet-5-5): TaskStop kills claude.exe and its bash wrappers. An in-flight grandchild command (sleep.exe) survives as an orphan and runs to completion. MVP decision: no kill command; document the orphan behaviour; a sweep is an open item. Sent to the spec writer.
 
 ## Adepts
 

docs/workpad/plans/2026-10-17-00-gremlins-mvp-spec.md

@@ -0,0 +1,140 @@
+# Gremlins MVP: implementation spec
+
+- Date: 2026-10-17 · Author model: claude-opus-5-5 · Timebox: 20 min, one agent
+- Inputs: [code verdict](../reports/2026-10-16-22-gremlins-code-verdict.md) (§3 keep, §5 minimal), [post-mortem](../reports/2026-10-16-23-headless-forks-postmortem.md) RC1–RC4, [brainstorm](../brainstorm/2026-10-16-22-grotto-plan-structure.md) "Steering sentences cut both ways"
+- Prototype: `.claude/worktrees/headless-forks/plugins/spire/skills/gremlins/gremlins.py` @ `ab1d945` (cited as `P:<line>`)
+
+**How to read this.** Every module, function and CLI command names the acceptance box (B1–B8) it serves. Anything that serves no box carries a one-line reason, or it is out. The builder does the same in the hand-back: code that traces to no box gets a one-line reason, and a missing failure path that a box names counts as a gap. Size figures are **estimates to justify deviations against, not caps**. Going over is fine with a reason, and so is coming in under, as long as no box's failure path was dropped to get there.
+
+## Acceptance boxes
+
+| | Box |
+|---|---|
+| B1 | `train` creates a base with one blocking `claude -p` call and returns its session id. The launch fingerprint (binary, full model id, flags, cwd, appended prompt) is stored once. |
+| B2 | `spawn` forks the base (`--resume <base> --fork-session`) with a brief, as one blocking call that reuses the fingerprint verbatim, so the prompt cache hits. |
+| B3 | Several gremlins run at once. The orchestrator runs each `spawn` through Bash `run_in_background` and gets the completion notice. The helper has no detach or supervisor code. |
+| B4 | `TaskStop` on the Bash call stops a gremlin. Verified on Windows 2.1.299, so there is no `kill` command (see §7 for the orphaned in-flight command). |
+| B5 | Usage per call comes from the result JSON: the real model from `modelUsage`, four token columns, and the cost delta against the base. One appended JSON line per call in a ledger. |
+| B6 | Warmth is reported only (last cache hit, TTL left). No pings. |
+| B7 | `rebase` resumes the base without forking and passes a short summary of the changes. Optional, tiny. |
+| B8 | No platform-specific process code: Windows Git Bash, WSL, Linux, macOS. The env scrub stays. |
+
+## 1. Ecosystem inventory: what already exists, so the helper must not rebuild it
+
+| Provider | Gives us | So the helper does NOT build |
+|---|---|---|
+| Harness | Bash `run_in_background` with a completion notice that carries stdout. A call that passes its timeout moves to the background and keeps running (post-mortem RC1). `TaskStop`. The orchestrator's own task list. | detach, supervisor, `wait`, heartbeats, the orphan watchdog, reaping of lost jobs, a "running jobs" poller (B3, B4) |
+| `claude -p` | `--session-id <uuid>` (id known before launch), `--resume … --fork-session`, `--append-system-prompt-file`, `--strict-mcp-config`, `--output-format stream-json --verbose`. Its events: `system/init`; `assistant` with per-request `usage`; `result` with `session_id`, `usage`, `modelUsage`, `total_cost_usd`, `is_error`, `api_error_status`. Session storage. Price maths. | price tables, transcript reading, cost computation, session bookkeeping beyond the ids (B1, B2, B5) |
+| Python stdlib | `subprocess.Popen` plus `terminate()` (portable), `signal`, `json`, `uuid`, `argparse`, `string.Template`, `os.replace`, `shutil.which` | ctypes, CIM, Toolhelp32, pid-reuse guards, process-tree walks, a lock manager (B8) |
+| uv | an interpreter for a PEP 723 script with no dependencies (`uv run --script`) | interpreter discovery (`supervisor_interpreter`) |
+| git | `log` / `diff --shortstat` for the change summary | its own change tracking (B7) |
+
+Concurrency without locks (B3): each job writes only its own `jobs/<J>.json`. Only `train` and `rebase` write `bases/<B>.json`. The ledger is append-only, one `write()` per line. Warmth is **derived** from the ledger (B6), so spawns never write base files.
+
+## 2. Module structure (`plugins/spire/skills/gremlins/`)
+
+Entry point: `gremlins.py`, a PEP 723 script with `dependencies = []` that puts its own directory on `sys.path` and imports the package `grm/`. Run it as `uv run --script gremlins.py <cmd> …`. The templates and `preamble.md` stay where they are.
+
+| Module | Purpose | Public functions | Boxes | Est. lines |
+|---|---|---|---|---|
+| `grm/store.py` | state-dir layout, atomic JSON, ledger append, run/base/job records | `state_dir(run) -> Path`; `read_json(p) -> dict\|None`; `write_json(p, obj)`; `ledger_append(run, line: dict)`; `ledger_lines(run) -> list[dict]` | B1, B3, B5 | 70 |
+| `grm/launch.py` | fingerprint, env scrub, binary lookup, argv | `resolve_binary(explicit=None) -> str\|None`; `scrubbed_env(binary, environ=None) -> dict`; `build_fingerprint(*, binary, model, effort, cwd, agent, preamble_path) -> dict`; `train_argv(fp, sid) -> list`; `fork_argv(fp, base_sid, sid) -> list`; `resume_argv(fp, base_sid) -> list` | B1, B2, B7, B8 | 110 (40 are the kept scrub constants) |
+| `grm/stream.py` | stream reader and accounting (pure functions, no IO) | `StreamState.feed(ev, t)`; `classify(ss, exit_code, stopped) -> (status, reason)`; `usage_from_result(result, before_mu, cost_before) -> dict`; `api_error_reason(res)`; `cache_verdict(first_usage, base_ctx)`; `infer_ttl(tokens, prev)`; `warmth(last_hit_ts, ttl_s, now) -> dict` | B2, B5, B6 | 170 |
+| `grm/call.py` | the only subprocess code: run one `claude -p` in the foreground, tee the stream to `raw/<J>.jsonl`, forward a stop | `run_claude(argv, prompt: str, env, cwd, raw_path, on_start) -> (StreamState, exit_code, stopped: bool)` | B1–B4, B8 | 60 |
+| `grm/cli.py` | argparse plus the commands; each command is one record → call → record sequence | `main(argv) -> int`; `cmd_init/train/spawn/rebase/status(args) -> dict` | all | 200 |
+| `tests/` | see §6 | | | about 300 |
+
+**Total: about 610 lines of code** (the verdict estimated 600–800).
+
+`call.py` details:
+- The prompt goes in on stdin, which avoids the Windows argv length limit.
+- `on_start(child_pid)` records the pid in the job file.
+- A handler for SIGTERM and SIGINT, plus `getattr(signal, "SIGBREAK", None)`, calls `child.terminate()` and sets `stopped`. Reason for the `getattr`: it is the one portable way to see Ctrl-Break on Windows, and it is not process code.
+- It uses stream-json rather than `--output-format json`. Reason: the first `assistant` usage is the only way to prove the B2 cache hit, and a stopped call (B4) leaves a partial trace behind.
+
+### Data model
+
+- **`run.json`** (B1; written once by `init`, never rewritten):
+
+  ```
+  {run, created, fingerprint: {binary, binary_version, model: "claude-opus-5-5[1m]", effort,
+   permission_mode, tools, agent, cwd, append_prompt: {path, sha256}, mcp_config, env_pinned},
+   argv: [...full fingerprinted argv...], fp_hash}
+  ```
+
+  Spawn reuses `argv` **as stored** (B2) and only appends `--resume/--fork-session/--session-id`.
+- **`bases/<B>.json`** (B1, B7):
+
+  ```
+  {base, session_id, train_job, commit, trained_at, context_tokens, model_usage, cost_cumulative_usd, ttl_s, rebases}
+  ```
+
+- **`jobs/<J>.json`** (B3, B4):
+
+  ```
+  {job, kind: train|spawn|rebase, base, session_id, status: running|ok|error|stopped, reason,
+   started, ended, child_pid, raw, note, cache, result_excerpt}
+  ```
+
+- **`ledger.jsonl`**, one line per finished or stopped call (B5):
+
+  ```
+  {ts, run, job, kind, base, session_id, model, models_used,
+   tokens: {input, output, cache_creation, cache_read, cache_creation_1h, cache_creation_5m},
+   cost_delta_usd, cost_cumulative_usd, cache, status, wall_s}
+  ```
+
+  `model` is `models_used[0]` from the `modelUsage` delta. The 1h/5m split stays because `infer_ttl` needs it (B6).
+- **Warmth** (B6): the last ledger line for base B whose `tokens.cache_read > 0` gives `last_hit`. The ttl is `infer_ttl` over that line's tokens. Output: `{base_state: warm|expiring|cold, warm_until, ttl_left_s}`.
+
+## 3. Reuse from the prototype (copy, then strip review-ID comments)
+
+| Into | Prototype source | Change |
+|---|---|---|
+| `stream.py` | `StreamState` P:1459–1522, `context_tokens` P:1525, `usage_from_result` P:1573–1601 (+`MU_FIELDS` P:158), `_API_ERROR_TEXT`/`api_error_reason` P:1604–1637, `cache_verdict` P:1640 (+`CACHE_HIT_RATIO` P:153), `infer_ttl` P:1650, `wrote_only_5m` P:1663, `base_warmth` P:1675–1690 (+`TTL_1H/5M` P:137) | `base_warmth` becomes `warmth(last_hit, ttl, now)`, margin `max(120, ttl/10)`, with no `warm_margin_s`. Round costs once, at 6 decimals (verdict §6 rounding finding). |
+| `stream.py` `classify` | the if-chain inside `_conclude` P:1771–1790 | rewrite it as one `match` over `(stopped, exit_code, ss.result, api_error_reason(...))` |
+| `launch.py` | `ENV_PINNED_LAUNCH`, `SCRUB_*` P:86–109, `scrub_class` / `_venv_prefixes` / `_under` / `scrubbed_env` P:1235–1282, `resolve_binary(_via)` P:1309–1335, `claude_argv` P:2652–2665, `canonical_json` / `_sha256_*` P:240–250 | drop `scrub_report`/`scrub_unknown` (only `where` uses them); `norm_path` becomes `os.path.abspath` |
+| `store.py` | `_retry`, `atomic_write_bytes`, `write_json_atomic`, `read_bytes_retry` P:829–866; `ledger_append` P:1046–1059 | the ledger append drops its `RegistryLock` precondition and per-job dedupe: one call writes once |
+| `cli.py` | `render_template` P:1738; `delta_header` P:2677–2697 + `git` P:1214 | used as is |
+| files | `templates/{train,spawn,rebase}.md`, `preamble.md` | remove `${commit_rule}` and `${mode}` from `spawn.md`. Reason: the default rule contradicted no-commit briefs (verdict §3), so the brief now carries commit instructions. |
+
+Do not port: `Platform`/`WindowsPlatform`/`PosixPlatform`, `RegistryLock`, `Supervisor`, `SupClock`, `reap_lost`, `kill_tree`/`process_tree`, ping and `tick`, `due_flags` and the drift checks, `wait`, `where`, `model_admission`, `--method`.
+
+## 4. CLI surface (`G = uv run --script gremlins.py`; every command prints one JSON object on stdout)
+
+- `G init --run R --model M [--effort E] [--cwd D] [--agent A] [--binary P]` writes `run.json` with the fingerprint and the rendered preamble (B1). It refuses to overwrite an existing run.
+- `G train --run R --base B --brief F` makes one blocking call with `--session-id`, then writes `bases/B.json` and prints `{status, session_id, cost, …}` (B1).
+- `G spawn --run R --base B --job J --brief F` makes one blocking fork with the spawn template plus `delta_header` and prints `{status, cache, note, cost, …}` (B2, B3, B5). Reason for the delta: a fork knows only the base's commit, and it reuses B7's function.
+- `G rebase --run R --base B --brief F` resumes the base without a fork; the prompt is the rebase template + `delta_header` + the brief. Afterwards it updates `bases/B.json` from the result. It warns when jobs of B are still marked `running` (B7).
+- `G status --run R [--job J]` shows bases with their warmth, plus the job records (B6; B3 for reading a result again).
+- There is no `kill` command. B4 was verified on Windows with 2.1.299: `TaskStop` on the backgrounded Bash call kills `claude.exe` and its bash wrappers.
+- Out of the MVP, each with its reason: `continue` (no box), `ping` (B6 says no), `usage` aggregation (no box; the ledger lines are greppable), `list`/`wait`/`expire` (the harness covers them).
+
+## 5. Build order (sequencing, not a ranking of value)
+
+1. `store` + `launch` + `stream` (ported) + `call` + `init`/`train`/`spawn`. **Gate:** a live run of `init → train → spawn` shows `cache: hit` and two ledger lines with real model, tokens and cost delta. Nothing else gets built before this gate passes.
+2. B3: two `spawn`s via `run_in_background` at once; both notices arrive, two job files, two ledger lines.
+3. B6: `status` with warmth.
+4. B4: the stop-signal path in `call.py`, then one live `TaskStop` on a backgrounded `spawn` to confirm the job ends without a ledger line and nothing beyond an in-flight command survives.
+5. B7: `rebase`.
+6. B8: run step 1's gate on one non-Windows host (WSL is enough).
+
+## 6. Tests (stdlib `unittest`; a fake `claude` = `[sys.executable, tests/fake_claude.py]` written into `run.json["argv"]`)
+
+The tests cover the happy paths and the failure paths each box names. Add no other tests without a one-line reason.
+
+- **Happy (B1, B2, B5):** `train` stores the base with the session id from `--session-id`. `spawn`'s argv starts with the stored `argv` byte for byte, then `--resume <base> --fork-session`. The ledger line has `model`, four token columns and `cost_delta = total − base cumulative`.
+- **B1 failures:** the binary is missing (error before launch, no job file); a non-zero exit or no `result` event gives `status: error` and no base file; an API error maps through `api_error_reason`.
+- **B2 failures:** unknown base gives an error with no launch; first-usage `cache_read` of 0 gives `cache: miss`, which is reported, not an error.
+- **B3:** two `spawn`s in parallel threads give two job files and two whole ledger lines.
+- **B4:** SIGTERM to a helper running a fake that sleeps leaves the child gone, the job `stopped`, and the partial raw file kept.
+- **B5:** `modelUsage` with two models gives both in `models_used`; a missing `total_cost_usd` gives `cost_delta: null`.
+- **B6:** no hit gives `cold`; 5m-only writes give `ttl_s` 300; a hit at `now − ttl + 60` gives `expiring`.
+- **B7:** the rebase argv has `--resume` and no `--fork-session`; the base record takes the result's `session_id`.
+- **B8:** the scrub drops `SCRUB_DROP` and `UV_*`, sets the pinned vars, and removes venv PATH entries.
+
+## 7. Open risks per box
+
+- **B4 (verified, with a known leftover).** `TaskStop` kills `claude.exe` and its bash wrappers. A command the gremlin had in flight (in the check, a `sleep 120` run through the gremlin's Bash) **survives as an orphan and runs to its own end**. That is accepted for the MVP. A later sweep is an open item, outside MVP scope; a full tree kill would need a Job Object, and B8 forbids that. On a hard kill, the helper's signal handler may never run, so the job file can stay `running` with no ledger line. The orchestrator knows which calls it stopped, so `status` reports the record as written. Reason the handler stays anyway: on POSIX and with Ctrl-C it does run, and it costs about 10 lines.
+- **B7.** Resuming without a fork may give the session a new id in `-p` mode. Take `session_id` from the result; never assume it stays the same. A `spawn` started during a `rebase` forks a session that is still being written. The helper only warns; keeping them apart is the orchestrator's job.
+- **B2.** A cache hit needs more than the same argv: the same env pins, the same cwd, and a system prompt that does not change (for example, the date rolls over at midnight). The helper reports `cache`; it does not prevent misses.