📝 Headless forks Phase 1: contract digest for the block owners, run log intake

docs/workpad/plans/2026-10-16-12-headless-forks-phase1-contract.md

@@ -0,0 +1,121 @@
+# Phase 1 contract digest — `spire:forks` helper core
+
+**Written by**: the Athanor, `claude-opus-5-5[1m]`, 2026-10-16 12:20 +0200, at Phase 1 intake.
+**What this is**: the brief both Phase 1 block owners build against. It does not replace the build plan; it pins the points the build plan, the plan, the design and the preflight report leave to the builder, so that blocks 1.1 and 1.2 agree. Where this file and the build plan disagree, tell the Athanor in your handoff note and follow the build plan.
+
+Sources, in order of precedence:
+
+1. Build plan `docs/workpad/plans/2026-10-16-02-headless-forks-build.md`: §4 (C1–C8, N1–N3, usage columns), §6 Phase 1 table.
+2. Preflight report `docs/workpad/reports/2026-10-16-09-headless-forks-preflight.md`: "Defaults Phase 1 must use" table (measured on this host).
+3. Coordinator decisions after Phase 0: in the run log `docs/workpad/reports/2026-10-16-08-headless-forks-run.md` (Phase 0 close, last bullet).
+4. Plan `docs/workpad/plans/2026-10-16-01-headless-forks.md`: §2 (runtime), §3 decisions 5–8, §4 (contracts), §7 Phase 1.
+5. Design `docs/workpad/designs/2026-10-16-01-headless-forks.md`: §3.2–§3.5, §4, §5 (only the threshold defaults and the warmth formula), §6, §7.
+
+Probe code from Phase 0 that you may reuse (read it, copy what helps; it is throwaway and not in git): `C:\Users\g4b\AppData\Local\Temp\claude\G--dev-projects-reliquary\42c87f82-5542-46aa-8d01-4166e3f0bcfe\scratchpad\headless-forks-p0\probes\` — `p0lib.py` (scrub, command line, stream parser, pid + creation-time liveness, job-object query, `visible_windows()`), `p01_launch.py` (rung 5 = CIM hidden), `p01_sup.py`, `p01_check.py` (tree snapshot, `taskkill`).
+
+## Files
+
+All inside `plugins/spire/skills/forks/`. Nothing else in the repo changes in Phase 1.
+
+| File | Block | Notes |
+|---|---|---|
+| `forks.py` | 1.1 skeleton, 1.2 the rest | One file, stdlib only, the exact PEP 723 header of plan §2. LF line endings. |
+| `preamble.md` | 1.1 (moved from 1.2: `init` hashes it) | Appended system prompt for every base and fork. Content below. |
+| `templates/train.md`, `templates/fork.md` | 1.2 | Prompt frames (design §2). `fork.md` carries the stale-snapshot sentence (R13). |
+| `tests/test_forks.py` | 1.1 starts, 1.2 extends | PEP 723, `unittest`, run as `uv run plugins/spire/skills/forks/tests/test_forks.py`. Imports `forks.py` by path (`importlib.util`). Stubs and temp dirs are created at test runtime under `tempfile`; nothing generated is committed. |
+
+## Command surface and exit codes (Phase 1)
+
+`where`, `init`, `train`, `fork`, `continue`, `wait`, `list`, `expire`, `kill`, internal `_supervise`. Phase 2 commands `rebase`, `usage`, `tick` exist in the dispatch and exit 2 with `{"error":"not implemented","phase":2}` until Phase 2. Every command except `where` takes `--run <run log .md>`.
+
+Exit codes: `0` ok · `2` usage error or refused · `3` `wait` condition not met at timeout · `4` `wait` condition met and at least one job in it did not end `ok` (C4).
+
+stdout: exactly one JSON object per call, newline-terminated. All diagnostics on stderr. Errors are JSON on stdout too (`{"error": "...", ...}`) with exit 2.
+
+## Paths
+
+- `--run`, `--brief`, `--cwd` accept relative, `G:\…`, `G:/…` and Git Bash `/g/…` forms (also `/c/…`). Normalise to an absolute Windows path with backslashes (on POSIX: `os.path.abspath`). Store and hash only the normalised form.
+- State dir = the run log path with `.md` removed plus `-forks`, as a sibling: `reports/2026-10-16-08-x-run.md` → `reports/2026-10-16-08-x-run-forks/`.
+- State dir contents: `registry.json`, `registry.lock`, `ledger.jsonl`, `.gitignore` (`raw/` and `*.lock`), `mcp-empty.json` (`{"mcpServers":{}}`), `jobs/<job>.json`, `briefs/<job>.md`, `notes/`, `results/<job>.json`, `logs/`, `raw/<job>.jsonl`, `raw/<job>.err`, `raw/<job>.env.json` (the child env; it is under `raw/` so it is never committed), `raw/<job>.sup.log` (supervisor's own stderr).
+- Job ids `j0001`, `j0002`, … per run.
+
+## State files (C3)
+
+- `registry.json`: `{"schema": 1, "config": {...}, "bases": {name: {...}}, "jobs": {job: {...index...}}, "run": {"last_orchestrator_call_at": iso, ...}}`. Writes only under `registry.lock` (create-exclusive; a lock older than 30 s is stale and may be broken), to a temp file then `os.replace`, retrying `PermissionError` at least 20 times, 50 ms apart. Readers retry the same way.
+- `jobs/<job>.json`: per-job runtime state — supervisor pid + creation time, claude pid + creation time, `heartbeat_at`, `turns`, `last_event_at`, `status` while running. Its only writer is that job's supervisor (except `kill`, which may write the final status if the supervisor is dead). Same atomic write with retries.
+- `ledger.jsonl`: append-only, appended under `registry.lock`.
+- Times are ISO 8601 with offset; one module function `_now()` returns epoch seconds and is the only clock (tests monkeypatch it).
+- PIDs are always stored with their creation time and compared as a pair (A19). Windows: `OpenProcess` + `GetProcessTimes`; POSIX: `/proc/<pid>/stat` start time when present, else pid only.
+
+## Launch fields (`init`; design §3.3, plan §4.3)
+
+`init --run R [--model ID] [--effort E] [--permission-mode M] [--tools CSV] [--allowed-tools ...] [--disallowed-tools ...] [--agent NAME] [--cwd DIR] [--binary PATH] [--parallel-cap N] [--orphan-after S] [--pings on|off] [--max-minutes N]` plus the thresholds below.
+
+- `model`: required full id matching `^claude-[a-z0-9-]+(\[1m\])?$`; aliases refused (exit 2).
+- `effort` default `high`; `permission_mode` default `dontAsk` (`plan` refused); `tools` default `Bash,Read,Edit,Write,Glob,Grep,Skill`; `allowed_tools` default = tools; `disallowed_tools` default `Bash(git push:*)`, `Bash(git rebase:*)`, `Bash(git reset --hard:*)`, `Bash(claude:*)`.
+- `mcp`: always `--strict-mcp-config --mcp-config <state>/mcp-empty.json`.
+- `append_system_prompt`: the skill's `preamble.md`, stored as path + sha256.
+- `agent`: optional name; stored with the sha256 of the resolved `<cwd>/.claude/agents/<name>.md` (refuse if not found).
+- `cwd`: default the git toplevel of the current directory (`git --no-optional-locks rev-parse --show-toplevel`), normalised.
+- `binary`: `--binary`, else `$CLAUDE_CODE_EXECPATH`, else `shutil.which("claude")`; stored as **absolute path** + `version` (`<binary> --version`, first token) + `size` + `mtime`. Launches always use this stored path, never PATH lookup.
+- `env.pinned`: `CLAUDE_CODE_PROMPT_CACHE_TTL=1h`, `DISABLE_AUTOUPDATER=1`.
+- Fingerprint hash: sha256 of canonical JSON (sorted keys, no spaces) of all launch fields above.
+- Config extras: `parallel_cap` 2, `orphan_after` 1200 s (N1), `pings` `on` (P0.6 passed; used in Phase 2), `ttl_s` 3600, `max_minutes` 90, `stall_after_s` 900, thresholds `rebase_commits` 6, `rebase_lines` 1500, `rebase_delta_tokens` 12000, `max_base_tokens` 250000, `max_rebases` 4.
+- Idempotent: re-running `init` with identical values succeeds and returns the config. Different values: allowed while no base exists (config replaced), refused with exit 2 once a base exists.
+- **C8 model admission** (rule measured in P0.5): read `availableModels` from `~/.claude/settings.json`, `<cwd>/.claude/settings.json`, `<cwd>/.claude/settings.local.json`. If no file has the key, admit. Otherwise admit iff some list contains the exact id, or the id without `[1m]`, or the id's family alias (`opus`, `sonnet`, `haiku`, `fable`, taken from the second dash-separated token of the id). Else exit 2 with a message naming the id and the files that hold lists, and saying the user must add the id. Never write a settings file.
+
+## `where` (G-where)
+
+Prints one JSON object: `script` (absolute path of `forks.py`), `python` (version, `executable`, `base_executable`), `uv` (version from `uv --version` if found, and `SCRIPTS_UV` if set), `binary` (resolved path, version, size, mtime, how resolved), `git` (resolved path and version), `spawn` (the constant naming the method: `cim-hidden` on Windows, `setsid` on POSIX), `scrub` (`drop`, `keep`, `pinned` lists, and `unknown`: host `CLAUDE_*`/`ANTHROPIC_*`/`CLAUDE_CODE_*` vars on neither list), `ok` (bool). Exit 0 when Python ≥ 3.10 and a binary resolves; otherwise exit 2 with `ok: false` and the reason. `where` never writes anything.
+
+## Env scrub (one constant; design §4 + plan §2 + P0.0)
+
+- Drop: the twelve Desktop vars (design §4), `CLAUDE_EFFORT`, `ANTHROPIC_MODEL`, `ANTHROPIC_DEFAULT_*_MODEL`, `ANTHROPIC_SMALL_FAST_MODEL`, `CLAUDE_CODE_SUBAGENT_MODEL`, `ENABLE_PROMPT_CACHING_1H`, `FORCE_PROMPT_CACHING_5M`, `CLAUDE_CODE_SUBAGENT_PROMPT_CACHE_TTL`, `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_USE_BEDROCK`, `CLAUDE_CODE_USE_VERTEX`, `VIRTUAL_ENV`, every `UV_*`; PATH entries under `VIRTUAL_ENV` or under `sys.prefix` when it differs from `sys.base_prefix`.
+- Keep: everything else, including `CLAUDECODE`, `CLAUDE_CODE_CHILD_SESSION`, `AI_AGENT`, `CLAUDE_PID`, `CLAUDE_CODE_GIT_BASH_PATH`, `PATH`, `HOME`/`USERPROFILE`.
+- Pinned on top: `CLAUDE_CODE_PROMPT_CACHE_TTL=1h`, `DISABLE_AUTOUPDATER=1`, `SPIRE_FORKS_CHILD=1` (N3), and `CLAUDE_CODE_EXECPATH=<the stored binary path>` (coordinator: so the 2.1.283 slip cannot recur).
+- The env is computed by the launching command (which has the real host env) and written to `raw/<job>.env.json`. The supervisor (which gets the registry default env from CIM) reads that file and passes exactly that env to `claude`.
+
+## Spawning (C6, P0.1, coordinator)
+
+- Windows, default and only Phase 1 method `cim-hidden`: the launching command runs `powershell -NoProfile -NonInteractive -Command` with `$si = New-CimInstance -ClassName Win32_ProcessStartup -ClientOnly -Property @{ShowWindow=[uint16]0}; $r = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{CommandLine=$env:SPIRE_FORKS_CMD; CurrentDirectory=$env:SPIRE_FORKS_CWD; ProcessStartupInformation=$si}; "{0} {1}" -f $r.ReturnValue, $r.ProcessId` (command line and cwd passed through env vars, never interpolated). The `powershell` call itself is started with `CREATE_NO_WINDOW`. Non-zero `ReturnValue` → the job ends `failed`, reason `spawn_failed`.
+- Supervisor command line: `<interp> -I <abs forks.py> _supervise --run <run> --job <job>`, `<interp>` = `sys._base_executable` if it differs from `sys.executable`, else `sys.executable`. Under `-I` the script dir is not on `sys.path`; do not rely on it.
+- The supervisor redirects its own stdout/stderr to `raw/<job>.sup.log` at start (it has no inherited handles).
+- Every process the supervisor starts (`claude`) gets `CREATE_NO_WINDOW` (`0x08000000`). Never `DETACHED_PROCESS` or `CREATE_NEW_CONSOLE`.
+- POSIX: `subprocess.Popen(..., start_new_session=True)` from the launching command, stdio to files; kill by process group. Implement, keep simple; it is not tested on this host.
+- **No visible window** is an acceptance criterion: on Windows, the tests assert that no visible top-level window belongs to the supervisor or any descendant while a stubbed job runs (`EnumWindows` + `IsWindowVisible` + `GetWindowThreadProcessId`, as in `p0lib.visible_windows()`).
+
+## `claude` command line (fingerprint every time; flags are not inherited by forks)
+
+`<binary> -p --output-format stream-json --verbose --model M --effort E --permission-mode P --tools T --allowedTools A --disallowedTools D --strict-mcp-config --mcp-config <state>/mcp-empty.json --append-system-prompt-file <preamble> [--agent N]` plus per kind: `train` `--session-id <new uuid>`; `fork` `--resume <base session> --fork-session --session-id <new uuid>` (P0.7: honoured); `continue` `--resume <fork session>` (no fork flag); `ping` (Phase 2). Prompt on stdin from `briefs/<job>.md`. cwd = the base's cwd. List flags are passed comma-joined, as in the probes.
+
+## Supervision and results (1.2)
+
+- Stream parse: `system/init` (session id, runtime fingerprint: sorted tool names hash, `permissionMode`, plugin names + versions, agents, `apiKeySource`); each `assistant` event bumps `turns`/`last_event_at` (count messages by `message.id`); the first assistant event's `usage` gives the cache verdict; `system/informational` with `level: warning` whose content contains `is restricted` → model refused; `result` gives the totals.
+- Usage rule (P0.3): the four columns and the 1h/5m split from `result.usage`; `models_used` = keys of `result.modelUsage` whose counts grew against the session total before the call; cost Δ = `total_cost_usd` − the session total before the call (base cumulative for a fork, the fork's last total for `continue`, 0 for `train`); `usage_mismatch` flag when `result.usage` ≠ the `modelUsage` delta. Never sum stream events for output tokens. `turns` = `result.num_turns`.
+- Base context tokens = `input + cache_creation + cache_read` of the base's last assistant message (last stream event for that message id). Stored on the base at train end, with `base_commit` (`git --no-optional-locks rev-parse HEAD` in the cwd), training cost, cumulative `modelUsage`, `last_hit`.
+- Cache verdict (C5): first assistant event `cache_read` vs the base's context tokens: `hit` ≥ 0.9, `partial` > 0, `miss` = 0. `train` jobs get `cache: null`.
+- Warmth in Phase 1 (C1): `last_hit` = time of the first assistant event of the most recent call on this base or any of its forks; `warm_until = last_hit + ttl_s`; `margin = max(120, ttl_s/10)`; `base_state` warm / expiring / cold. `rebase_due` and `retrain_due` are `false`, `reasons: []`.
+- `cold_start` flag when the base was `cold` at launch, or the verdict is not `hit` and the fingerprint is `ok`.
+- Model refused: the warning event, or `models_used` not containing the requested id (compare including `[1m]`; P0.5: the `modelUsage` key carries the suffix) → status `failed`, reason `model_refused`.
+- Watchdogs: wall time > `max_minutes` → kill tree, `timeout`; no stream output for `stall_after_s` → flag `stalled` (not killed); non-empty `permission_denials` → flag `needs_attention`; `is_error` or non-zero exit → `failed` with a reason.
+- Heartbeat: the supervisor writes `heartbeat_at` at least every 30 s. `wait` and `list` mark a running job `lost` when its heartbeat is older than 120 s **and** its supervisor pid + creation time is dead; a ledger line with whatever was captured.
+- N1 orphan watchdog: `wait`, `fork`, `continue`, `train`, `list` (and Phase 2's `rebase`) update `run.last_orchestrator_call_at`. A supervisor whose job is running kills its `claude` tree when that is older than `orphan_after`; status `killed`, flag `orphaned`, ledger line.
+- N3: launching commands (`train`, `fork`, `continue`, and Phase 2's `rebase`/`tick` pings) exit 2 when `SPIRE_FORKS_CHILD` is set. `where`, `list`, `wait` (and `usage`) stay allowed.
+- N2: `fork --mode parallel` requires `--paths p…`; stored on the job and echoed as `paths` in the result. Sequential forks may pass it.
+- Parallel cap: at most `parallel_cap` running jobs of kind `fork`/`continue`; a launch over the cap exits 2.
+- Refusals (exit 2): fork from a base that is not `active` without `--force`; `continue` of a running job; `continue` of a non-fork job; any launch-field override on `fork`/`continue`; missing binary (stored path gone).
+- Binary re-check at every launch (version, size, mtime against the base record): a difference is reported as fingerprint drift `binary` in the result (Phase 2 turns it into `rebase_due`).
+- Fork delta header (prepended by the helper to the brief, inside `templates/fork.md`): `git --no-optional-locks log --oneline <base_commit>..HEAD` and `git --no-optional-locks diff --shortstat <base_commit>`.
+- Result object: exactly plan §4.2's fields, plus `paths` (N2), `reason` (null unless failed/killed/lost), `model_requested`. Written to `results/<job>.json`; the ledger line = design §6's shape + `status`, `flags`, `reason`.
+- `fork` launch return (C1): `{job, base, base_state, warm_until, rebase_due, retrain_due, reasons}`. `train` returns `{job, base, session_id}`; `continue` returns `{job}`.
+- `wait [job…] | --all [--any] [--timeout S]` (default 540, max 570): polls about every 2 s; returns `{done: [results not returned before], running: [{job, block, elapsed_s, last_event_s_ago, turns}], pings: []}`; exit per C4; marks returned results as delivered so each result is returned by at least one `wait`. `list` always shows every job and base.
+- `kill --job J` (and `expire` of a base with running jobs): snapshot the supervisor's whole process tree first (pids + creation times; use a Toolhelp32 snapshot via ctypes rather than PowerShell for speed), `taskkill /T /F /PID <supervisor>`, then kill any recorded pid still alive (sweep), verify none of the snapshot survives, set `killed`. A dead supervisor is handled too (kill the claude tree from the job file).
+- `expire --base B | --all`: mark `retired`; running jobs of that base are killed with the sweep (coordinator), flag `expired`.
+
+## `preamble.md` (content owned by 1.1; short)
+
+A headless implementer trained once and then forked per block; it may be resumed or forked later. Do the work yourself; never dispatch agents and never start `claude` or the forks helper. Never use `run_in_background` on Bash (coordinator decision): run every command in the foreground. Stage and commit by explicit paths only, and only when the brief says this fork commits. Write the handoff note to the path the brief names, in the run-log handoff-note format (`plugins/spire/skills/implementation/references/run-log.md`), stamping the exact model id. Gate and test output go to files under the state dir's `logs/` (`.txt`).
+
+## Not in Phase 1
+
+Warmth inference and pings, `rebase`, `rebase_due`/`retrain_due` reasons, `usage` rendering, `tick` (Phase 2). `SKILL.md`, athanor and implementation text (Phase 3). Version bump (Phase 3).