🤖 forks: launch, supervise, wait, kill (block 1.2)
This commit is contained in:
@@ -96,3 +96,41 @@ Gate after 1.1 (Athanor, 12:29): G-unit `Ran 52 tests … OK`, exit 0; G-where e
|
|||||||
- Left for later: everything the digest assigns to 1.2 (templates, launch, supervise, results, ledger lines, `wait`/`list`/`expire`/`kill`, no-visible-window test, `run.last_orchestrator_call_at`); Phase 2 commands.
|
- Left for later: everything the digest assigns to 1.2 (templates, launch, supervise, results, ledger lines, `wait`/`list`/`expire`/`kill`, no-visible-window test, `run.last_orchestrator_call_at`); Phase 2 commands.
|
||||||
- Expected broken: `train`, `fork`, `continue`, `wait`, `list`, `expire`, `kill`, `_supervise` exit 2 "not implemented" (block 1.2); `rebase`, `usage`, `tick` (Phase 2).
|
- Expected broken: `train`, `fork`, `continue`, `wait`, `list`, `expire`, `kill`, `_supervise` exit 2 "not implemented" (block 1.2); `rebase`, `usage`, `tick` (Phase 2).
|
||||||
- Commits: `d36ac2f` (code, tests, preamble, this note), then a one-line follow-up filling in this hash.
|
- Commits: `d36ac2f` (code, tests, preamble, this note), then a one-line follow-up filling in this hash.
|
||||||
|
|
||||||
|
### Block 1.2 — Launch and supervise
|
||||||
|
|
||||||
|
- Model: `claude-opus-5-5[1m]`
|
||||||
|
- Start / end: Fri Oct 16 12:31:01 WEDT 2026 / Fri Oct 16 15:48:45 WEDT 2026 (`date`; the host clock read 12:31 at my first call and 15:42 at the first manual run, so the span includes a jump like the one Phase 0 saw; the gates ran 15:47:38–15:48:30)
|
||||||
|
- Done: `forks.py` gains `train`, `fork`, `continue`, `wait`, `list`, `expire`, `kill`, `_supervise`. Launch writes the job (registry index, status `created`) under the lock, `briefs/<job>.md` (template + brief) and `raw/<job>.env.json` (scrubbed env with `CLAUDE_CODE_EXECPATH` = stored binary and `SPIRE_FORKS_CHILD=1`), then spawns `<base interp> -I forks.py _supervise --run --job` through `powershell` (`CREATE_NO_WINDOW`) → CIM `Win32_Process.Create` with `ShowWindow=0`, command line and cwd passed in `SPIRE_FORKS_CMD`/`SPIRE_FORKS_CWD`; `ReturnValue` ≠ 0 ends the job `failed`/`spawn_failed`. The supervisor redirects its stdio to `raw/<job>.sup.log`, starts `claude` with exactly the env file and `CREATE_NO_WINDOW` (stored absolute binary path, full fingerprinted command line every call, `--session-id` on train and fork, `--resume … --fork-session` on fork, `--resume` only on continue, `--max-budget-usd` passed through), feeds the prompt on stdin, parses the stream (`StreamState`: init, message ids, first/last assistant usage, `system/informational` restriction warning, result), and is the only writer of `jobs/<job>.json` (heartbeat every 10 s). Watchdogs: `max_minutes` → kill tree, `timeout`; `stall_after_s` of no stdout → flag `stalled`; N1 orphan check every 1 s → kill tree, `killed` + `orphaned`; restriction warning → kill tree, `failed`/`model_refused`. Finalising (`finalize_job`, once, under the lock): `results/<job>.json` (plan §4.2 fields + `paths`, `reason`, `model_requested`), ledger line (design §6 + `status`, `flags`, `reason`), registry update (status, cumulative cost and `modelUsage` for the session, base `last_hit`; on train: base `active` with `base_commit`, context tokens, training cost, cumulative `modelUsage`, runtime fingerprint). Usage rule, `usage_mismatch`, cache verdict, `cold_start`, C1 warmth, fingerprint `ok|drift:<fields>` (launch: `binary`, `append_system_prompt`, `agent`; runtime: `tools`, `permissionMode`, `plugins`, `agents`, `apiKeySource` against the base's init). `wait` per C4, `list` (all bases with warmth, all jobs, running info), `lost` reaper (heartbeat > 120 s and supervisor pid + ctime dead), `kill` (Toolhelp32 tree snapshot with ctimes → `taskkill /T /F` → sweep with `TerminateProcess` by pid + ctime → verify up to 10 s), `expire` (retire, kill running jobs of the base, flag `expired`). Refusals: N2, N3, cap, launch-field overrides on fork/continue (each field accepted by the parser only to be refused), base not active without `--force`, continue of a running or non-fork job, missing binary (`reason: binary_missing`), base name taken. New `templates/train.md` and `templates/fork.md` (stale-snapshot sentence, delta header from `git --no-optional-locks log --oneline <base_commit>..HEAD` and `diff --shortstat <base_commit>`, commit rule per mode, note path, logs dir). `preamble.md` unchanged. Tests: 17 new (69 total): a generated stream stub `claude` (`stub/claude.cmd` → base interpreter `-I stub.py`, behaviour per call from `@stub …` lines in the brief, sessions kept beside it so resume/fork report cumulative `modelUsage` and `total_cost_usd`), real CIM launches with the stub as `--binary`, `RuleTests` on the real P0.2 seq-1 numbers, and a module-end check that no process whose command line contains the tests' temp prefix survives.
|
||||||
|
- Gates:
|
||||||
|
- G-unit `uv run plugins/spire/skills/forks/tests/test_forks.py` → exit 0, `Ran 69 tests in 52.001s` / `OK` (suite wall 52 s; two earlier full runs at 68 tests: 51.7 s each, OK).
|
||||||
|
- G-where `uv run plugins/spire/skills/forks/forks.py where` → exit 0, stdout one line, a JSON object, `ok: true`, `spawn: cim-hidden`, binary 2.1.299.
|
||||||
|
- Leftover processes after the gates: none. Checked by command line (`forks-test-`, `_supervise`, my manual scratch run `b12-manual`) via `Get-CimInstance Win32_Process`, not by name; also asserted by the suite's `tearDownModule`.
|
||||||
|
- Manual check (scratch run in the session scratchpad, stub binary): the supervisor's parent is `WmiPrvSE.exe`; its tree is python → conhost, cmd (claude) → conhost, python stub → python grandchild; `kill` snapshot 6 pids, `taskkill_rc` 0, no survivors.
|
||||||
|
- Deviations from plan:
|
||||||
|
- `fork --max-minutes` and `continue --max-minutes` take a positive float (the tests use 0.06 min); `init --max-minutes` stays an integer.
|
||||||
|
- `fork`/`continue` also run the `lost` reaper before the cap check (the digest names `wait` and `list`), so a dead supervisor cannot block the cap.
|
||||||
|
- The `lost` reaper kills the job's leftover `claude` tree (from the job file) before marking it `lost`; the digest only says to mark it.
|
||||||
|
- On the restriction warning the supervisor kills `claude` at once instead of letting the substituted model finish (saves the spend; the result still reads `failed`/`model_refused`).
|
||||||
|
- After `claude` exits, the supervisor sweeps any descendants of it that are still alive (the "backgrounded `sleep 240`" trap); logged in `sup.log`.
|
||||||
|
- `train` takes no launch-field overrides (the design's table allows them; the plan's invariant and the digest do not); they are usage errors there.
|
||||||
|
- Flagged for review (semantics the digest left open):
|
||||||
|
- `wait --all` = every job not yet *settled*; a job is settled when it was in the condition of a `wait` that returned 0 or 4. `done` holds only results not delivered before. So the mixed case gives 3 (failed result in `done`), then 4 when the rest finish (the failed job stays in the condition without being returned again), then 0 on an empty set. Explicit job lists use the same delivery marks. `--any`: met when any member has finished (or the set is empty); exit 4 if any finished member is not `ok`. `wait` with neither jobs nor `--all` is a usage error; `--timeout` > 570 is refused (exit 2).
|
||||||
|
- `reason` vocabulary: `model_refused`, `is_error`, `exit_code`, `no_result`, `spawn_failed`, `launch_error`, `supervisor_error` (failed); `kill`, `expire`, `orphaned` (killed); `supervisor_dead` (lost). `timeout` has `reason: null` (the digest: null unless failed/killed/lost). Details go to the registry's `reason_detail` (shown by `list`), not into the result or ledger.
|
||||||
|
- Precedence when deciding a finished call: model refused > no result (`exit_code` if non-zero, else `no_result`) > `is_error` > non-zero exit. A model is "refused" when the warning appeared or the requested id (with `[1m]`) is not among the `modelUsage` keys that grew; extra models do not count.
|
||||||
|
- Without a `result` event (killed, lost, died): `tokens` are all `null`, `cost_delta_usd` `null`, `models_used` `[]`, `turns` = distinct assistant message ids seen; cache verdict still from the first assistant event if one arrived.
|
||||||
|
- `last_hit` for a train is the time of its **last** assistant event (the full prefix is written there); for fork/continue the first, written to the base as soon as it arrives. A continue refreshes its base's `last_hit`.
|
||||||
|
- `base_state` is `null` for train jobs; `cold_start` is only computed for fork/continue (a train is always cold).
|
||||||
|
- `continue` sends the brief as given (no template; there is no `templates/continue.md`). Its `parent_session` is the fork's parent (the base session); its cost and `modelUsage` "before" are the fork session's last cumulative totals (updated by every continue), falling back to the base's when the fork has none (a killed fork: the delta then includes the killed fork's spend).
|
||||||
|
- `note` = `notes/<block>.md` (block sanitised) if that file exists when the job ends, else `null`; `fork.md` tells the fork that path. Train jobs have `note: null`.
|
||||||
|
- A failed training leaves the base `failed`; `train` may reuse a name only when that base is `failed`.
|
||||||
|
- `kill` of a job that is no longer running exits 0 with `killed: false` and its status; survivors after the 10 s sweep give `ok: false` (exit still 0).
|
||||||
|
- A `kill` that lands while the supervisor holds the registry lock to finalise leaves a stale lock (broken after 30 s) and, in a narrow window, a duplicate ledger line. `kill` marks `kill_requested` first; a supervisor that finalises after that records `killed` unless the call ended `ok`.
|
||||||
|
- `wait` updates `last_orchestrator_call_at` at its start and end only (a 570 s wait stays under the 1200 s `orphan_after`).
|
||||||
|
- Stall is measured on any stdout line, not only parsed JSON events.
|
||||||
|
- `process_tree` follows parent pids through dead processes (so orphans of a dead `claude` are found) but stops at a node whose pid now belongs to a process with another creation time; this reuse guard is not exercised by a test.
|
||||||
|
- `spawn_failed` shows up in `wait`, not in the launch return (the launch return is C1's fixed shape).
|
||||||
|
- Not tested: the POSIX path (`setsid` spawn, process-group kill, `/proc` tree) — written, never run; `--agent` drift (only the code path, no test); a real `claude` (no test spends tokens; Phase 1.3 smoke); supervisor survival across the end of the orchestrator's turn (1.3).
|
||||||
|
- Left for later: warmth inference, pings, `rebase`, `rebase_due`/`retrain_due` reasons, `usage`, `tick` (Phase 2); `SKILL.md` (Phase 3).
|
||||||
|
- Expected broken: `rebase`, `usage`, `tick` exit 2 "not implemented" (Phase 2).
|
||||||
|
- Commits: see the follow-up line below.
|
||||||
|
|||||||
+1294
-15
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
|||||||
|
# Block ${block} (spire:forks)
|
||||||
|
|
||||||
|
You are a fork of base `${base}` (run log `${run}`), now working on block `${block}` (phase ${phase}, mode ${mode}).
|
||||||
|
|
||||||
|
Your system prompt was snapshotted when the base was trained: the git status, the CLAUDE.md files and the environment it shows are **stale**. The delta header below lists what changed in the repository since then. Trust it and the files on disk over the snapshot and over what you remember from training; re-read every file before you change it.
|
||||||
|
|
||||||
|
## Delta since training
|
||||||
|
|
||||||
|
${delta}
|
||||||
|
|
||||||
|
## Rules for this fork
|
||||||
|
|
||||||
|
- ${commit_rule}
|
||||||
|
- Write your handoff note to `${note_path}`.
|
||||||
|
- Write gate and test output to `.txt` files under `${logs_dir}`.
|
||||||
|
|
||||||
|
## Brief
|
||||||
|
|
||||||
|
${brief}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
# Training (spire:forks)
|
||||||
|
|
||||||
|
You are base `${base}` of the build whose run log is `${run}`. This session is trained once and then forked, once per block; every fork starts from what you know at the end of this session. Build an accurate picture of the code the brief below points you at: read it, trace the seams, note the invariants and the traps.
|
||||||
|
|
||||||
|
- Do not change code in this session. If the brief asks for trunk notes, write them to the path it names; that is the only file you write.
|
||||||
|
- The helper's state dir is `${state_dir}`. Forks write handoff notes under `${notes_dir}` and gate and test output under `${logs_dir}`.
|
||||||
|
- Finish with one line `READY`, then a short summary of what you read and anything the brief asked for that you could not do.
|
||||||
|
|
||||||
|
## Brief
|
||||||
|
|
||||||
|
${brief}
|
||||||
@@ -551,29 +551,34 @@ class InitTests(Base):
|
|||||||
|
|
||||||
# --------------------------------------------------------------------------------------
|
# --------------------------------------------------------------------------------------
|
||||||
class CliTests(Base):
|
class CliTests(Base):
|
||||||
def test_not_implemented(self):
|
def test_phase2_not_implemented(self):
|
||||||
run_log = os.path.join(self.tmp, "r-run.md")
|
run_log = os.path.join(self.tmp, "r-run.md")
|
||||||
cases = {
|
for argv in (("rebase",), ("usage", "--phase", "1"), ("tick",)):
|
||||||
("train", "--base", "B1", "--brief", "b.md"): {"block": "1.2"},
|
|
||||||
("fork", "--base", "B1", "--block", "1.1", "--brief", "b.md"): {"block": "1.2"},
|
|
||||||
("continue", "--job", "j0001", "--brief", "b.md"): {"block": "1.2"},
|
|
||||||
("wait", "--all"): {"block": "1.2"},
|
|
||||||
("list",): {"block": "1.2"},
|
|
||||||
("expire", "--all"): {"block": "1.2"},
|
|
||||||
("kill", "--job", "j0001"): {"block": "1.2"},
|
|
||||||
("_supervise", "--job", "j0001"): {"block": "1.2"},
|
|
||||||
("rebase",): {"phase": 2},
|
|
||||||
("usage", "--phase", "1"): {"phase": 2},
|
|
||||||
("tick",): {"phase": 2},
|
|
||||||
}
|
|
||||||
for argv, want in cases.items():
|
|
||||||
with self.subTest(cmd=argv[0]):
|
with self.subTest(cmd=argv[0]):
|
||||||
code, out, raw = call([argv[0], "--run", run_log, *argv[1:]])
|
code, out, raw = call([argv[0], "--run", run_log, *argv[1:]])
|
||||||
self.assertEqual(code, 2)
|
self.assertEqual(code, 2)
|
||||||
self.assert_one_json_line(raw)
|
self.assert_one_json_line(raw)
|
||||||
self.assertEqual(out["error"], "not implemented")
|
self.assertEqual(out["error"], "not implemented")
|
||||||
for k, v in want.items():
|
self.assertEqual(out["phase"], 2)
|
||||||
self.assertEqual(out[k], v)
|
self.assertFalse(os.path.exists(run_log[:-3] + "-forks"))
|
||||||
|
|
||||||
|
def test_commands_need_init(self):
|
||||||
|
run_log = os.path.join(self.tmp, "r-run.md")
|
||||||
|
brief = os.path.join(self.tmp, "b.md")
|
||||||
|
with open(brief, "w") as f:
|
||||||
|
f.write("x\n")
|
||||||
|
cases = (("train", "--base", "B1", "--brief", brief),
|
||||||
|
("fork", "--base", "B1", "--block", "1.1", "--brief", brief),
|
||||||
|
("continue", "--job", "j0001", "--brief", brief),
|
||||||
|
("wait", "--all"), ("list",), ("expire", "--all"), ("kill", "--job", "j0001"),
|
||||||
|
("_supervise", "--job", "j0001"))
|
||||||
|
for argv in cases:
|
||||||
|
with self.subTest(cmd=argv[0]):
|
||||||
|
code, out, raw = call([argv[0], "--run", run_log, *argv[1:]])
|
||||||
|
self.assertEqual(code, 2, out)
|
||||||
|
self.assert_one_json_line(raw)
|
||||||
|
self.assertIn("error", out)
|
||||||
|
self.assertNotEqual(out["error"], "not implemented")
|
||||||
self.assertFalse(os.path.exists(run_log[:-3] + "-forks"))
|
self.assertFalse(os.path.exists(run_log[:-3] + "-forks"))
|
||||||
|
|
||||||
def test_unknown_command_and_missing_run(self):
|
def test_unknown_command_and_missing_run(self):
|
||||||
@@ -602,7 +607,7 @@ class CliTests(Base):
|
|||||||
self.assertEqual(code, 2)
|
self.assertEqual(code, 2)
|
||||||
self.assertIn("inside a fork", out["error"])
|
self.assertIn("inside a fork", out["error"])
|
||||||
code, out, _ = call(["list", "--run", run_log])
|
code, out, _ = call(["list", "--run", run_log])
|
||||||
self.assertEqual(out["error"], "not implemented") # not refused as a launch
|
self.assertNotIn("inside a fork", out["error"]) # not refused as a launch (no init yet)
|
||||||
stub = make_stub_binary(self.tmp)
|
stub = make_stub_binary(self.tmp)
|
||||||
self.assertEqual(call(["where", "--binary", stub])[0], 0)
|
self.assertEqual(call(["where", "--binary", stub])[0], 0)
|
||||||
|
|
||||||
@@ -669,5 +674,914 @@ class SubprocessTests(Base):
|
|||||||
self.assertEqual(self.assert_one_json_line(r.stdout.decode("utf-8"))["error"], "usage")
|
self.assertEqual(self.assert_one_json_line(r.stdout.decode("utf-8"))["error"], "usage")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------------------
|
||||||
|
# Launch and supervise (block 1.2): a stream-json stub `claude`, the real detached spawn.
|
||||||
|
# --------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
TEST_MODEL = "claude-opus-5-5[1m]"
|
||||||
|
PRICE = {"in": 5e-6, "out": 25e-6, "cw": 10e-6, "cr": 0.5e-6}
|
||||||
|
|
||||||
|
# The stub `claude`. Behaviour is chosen per call by `@stub key[=value] ...` lines in the prompt
|
||||||
|
# (the brief reaches the stub on stdin). Sessions are kept next to the stub so that resume and
|
||||||
|
# fork see the cumulative modelUsage / total_cost_usd of the session, like the real CLI.
|
||||||
|
STUB_PY = r'''
|
||||||
|
import json, os, subprocess, sys, time, uuid
|
||||||
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||||
|
SESS = os.path.join(HERE, "sessions")
|
||||||
|
CALLS = os.path.join(HERE, "calls")
|
||||||
|
VERSION = "%(version)s"
|
||||||
|
PRICE = %(price)r
|
||||||
|
ENV_KEYS = ("CLAUDE_CODE_EXECPATH", "SPIRE_FORKS_CHILD", "CLAUDE_CODE_PROMPT_CACHE_TTL",
|
||||||
|
"DISABLE_AUTOUPDATER", "VIRTUAL_ENV", "CLAUDE_EFFORT")
|
||||||
|
|
||||||
|
|
||||||
|
def emit(o):
|
||||||
|
sys.stdout.write(json.dumps(o) + "\n")
|
||||||
|
sys.stdout.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def opt(a, name):
|
||||||
|
if name in a and a.index(name) + 1 < len(a):
|
||||||
|
return a[a.index(name) + 1]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def load(sid):
|
||||||
|
try:
|
||||||
|
with open(os.path.join(SESS, sid + ".json")) as f:
|
||||||
|
return json.load(f)
|
||||||
|
except OSError:
|
||||||
|
return {"modelUsage": {}, "cost": 0.0, "context": 0}
|
||||||
|
|
||||||
|
|
||||||
|
def save(path, obj):
|
||||||
|
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||||
|
tmp = path + ".%%d.tmp" %% os.getpid()
|
||||||
|
with open(tmp, "w") as f:
|
||||||
|
json.dump(obj, f)
|
||||||
|
os.replace(tmp, path)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
a = sys.argv[1:]
|
||||||
|
if "--version" in a:
|
||||||
|
print(VERSION + " (Claude Code stub)")
|
||||||
|
return 0
|
||||||
|
prompt = sys.stdin.read()
|
||||||
|
d = {}
|
||||||
|
for line in prompt.splitlines():
|
||||||
|
s = line.strip()
|
||||||
|
if s.startswith("@stub "):
|
||||||
|
for tok in s[6:].split():
|
||||||
|
k, _, v = tok.partition("=")
|
||||||
|
d[k] = v if v else "1"
|
||||||
|
model = opt(a, "--model") or "claude-stub"
|
||||||
|
resume, fork, sid = opt(a, "--resume"), "--fork-session" in a, opt(a, "--session-id")
|
||||||
|
session = resume if (resume and not fork) else (sid or str(uuid.uuid4()))
|
||||||
|
prior = load(resume) if resume else {"modelUsage": {}, "cost": 0.0, "context": 0}
|
||||||
|
rec = {"argv": a, "cwd": os.getcwd(), "pid": os.getpid(), "session": session, "directives": d,
|
||||||
|
"env": {k: os.environ.get(k) for k in ENV_KEYS}, "prompt": prompt}
|
||||||
|
if "child" in d:
|
||||||
|
c = subprocess.Popen([sys.executable, "-I", "-c", "import time; time.sleep(600)"],
|
||||||
|
stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||||
|
rec["child_pid"] = c.pid
|
||||||
|
save(os.path.join(CALLS, "%%d.json" %% os.getpid()), rec)
|
||||||
|
tools = [t for t in (opt(a, "--tools") or "").split(",") if t]
|
||||||
|
emit({"type": "system", "subtype": "init", "cwd": os.getcwd(), "session_id": session, "model": model,
|
||||||
|
"permissionMode": opt(a, "--permission-mode"), "tools": tools, "mcp_servers": [],
|
||||||
|
"apiKeySource": d.get("apikey", "none"), "claude_code_version": VERSION,
|
||||||
|
"plugins": [{"name": "spire", "path": "x", "source": "spire@reliquary",
|
||||||
|
"version": d.get("plugin", "0.2.2")}],
|
||||||
|
"agents": ["claude", "general-purpose"]})
|
||||||
|
used = model
|
||||||
|
if "restricted" in d:
|
||||||
|
used = "claude-opus-5-5" if d["restricted"] == "1" else d["restricted"]
|
||||||
|
emit({"type": "system", "subtype": "informational", "level": "warning", "session_id": session,
|
||||||
|
"content": "Model \"%%s\" is restricted by your organization's settings. Using %%s instead."
|
||||||
|
%% (model, used)})
|
||||||
|
if "model" in d:
|
||||||
|
used = d["model"] # silent substitution
|
||||||
|
if "silent" in d:
|
||||||
|
time.sleep(float(d["silent"]))
|
||||||
|
turns = int(d.get("turns", "3"))
|
||||||
|
ctx = int(prior.get("context") or 0)
|
||||||
|
tot = {"in": 0, "out": 0, "cw": 0, "cr": 0}
|
||||||
|
prev = ctx
|
||||||
|
for i in range(turns):
|
||||||
|
if i == 0:
|
||||||
|
mode = d.get("cache", "hit" if resume else "cold")
|
||||||
|
cr = {"hit": ctx, "partial": ctx // 2, "miss": 0, "cold": 0}[mode]
|
||||||
|
cw = (ctx - cr) + 1000
|
||||||
|
else:
|
||||||
|
cr, cw = prev, 300
|
||||||
|
inp, out = 2, 40
|
||||||
|
u = {"input_tokens": inp, "cache_creation_input_tokens": cw, "cache_read_input_tokens": cr,
|
||||||
|
"cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": cw},
|
||||||
|
"output_tokens": 1, "service_tier": "standard"}
|
||||||
|
mid = "msg_stub_" + uuid.uuid4().hex[:12]
|
||||||
|
for block in ("thinking", "text"): # two events per message id, interim output_tokens
|
||||||
|
emit({"type": "assistant", "session_id": session,
|
||||||
|
"message": {"id": mid, "model": model.replace("[1m]", ""), "role": "assistant",
|
||||||
|
"content": [{"type": block}], "usage": u}})
|
||||||
|
emit({"type": "rate_limit_event", "rate_limit_info": {"status": "allowed"}})
|
||||||
|
tot["in"] += inp; tot["out"] += out; tot["cw"] += cw; tot["cr"] += cr
|
||||||
|
prev = inp + cw + cr
|
||||||
|
if i == 0 and "hang" in d:
|
||||||
|
time.sleep(600 if d["hang"] == "1" else float(d["hang"]))
|
||||||
|
if i == 0 and "die" in d:
|
||||||
|
os._exit(int(d.get("code", "3")))
|
||||||
|
if "sleep" in d:
|
||||||
|
time.sleep(float(d["sleep"]))
|
||||||
|
cost = (tot["in"] * PRICE["in"] + tot["out"] * PRICE["out"] + tot["cw"] * PRICE["cw"]
|
||||||
|
+ tot["cr"] * PRICE["cr"])
|
||||||
|
mu = json.loads(json.dumps(prior.get("modelUsage") or {}))
|
||||||
|
m = mu.setdefault(used, {"inputTokens": 0, "outputTokens": 0, "cacheReadInputTokens": 0,
|
||||||
|
"cacheCreationInputTokens": 0, "webSearchRequests": 0, "costUSD": 0.0,
|
||||||
|
"contextWindow": 1000000})
|
||||||
|
m["inputTokens"] += tot["in"]; m["outputTokens"] += tot["out"]
|
||||||
|
m["cacheReadInputTokens"] += tot["cr"]; m["cacheCreationInputTokens"] += tot["cw"]
|
||||||
|
m["costUSD"] += cost
|
||||||
|
total = float(prior.get("cost") or 0.0) + cost
|
||||||
|
usage = {"input_tokens": tot["in"], "cache_creation_input_tokens": tot["cw"],
|
||||||
|
"cache_read_input_tokens": tot["cr"],
|
||||||
|
"output_tokens": tot["out"] + (7 if "mismatch" in d else 0),
|
||||||
|
"cache_creation": {"ephemeral_1h_input_tokens": tot["cw"], "ephemeral_5m_input_tokens": 0}}
|
||||||
|
denials = ([{"tool_name": "Bash", "tool_use_id": "toolu_stub", "tool_input": {"command": "rm -rf build"}}]
|
||||||
|
if "deny" in d else [])
|
||||||
|
is_error = "error" in d
|
||||||
|
save(os.path.join(SESS, session + ".json"), {"modelUsage": mu, "cost": total, "context": prev})
|
||||||
|
emit({"type": "result", "subtype": "error_during_execution" if is_error else "success",
|
||||||
|
"is_error": is_error, "num_turns": turns, "total_cost_usd": total, "usage": usage,
|
||||||
|
"modelUsage": mu, "permission_denials": denials, "session_id": session, "result": "READY"})
|
||||||
|
return 1 if is_error else int(d.get("exit", "0"))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
|
'''
|
||||||
|
|
||||||
|
|
||||||
|
def stub_interpreter() -> str:
|
||||||
|
base = getattr(sys, "_base_executable", None)
|
||||||
|
return base if base and os.path.isfile(base) else sys.executable
|
||||||
|
|
||||||
|
|
||||||
|
def make_stream_stub(d: str) -> str:
|
||||||
|
"""Generate the stream stub under `d` (never committed). Windows: a `.cmd` wrapper that runs
|
||||||
|
the base interpreter with -I; POSIX: a sh wrapper."""
|
||||||
|
sd = os.path.join(d, "stub")
|
||||||
|
os.makedirs(sd, exist_ok=True)
|
||||||
|
py = os.path.join(sd, "stub.py")
|
||||||
|
with open(py, "w", newline="\n") as f:
|
||||||
|
f.write(STUB_PY % {"version": STUB_VERSION, "price": PRICE})
|
||||||
|
if IS_WINDOWS:
|
||||||
|
p = os.path.join(sd, "claude.cmd")
|
||||||
|
with open(p, "w", newline="\r\n") as f:
|
||||||
|
f.write('@echo off\n"%s" -I "%s" %%*\n' % (stub_interpreter(), py))
|
||||||
|
else:
|
||||||
|
p = os.path.join(sd, "claude")
|
||||||
|
with open(p, "w", newline="\n") as f:
|
||||||
|
f.write('#!/bin/sh\nexec "%s" -I "%s" "$@"\n' % (stub_interpreter(), py))
|
||||||
|
os.chmod(p, 0o755)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def stub_calls(stub: str) -> list[dict]:
|
||||||
|
d = os.path.join(os.path.dirname(stub), "calls")
|
||||||
|
out = []
|
||||||
|
for n in sorted(os.listdir(d)) if os.path.isdir(d) else []:
|
||||||
|
if n.endswith(".json"):
|
||||||
|
with open(os.path.join(d, n)) as f:
|
||||||
|
out.append(json.load(f))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def stub_call_for(stub: str, job_prompt_marker: str) -> dict | None:
|
||||||
|
for c in stub_calls(stub):
|
||||||
|
if job_prompt_marker in c.get("prompt", ""):
|
||||||
|
return c
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def price(tokens: dict) -> float:
|
||||||
|
return (tokens["input"] * PRICE["in"] + tokens["output"] * PRICE["out"]
|
||||||
|
+ tokens["cache_creation"] * PRICE["cw"] + tokens["cache_read"] * PRICE["cr"])
|
||||||
|
|
||||||
|
|
||||||
|
def visible_windows() -> list[tuple]:
|
||||||
|
"""Visible top-level windows: [(hwnd, pid, class, title)] (Windows only)."""
|
||||||
|
import ctypes
|
||||||
|
import ctypes.wintypes as wt
|
||||||
|
u32 = ctypes.WinDLL("user32", use_last_error=True)
|
||||||
|
out = []
|
||||||
|
proto = ctypes.WINFUNCTYPE(wt.BOOL, wt.HWND, wt.LPARAM)
|
||||||
|
|
||||||
|
def cb(hwnd, _):
|
||||||
|
if u32.IsWindowVisible(hwnd):
|
||||||
|
pid = wt.DWORD()
|
||||||
|
u32.GetWindowThreadProcessId(hwnd, ctypes.byref(pid))
|
||||||
|
cls = ctypes.create_unicode_buffer(256)
|
||||||
|
u32.GetClassNameW(hwnd, cls, 256)
|
||||||
|
ttl = ctypes.create_unicode_buffer(256)
|
||||||
|
u32.GetWindowTextW(hwnd, ttl, 256)
|
||||||
|
out.append((int(hwnd or 0), pid.value, cls.value, ttl.value))
|
||||||
|
return True
|
||||||
|
u32.EnumWindows(proto(cb), 0)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for(pred, timeout=30.0, step=0.1):
|
||||||
|
deadline = time.monotonic() + timeout
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
v = pred()
|
||||||
|
if v:
|
||||||
|
return v
|
||||||
|
time.sleep(step)
|
||||||
|
return pred()
|
||||||
|
|
||||||
|
|
||||||
|
STATE_DIRS: list[str] = [] # every launch test's state dir, for the module-end process check
|
||||||
|
|
||||||
|
|
||||||
|
class LaunchBase(Base):
|
||||||
|
"""One run per test: a git repo as cwd, a stream stub as --binary, the real detached spawn."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
super().setUp()
|
||||||
|
for name, val in (("WAIT_POLL_S", 0.2),):
|
||||||
|
p = mock.patch.object(forks, name, val)
|
||||||
|
p.start()
|
||||||
|
self.addCleanup(p.stop)
|
||||||
|
p = mock.patch.object(forks, "USER_SETTINGS_PATH", os.path.join(self.tmp, "home", "settings.json"))
|
||||||
|
p.start()
|
||||||
|
self.addCleanup(p.stop)
|
||||||
|
self.cwd = os.path.join(self.tmp, "proj")
|
||||||
|
os.makedirs(self.cwd)
|
||||||
|
self.git("init", "-q")
|
||||||
|
self.commit_file("README.md", "hello\n", "initial")
|
||||||
|
os.makedirs(os.path.join(self.tmp, "reports"))
|
||||||
|
self.run_log = os.path.join(self.tmp, "reports", "2026-10-16-12-t-run.md")
|
||||||
|
self.state = forks.state_dir_for(self.run_log)
|
||||||
|
STATE_DIRS.append(self.state)
|
||||||
|
self.stub = make_stream_stub(self.tmp)
|
||||||
|
self.addCleanup(self.cleanup_processes)
|
||||||
|
self.nbrief = 0
|
||||||
|
|
||||||
|
# --- helpers ---
|
||||||
|
def git(self, *args):
|
||||||
|
r = subprocess.run(["git", "-c", "user.name=t", "-c", "user.email=t@t", *args], cwd=self.cwd,
|
||||||
|
capture_output=True, timeout=60)
|
||||||
|
self.assertEqual(r.returncode, 0, r.stderr)
|
||||||
|
return r.stdout.decode().strip()
|
||||||
|
|
||||||
|
def commit_file(self, name, text, msg):
|
||||||
|
with open(os.path.join(self.cwd, name), "w", newline="\n") as f:
|
||||||
|
f.write(text)
|
||||||
|
self.git("add", name)
|
||||||
|
self.git("commit", "-q", "-m", msg)
|
||||||
|
|
||||||
|
def setup_run(self, *extra, orphan_after=600, stall=900, cap=2):
|
||||||
|
code, out, _ = call(["init", "--run", self.run_log, "--model", TEST_MODEL, "--cwd", self.cwd,
|
||||||
|
"--binary", self.stub, "--orphan-after", str(orphan_after),
|
||||||
|
"--stall-after-s", str(stall), "--parallel-cap", str(cap), *extra])
|
||||||
|
self.assertEqual(code, 0, out)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def brief(self, *directives, text="Do the thing."):
|
||||||
|
self.nbrief += 1
|
||||||
|
p = os.path.join(self.tmp, "brief-%d.md" % self.nbrief)
|
||||||
|
marker = "BRIEF-MARKER-%d" % self.nbrief
|
||||||
|
with open(p, "w", newline="\n") as f:
|
||||||
|
f.write("%s\n%s\n" % (marker, text))
|
||||||
|
if directives:
|
||||||
|
f.write("@stub %s\n" % " ".join(directives))
|
||||||
|
return p
|
||||||
|
|
||||||
|
def cmd(self, *argv, code=0):
|
||||||
|
c, out, raw = call([argv[0], "--run", self.run_log, *argv[1:]])
|
||||||
|
self.assert_one_json_line(raw)
|
||||||
|
if code is not None:
|
||||||
|
self.assertEqual(c, code, out)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def train(self, base="B1", *directives):
|
||||||
|
out = self.cmd("train", "--base", base, "--brief", self.brief(*directives))
|
||||||
|
self.assertEqual(set(out), {"job", "base", "session_id"})
|
||||||
|
return out
|
||||||
|
|
||||||
|
def fork(self, *directives, base="B1", block="1.1", extra=(), code=0):
|
||||||
|
return self.cmd("fork", "--base", base, "--block", block, "--brief", self.brief(*directives),
|
||||||
|
*extra, code=code)
|
||||||
|
|
||||||
|
def wait(self, *jobs, timeout=60, any_=False):
|
||||||
|
argv = ["wait", "--run", self.run_log, "--timeout", str(timeout)]
|
||||||
|
argv += list(jobs) if jobs else ["--all"]
|
||||||
|
if any_:
|
||||||
|
argv.append("--any")
|
||||||
|
code, out, raw = call(argv)
|
||||||
|
self.assert_one_json_line(raw)
|
||||||
|
return code, out
|
||||||
|
|
||||||
|
def finish(self, job, timeout=60):
|
||||||
|
"""Wait for one job and return its result (from `done`)."""
|
||||||
|
code, out = self.wait(job, timeout=timeout)
|
||||||
|
self.assertIn(code, (0, 4), out)
|
||||||
|
got = [r for r in out["done"] if r["job"] == job]
|
||||||
|
self.assertEqual(len(got), 1, out)
|
||||||
|
return got[0]
|
||||||
|
|
||||||
|
def reg(self):
|
||||||
|
return forks.read_registry(self.state)
|
||||||
|
|
||||||
|
def runtime(self, job):
|
||||||
|
return forks.read_json(forks.job_file(self.state, job)) or {}
|
||||||
|
|
||||||
|
def ledger(self):
|
||||||
|
p = forks.ledger_path(self.state)
|
||||||
|
if not os.path.exists(p):
|
||||||
|
return []
|
||||||
|
with open(p, "rb") as f:
|
||||||
|
return [json.loads(x) for x in f.read().decode("utf-8").splitlines() if x.strip()]
|
||||||
|
|
||||||
|
def until_running(self, job, need_child=False):
|
||||||
|
"""Wait until the job's claude runs and the stub (started by a `fork_marked` brief) has
|
||||||
|
recorded itself, with its grandchild when `need_child`."""
|
||||||
|
def ready():
|
||||||
|
rt = self.runtime(job)
|
||||||
|
if not rt.get("claude"):
|
||||||
|
return None
|
||||||
|
c = stub_call_for(self.stub, "RUN-%s" % job)
|
||||||
|
if not c or (need_child and not c.get("child_pid")):
|
||||||
|
return None
|
||||||
|
return rt
|
||||||
|
rt = wait_for(ready, timeout=30)
|
||||||
|
self.assertTrue(rt, "job %s never got its claude running" % job)
|
||||||
|
return rt
|
||||||
|
|
||||||
|
def assert_tree_dead(self, recs, timeout=15):
|
||||||
|
wait_for(lambda: not any(forks.record_alive(r) for r in recs), timeout=timeout)
|
||||||
|
alive = [r for r in recs if forks.record_alive(r)]
|
||||||
|
self.assertEqual(alive, [], "processes still alive: %r" % alive)
|
||||||
|
|
||||||
|
def cleanup_processes(self):
|
||||||
|
reg = None
|
||||||
|
with contextlib.suppress(Exception):
|
||||||
|
reg = forks.read_registry(self.state)
|
||||||
|
for job, e in ((reg or {}).get("jobs") or {}).items():
|
||||||
|
rt = self.runtime(job)
|
||||||
|
recs = [r for r in (rt.get("supervisor"), e.get("supervisor"), rt.get("claude")) if r]
|
||||||
|
if recs:
|
||||||
|
forks.kill_tree(recs[0], extra=recs[1:])
|
||||||
|
|
||||||
|
def fork_marked(self, *directives, **kw):
|
||||||
|
"""A fork whose brief carries `RUN-<job>` so the stub call can be matched to the job."""
|
||||||
|
nxt = forks._next_job_id(self.reg())
|
||||||
|
p = self.brief(*directives, text="RUN-%s" % nxt)
|
||||||
|
out = self.cmd("fork", "--base", kw.get("base", "B1"), "--block", kw.get("block", "1.1"),
|
||||||
|
"--brief", p, *kw.get("extra", ()), code=kw.get("code", 0))
|
||||||
|
if kw.get("code", 0) == 0:
|
||||||
|
self.assertEqual(out["job"], nxt)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
class RuleTests(Base):
|
||||||
|
"""Pure functions: the P0.3 usage rule (real numbers from the P0.2 seq-1 fork), C5, C1."""
|
||||||
|
|
||||||
|
BASE_MU = {"claude-opus-5-5": {"inputTokens": 14, "outputTokens": 7250, "cacheReadInputTokens": 361690,
|
||||||
|
"cacheCreationInputTokens": 176553, "costUSD": 1.629818}}
|
||||||
|
FORK_RESULT = {
|
||||||
|
"usage": {"input_tokens": 14, "cache_creation_input_tokens": 3113, "cache_read_input_tokens": 1288193,
|
||||||
|
"output_tokens": 1448,
|
||||||
|
"cache_creation": {"ephemeral_1h_input_tokens": 3113, "ephemeral_5m_input_tokens": 0}},
|
||||||
|
"modelUsage": {"claude-opus-5-5": {"inputTokens": 28, "outputTokens": 8698,
|
||||||
|
"cacheReadInputTokens": 1649883,
|
||||||
|
"cacheCreationInputTokens": 179666, "costUSD": 1.9413766}},
|
||||||
|
"total_cost_usd": 1.9413766}
|
||||||
|
|
||||||
|
def test_usage_rule_real_fork(self):
|
||||||
|
u = forks.usage_from_result(self.FORK_RESULT, self.BASE_MU, 1.629818)
|
||||||
|
self.assertEqual(u["tokens"], {"input": 14, "output": 1448, "cache_creation": 3113,
|
||||||
|
"cache_creation_1h": 3113, "cache_creation_5m": 0,
|
||||||
|
"cache_read": 1288193})
|
||||||
|
self.assertEqual(u["models_used"], ["claude-opus-5-5"])
|
||||||
|
self.assertFalse(u["mismatch"])
|
||||||
|
self.assertAlmostEqual(u["cost_delta"], 0.311559, places=6)
|
||||||
|
|
||||||
|
def test_usage_rule_mismatch_and_new_model(self):
|
||||||
|
res = json.loads(json.dumps(self.FORK_RESULT))
|
||||||
|
res["usage"]["output_tokens"] += 1
|
||||||
|
self.assertTrue(forks.usage_from_result(res, self.BASE_MU, 1.629818)["mismatch"])
|
||||||
|
# a model that did not grow is not "used"; a train has no prior totals
|
||||||
|
res = json.loads(json.dumps(self.FORK_RESULT))
|
||||||
|
res["modelUsage"]["claude-haiku-5-5"] = {"inputTokens": 0, "outputTokens": 0}
|
||||||
|
self.assertEqual(forks.usage_from_result(res, self.BASE_MU, 0)["models_used"], ["claude-opus-5-5"])
|
||||||
|
self.assertTrue(forks.usage_from_result(self.FORK_RESULT, {}, 0)["mismatch"])
|
||||||
|
|
||||||
|
def test_cache_verdict(self):
|
||||||
|
u = lambda cr: {"cache_read_input_tokens": cr}
|
||||||
|
self.assertEqual(forks.cache_verdict(u(182202), 182204), "hit")
|
||||||
|
self.assertEqual(forks.cache_verdict(u(900), 1000), "hit")
|
||||||
|
self.assertEqual(forks.cache_verdict(u(899), 1000), "partial")
|
||||||
|
self.assertEqual(forks.cache_verdict(u(7469), 92877), "partial") # P0.6 base B
|
||||||
|
self.assertEqual(forks.cache_verdict(u(0), 1000), "miss")
|
||||||
|
self.assertIsNone(forks.cache_verdict(None, 1000))
|
||||||
|
|
||||||
|
def test_warmth(self):
|
||||||
|
cfg = {"ttl_s": 3600}
|
||||||
|
t0 = 1_800_000_000.0
|
||||||
|
base = {"last_hit": forks._iso(t0)}
|
||||||
|
self.assertEqual(forks.base_warmth(base, cfg, t0 + 10)["base_state"], "warm")
|
||||||
|
self.assertEqual(forks.base_warmth(base, cfg, t0 + 3600 - 361)["base_state"], "warm")
|
||||||
|
self.assertEqual(forks.base_warmth(base, cfg, t0 + 3600 - 359)["base_state"], "expiring")
|
||||||
|
self.assertEqual(forks.base_warmth(base, cfg, t0 + 3601)["base_state"], "cold")
|
||||||
|
self.assertEqual(forks._ts(forks.base_warmth(base, cfg, t0)["warm_until"]), t0 + 3600)
|
||||||
|
short = {"ttl_s": 300}
|
||||||
|
self.assertEqual(forks.base_warmth(base, short, t0 + 300 - 121)["base_state"], "warm")
|
||||||
|
self.assertEqual(forks.base_warmth(base, short, t0 + 300 - 119)["base_state"], "expiring")
|
||||||
|
self.assertEqual(forks.base_warmth({}, cfg, t0)["base_state"], "cold")
|
||||||
|
|
||||||
|
def test_stream_state(self):
|
||||||
|
ss = forks.StreamState()
|
||||||
|
u1 = {"input_tokens": 2, "cache_creation_input_tokens": 10, "cache_read_input_tokens": 100}
|
||||||
|
u2 = {"input_tokens": 2, "cache_creation_input_tokens": 5, "cache_read_input_tokens": 112}
|
||||||
|
for ev in ({"type": "system", "subtype": "init", "session_id": "s1", "tools": ["Read", "Bash"],
|
||||||
|
"permissionMode": "dontAsk", "apiKeySource": "none",
|
||||||
|
"plugins": [{"name": "spire", "version": "0.2.2"}], "agents": ["claude"]},
|
||||||
|
{"type": "assistant", "message": {"id": "m1", "usage": u1}},
|
||||||
|
{"type": "assistant", "message": {"id": "m1", "usage": u1}},
|
||||||
|
{"type": "system", "subtype": "informational", "level": "warning",
|
||||||
|
"content": "Model \"x\" is restricted by your organization's settings."},
|
||||||
|
{"type": "assistant", "message": {"id": "m2", "usage": u2}},
|
||||||
|
{"type": "result", "num_turns": 2}):
|
||||||
|
ss.feed(ev, 5.0)
|
||||||
|
self.assertEqual((ss.session_id, ss.turns), ("s1", 2))
|
||||||
|
self.assertEqual(ss.first_usage, u1)
|
||||||
|
self.assertEqual(forks.context_tokens(ss.last_usage), 119)
|
||||||
|
self.assertIn("is restricted", ss.refused)
|
||||||
|
rt = forks.runtime_fingerprint(ss.init)
|
||||||
|
self.assertEqual(rt["plugins"], ["spire@0.2.2"])
|
||||||
|
other = dict(rt, plugins=["spire@0.3.0"], apiKeySource="oauth")
|
||||||
|
self.assertEqual(forks.runtime_drift(other, rt, {}), ["apiKeySource", "plugins"])
|
||||||
|
self.assertEqual(forks.runtime_drift(rt, None, {"permission_mode": "default"}), ["permissionMode"])
|
||||||
|
|
||||||
|
|
||||||
|
class LaunchTests(LaunchBase):
|
||||||
|
def test_train_fork_continue_end_to_end(self):
|
||||||
|
self.setup_run()
|
||||||
|
t = self.train("B1", "turns=4")
|
||||||
|
self.assertEqual((t["job"], t["base"]), ("j0001", "B1"))
|
||||||
|
r = self.finish("j0001")
|
||||||
|
self.assertEqual(r["status"], "ok", r)
|
||||||
|
self.assertEqual(set(r), {"job", "kind", "base", "block", "phase", "status", "flags", "fingerprint",
|
||||||
|
"session", "parent_session", "models_used", "tokens", "cost_delta_usd",
|
||||||
|
"cache", "base_state", "wall_s", "turns", "denials", "note", "raw",
|
||||||
|
"paths", "reason", "model_requested"})
|
||||||
|
self.assertEqual(r["kind"], "train")
|
||||||
|
self.assertIsNone(r["cache"])
|
||||||
|
self.assertEqual(r["session"], t["session_id"])
|
||||||
|
self.assertIsNone(r["parent_session"])
|
||||||
|
self.assertEqual(r["models_used"], [TEST_MODEL])
|
||||||
|
self.assertEqual(r["model_requested"], TEST_MODEL)
|
||||||
|
self.assertEqual(r["fingerprint"], "ok")
|
||||||
|
self.assertEqual(r["turns"], 4)
|
||||||
|
self.assertEqual(r["flags"], [])
|
||||||
|
self.assertIsNone(r["reason"])
|
||||||
|
self.assertAlmostEqual(r["cost_delta_usd"], price(r["tokens"]), places=6)
|
||||||
|
self.assertEqual(r["raw"], "raw/j0001.jsonl")
|
||||||
|
self.assertTrue(os.path.isfile(os.path.join(self.state, r["raw"])))
|
||||||
|
base = self.reg()["bases"]["B1"]
|
||||||
|
self.assertEqual(base["status"], "active")
|
||||||
|
self.assertEqual(base["session"], t["session_id"])
|
||||||
|
self.assertEqual(base["base_commit"], self.git("rev-parse", "HEAD"))
|
||||||
|
# base context = input + creation + read of the last assistant message: 2 + 300 + (prev)
|
||||||
|
self.assertEqual(base["context_tokens"], 2 + 1000 + 3 * (2 + 300) + 0)
|
||||||
|
self.assertIsNotNone(base["last_hit"])
|
||||||
|
self.assertAlmostEqual(base["training_cost_usd"], r["cost_delta_usd"], places=6)
|
||||||
|
# the command line: absolute stored binary, full fingerprint, pre-assigned session id
|
||||||
|
call0 = stub_calls(self.stub)[0]
|
||||||
|
argv = call0["argv"]
|
||||||
|
self.assertEqual(argv[argv.index("--session-id") + 1], t["session_id"])
|
||||||
|
self.assertEqual(argv[argv.index("--model") + 1], TEST_MODEL)
|
||||||
|
self.assertEqual(argv[argv.index("--permission-mode") + 1], "dontAsk")
|
||||||
|
self.assertIn("--strict-mcp-config", argv)
|
||||||
|
self.assertEqual(argv[argv.index("--append-system-prompt-file") + 1], forks.preamble_path())
|
||||||
|
self.assertNotIn("--resume", argv)
|
||||||
|
self.assertEqual(self.reg()["jobs"]["j0001"]["argv"][0], forks.norm_path(self.stub))
|
||||||
|
self.assertEqual(os.path.normcase(call0["cwd"]), os.path.normcase(self.cwd))
|
||||||
|
self.assertEqual(call0["env"]["CLAUDE_CODE_EXECPATH"], forks.norm_path(self.stub))
|
||||||
|
self.assertEqual(call0["env"]["SPIRE_FORKS_CHILD"], "1")
|
||||||
|
self.assertEqual(call0["env"]["CLAUDE_CODE_PROMPT_CACHE_TTL"], "1h")
|
||||||
|
self.assertIsNone(call0["env"]["VIRTUAL_ENV"])
|
||||||
|
self.assertIn("READY", call0["prompt"]) # templates/train.md frame
|
||||||
|
# --- fork, after a commit on the trunk ---
|
||||||
|
self.commit_file("src.py", "x = 1\n", "add src after training")
|
||||||
|
os.makedirs(os.path.join(self.state, "notes"), exist_ok=True)
|
||||||
|
with open(os.path.join(self.state, "notes", "1.1.md"), "w") as f:
|
||||||
|
f.write("note\n")
|
||||||
|
f_out = self.fork("turns=2", extra=("--phase", "1", "--paths", "src.py", "docs/a.md"))
|
||||||
|
self.assertEqual(set(f_out), {"job", "base", "base_state", "warm_until", "rebase_due",
|
||||||
|
"retrain_due", "reasons"})
|
||||||
|
self.assertEqual(f_out["base_state"], "warm")
|
||||||
|
self.assertFalse(f_out["rebase_due"])
|
||||||
|
self.assertFalse(f_out["retrain_due"])
|
||||||
|
self.assertEqual(f_out["reasons"], [])
|
||||||
|
fj = f_out["job"]
|
||||||
|
r2 = self.finish(fj)
|
||||||
|
self.assertEqual(r2["status"], "ok", r2)
|
||||||
|
self.assertEqual(r2["kind"], "fork")
|
||||||
|
self.assertEqual(r2["cache"], "hit")
|
||||||
|
self.assertEqual(r2["phase"], 1)
|
||||||
|
self.assertEqual(r2["base_state"], "warm")
|
||||||
|
self.assertEqual(r2["flags"], [])
|
||||||
|
self.assertEqual(r2["fingerprint"], "ok")
|
||||||
|
self.assertEqual(r2["parent_session"], t["session_id"])
|
||||||
|
self.assertNotEqual(r2["session"], t["session_id"])
|
||||||
|
self.assertEqual(r2["paths"], ["src.py", "docs/a.md"])
|
||||||
|
self.assertEqual(r2["note"], "notes/1.1.md")
|
||||||
|
self.assertEqual(r2["models_used"], [TEST_MODEL])
|
||||||
|
# first request reads the base context, the second the base context plus the first turn
|
||||||
|
self.assertEqual(r2["tokens"]["cache_read"], 2 * base["context_tokens"] + 1002)
|
||||||
|
self.assertAlmostEqual(r2["cost_delta_usd"], price(r2["tokens"]), places=6)
|
||||||
|
fcall = [c for c in stub_calls(self.stub) if c["session"] == r2["session"]][0]
|
||||||
|
a = fcall["argv"]
|
||||||
|
self.assertEqual(a[a.index("--resume") + 1], t["session_id"])
|
||||||
|
self.assertIn("--fork-session", a)
|
||||||
|
self.assertEqual(a[a.index("--session-id") + 1], r2["session"])
|
||||||
|
self.assertIn("--append-system-prompt-file", a) # flags passed again
|
||||||
|
with open(os.path.join(self.state, "briefs", fj + ".md"), encoding="utf-8") as fh:
|
||||||
|
brief_txt = fh.read()
|
||||||
|
self.assertIn("add src after training", brief_txt) # delta header: git log
|
||||||
|
self.assertIn("1 file changed", brief_txt) # delta header: shortstat
|
||||||
|
self.assertIn("stale", brief_txt) # R13 sentence
|
||||||
|
self.assertIn(os.path.join(self.state, "notes", "1.1.md"), brief_txt)
|
||||||
|
# --- continue the fork: cost delta against the fork's own last total ---
|
||||||
|
c_out = self.cmd("continue", "--job", fj, "--brief", self.brief("turns=1"))
|
||||||
|
self.assertEqual(set(c_out), {"job"})
|
||||||
|
r3 = self.finish(c_out["job"])
|
||||||
|
self.assertEqual(r3["status"], "ok", r3)
|
||||||
|
self.assertEqual(r3["kind"], "continue")
|
||||||
|
self.assertEqual(r3["session"], r2["session"])
|
||||||
|
self.assertEqual(r3["block"], "1.1")
|
||||||
|
self.assertAlmostEqual(r3["cost_delta_usd"], price(r3["tokens"]), places=6)
|
||||||
|
self.assertEqual(r3["cache"], "hit")
|
||||||
|
ccall = [c for c in stub_calls(self.stub) if c["session"] == r2["session"] and c is not fcall
|
||||||
|
and "--fork-session" not in c["argv"]][0]
|
||||||
|
self.assertEqual(ccall["argv"][ccall["argv"].index("--resume") + 1], r2["session"])
|
||||||
|
self.assertNotIn("--session-id", ccall["argv"])
|
||||||
|
# ledger lines match the results
|
||||||
|
led = {x["job"]: x for x in self.ledger()}
|
||||||
|
self.assertEqual(sorted(led), ["j0001", fj, c_out["job"]])
|
||||||
|
for res in (r, r2, r3):
|
||||||
|
line = led[res["job"]]
|
||||||
|
for k in ("status", "flags", "tokens", "cost_delta_usd", "cache", "models_used", "session",
|
||||||
|
"parent_session", "fingerprint", "turns", "reason", "note"):
|
||||||
|
self.assertEqual(line[k], res[k], k)
|
||||||
|
self.assertEqual(line["binary"], STUB_VERSION)
|
||||||
|
self.assertIsNotNone(line["cost_cumulative_usd"])
|
||||||
|
# list shows every base and job; nothing running
|
||||||
|
lst = self.cmd("list")
|
||||||
|
self.assertEqual([b["name"] for b in lst["bases"]], ["B1"])
|
||||||
|
self.assertEqual(lst["bases"][0]["base_state"], "warm")
|
||||||
|
self.assertEqual(len(lst["jobs"]), 3)
|
||||||
|
self.assertEqual(lst["running"], [])
|
||||||
|
self.assertIsNotNone(self.reg()["run"]["last_orchestrator_call_at"])
|
||||||
|
# refusals on continue
|
||||||
|
self.cmd("continue", "--job", "j0001", "--brief", self.brief(), code=2) # not a fork
|
||||||
|
self.cmd("continue", "--job", "j0999", "--brief", self.brief(), code=2) # unknown
|
||||||
|
self.cmd("train", "--base", "B1", "--brief", self.brief(), code=2) # base exists
|
||||||
|
|
||||||
|
def test_cache_verdicts_and_cold_start(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
a = self.fork("cache=partial", extra=("--max-budget-usd", "2.5"))["job"]
|
||||||
|
b = self.fork("cache=miss")["job"]
|
||||||
|
argv_a = self.reg()["jobs"][a]["argv"]
|
||||||
|
self.assertEqual(argv_a[argv_a.index("--max-budget-usd") + 1], "2.5")
|
||||||
|
self.assertNotIn("--max-budget-usd", self.reg()["jobs"][b]["argv"])
|
||||||
|
code, out = self.wait(a, b)
|
||||||
|
self.assertEqual(code, 0, out)
|
||||||
|
res = {x["job"]: x for x in out["done"]}
|
||||||
|
self.assertEqual(res[a]["cache"], "partial")
|
||||||
|
self.assertEqual(res[b]["cache"], "miss")
|
||||||
|
for j in (a, b):
|
||||||
|
self.assertIn("cold_start", res[j]["flags"])
|
||||||
|
self.assertEqual(res[j]["status"], "ok")
|
||||||
|
self.assertEqual(res[j]["base_state"], "warm")
|
||||||
|
# a cold base: hit, but cold at launch -> cold_start
|
||||||
|
with forks.RegistryLock(self.state):
|
||||||
|
reg = forks.read_registry(self.state)
|
||||||
|
reg["bases"]["B1"]["last_hit"] = forks._iso(time.time() - 4000)
|
||||||
|
forks.write_registry(self.state, reg)
|
||||||
|
out = self.fork()
|
||||||
|
self.assertEqual(out["base_state"], "cold")
|
||||||
|
r = self.finish(out["job"])
|
||||||
|
self.assertEqual(r["cache"], "hit")
|
||||||
|
self.assertEqual(r["base_state"], "cold")
|
||||||
|
self.assertIn("cold_start", r["flags"])
|
||||||
|
# the hit refreshed last_hit: the base is warm again
|
||||||
|
self.assertEqual(self.cmd("list")["bases"][0]["base_state"], "warm")
|
||||||
|
|
||||||
|
def test_failures_flags_and_drift(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
cases = [("die=1",), ("error",), ("restricted=1",), ("model=claude-other-9",), ("deny",),
|
||||||
|
("mismatch",), ("plugin=0.3.0",)]
|
||||||
|
results = {}
|
||||||
|
for i in range(0, len(cases), 2):
|
||||||
|
jobs = [self.fork(*c)["job"] for c in cases[i:i + 2]]
|
||||||
|
code, out = self.wait(*jobs)
|
||||||
|
for x in out["done"]:
|
||||||
|
results[x["job"]] = x
|
||||||
|
r = [results["j%04d" % (i + 2)] for i in range(len(cases))]
|
||||||
|
die, err, restricted, substituted, deny, mismatch, plugin = r
|
||||||
|
self.assertEqual((die["status"], die["reason"]), ("failed", "exit_code"))
|
||||||
|
self.assertEqual(die["turns"], 1)
|
||||||
|
self.assertIsNone(die["cost_delta_usd"])
|
||||||
|
self.assertEqual((err["status"], err["reason"]), ("failed", "is_error"))
|
||||||
|
self.assertEqual((restricted["status"], restricted["reason"]), ("failed", "model_refused"))
|
||||||
|
self.assertEqual((substituted["status"], substituted["reason"]), ("failed", "model_refused"))
|
||||||
|
self.assertEqual(substituted["models_used"], ["claude-other-9"])
|
||||||
|
self.assertEqual(deny["status"], "ok")
|
||||||
|
self.assertIn("needs_attention", deny["flags"])
|
||||||
|
self.assertEqual(deny["denials"], 1)
|
||||||
|
self.assertEqual(mismatch["status"], "ok")
|
||||||
|
self.assertIn("usage_mismatch", mismatch["flags"])
|
||||||
|
self.assertEqual(plugin["fingerprint"], "drift:plugins")
|
||||||
|
self.assertNotIn("cold_start", plugin["flags"])
|
||||||
|
# model refused while training: the base fails
|
||||||
|
t = self.train("B2", "restricted=1")
|
||||||
|
r = self.finish(t["job"])
|
||||||
|
self.assertEqual((r["status"], r["reason"]), ("failed", "model_refused"))
|
||||||
|
self.assertEqual(self.reg()["bases"]["B2"]["status"], "failed")
|
||||||
|
self.fork(base="B2", code=2) # not active
|
||||||
|
# binary drift: the stored binary changed size and mtime since training
|
||||||
|
with open(self.stub, "a", newline="\r\n" if IS_WINDOWS else "\n") as f:
|
||||||
|
f.write("rem drift\n" if IS_WINDOWS else "# drift\n")
|
||||||
|
r = self.finish(self.fork()["job"])
|
||||||
|
self.assertEqual(r["status"], "ok")
|
||||||
|
self.assertEqual(r["fingerprint"], "drift:binary")
|
||||||
|
|
||||||
|
def test_timeout_and_stalled(self):
|
||||||
|
self.setup_run(stall=1)
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
slow = self.fork("silent=2.5")["job"]
|
||||||
|
hang = self.fork_marked("hang=1", extra=("--max-minutes", "0.06"))["job"]
|
||||||
|
rt = self.until_running(hang)
|
||||||
|
tree = forks.process_tree(rt["supervisor"])
|
||||||
|
self.assertGreaterEqual(len(tree), 3)
|
||||||
|
code, out = self.wait(slow, hang, timeout=60)
|
||||||
|
self.assertEqual(code, 4, out)
|
||||||
|
res = {x["job"]: x for x in out["done"]}
|
||||||
|
self.assertEqual(res[slow]["status"], "ok")
|
||||||
|
self.assertIn("stalled", res[slow]["flags"])
|
||||||
|
self.assertEqual(res[hang]["status"], "timeout")
|
||||||
|
self.assertIsNone(res[hang]["reason"])
|
||||||
|
self.assertIn("stalled", res[hang]["flags"])
|
||||||
|
self.assert_tree_dead(tree) # claude tree killed; the supervisor exited
|
||||||
|
|
||||||
|
def test_kill_tree_and_no_visible_window(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
job = self.fork_marked("hang=1", "child")["job"]
|
||||||
|
rt = self.until_running(job, need_child=True)
|
||||||
|
sup = rt["supervisor"]
|
||||||
|
tree = forks.process_tree(sup)
|
||||||
|
pids = {p["pid"] for p in tree}
|
||||||
|
stub = stub_call_for(self.stub, "RUN-%s" % job)
|
||||||
|
self.assertIn(sup["pid"], pids)
|
||||||
|
self.assertIn(rt["claude"]["pid"], pids)
|
||||||
|
self.assertIn(stub["pid"], pids)
|
||||||
|
self.assertIn(stub["child_pid"], pids)
|
||||||
|
if IS_WINDOWS:
|
||||||
|
# spawned through CIM: the supervisor's parent is the WMI provider host, not us
|
||||||
|
table = {pid: (ppid, name) for pid, ppid, name in forks.process_table()}
|
||||||
|
parent = table.get(table[sup["pid"]][0], (None, None))[1]
|
||||||
|
self.assertEqual((parent or "").lower(), "wmiprvse.exe")
|
||||||
|
self.assertIn("python.exe", [p["name"].lower() for p in tree])
|
||||||
|
self.assertIn("cmd.exe", [p["name"].lower() for p in tree])
|
||||||
|
all_windows = visible_windows()
|
||||||
|
self.assertGreater(len(all_windows), 0, "window enumeration sees nothing (no desktop?)")
|
||||||
|
for _ in range(3):
|
||||||
|
mine = [w for w in visible_windows() if w[1] in pids]
|
||||||
|
self.assertEqual(mine, [], "visible window of the job's process tree")
|
||||||
|
time.sleep(0.2)
|
||||||
|
out = self.cmd("kill", "--job", job)
|
||||||
|
self.assertTrue(out["killed"])
|
||||||
|
self.assertTrue(out["ok"], out)
|
||||||
|
self.assertEqual(out["survivors"], [])
|
||||||
|
self.assertEqual(out["status"], "killed")
|
||||||
|
snap_pids = {p["pid"] for p in out["snapshot"]}
|
||||||
|
for pid in (sup["pid"], rt["claude"]["pid"], stub["pid"], stub["child_pid"]):
|
||||||
|
self.assertIn(pid, snap_pids)
|
||||||
|
self.assert_tree_dead(tree, timeout=1)
|
||||||
|
r = self.finish(job)
|
||||||
|
self.assertEqual((r["status"], r["reason"]), ("killed", "kill"))
|
||||||
|
self.assertEqual(r["turns"], 1)
|
||||||
|
self.assertEqual([x["status"] for x in self.ledger() if x["job"] == job], ["killed"])
|
||||||
|
again = self.cmd("kill", "--job", job)
|
||||||
|
self.assertFalse(again["killed"])
|
||||||
|
self.cmd("kill", "--job", "j0999", code=2)
|
||||||
|
|
||||||
|
def test_lost(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
job = self.fork_marked("hang=1")["job"]
|
||||||
|
rt = self.until_running(job)
|
||||||
|
tree = forks.process_tree(rt["supervisor"])
|
||||||
|
self.assertTrue(forks.terminate(rt["supervisor"]))
|
||||||
|
self.assertTrue(wait_for(lambda: not forks.record_alive(rt["supervisor"]), 10))
|
||||||
|
lst = self.cmd("list") # heartbeat is fresh: not lost yet
|
||||||
|
self.assertEqual(lst["running"], [job])
|
||||||
|
later = time.time() + forks.LOST_AFTER_S + 30
|
||||||
|
with mock.patch.object(forks, "_now", lambda: later):
|
||||||
|
lst = self.cmd("list")
|
||||||
|
self.assertEqual(lst["lost_now"], [job])
|
||||||
|
self.assertEqual(lst["running"], [])
|
||||||
|
self.assert_tree_dead(tree) # its orphaned claude tree went with it
|
||||||
|
code, out = self.wait(job)
|
||||||
|
self.assertEqual(code, 4)
|
||||||
|
self.assertEqual((out["done"][0]["status"], out["done"][0]["reason"]), ("lost", "supervisor_dead"))
|
||||||
|
self.assertEqual([x["status"] for x in self.ledger() if x["job"] == job], ["lost"])
|
||||||
|
|
||||||
|
def test_orphan_watchdog(self):
|
||||||
|
self.setup_run() # orphan_after 600 s
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
job = self.fork_marked("hang=1", "child")["job"]
|
||||||
|
rt = self.until_running(job, need_child=True)
|
||||||
|
tree = forks.process_tree(rt["claude"])
|
||||||
|
self.assertGreaterEqual(len(tree), 2)
|
||||||
|
self.assertEqual(self.reg()["jobs"][job]["status"], "running")
|
||||||
|
# injected clock: the last orchestrator call is recorded as 2000 s ago
|
||||||
|
past = time.time() - 2000
|
||||||
|
with mock.patch.object(forks, "_now", lambda: past):
|
||||||
|
forks.touch_orchestrator(self.state)
|
||||||
|
# no orchestrator call from here on: poll the registry directly (no touch)
|
||||||
|
final = wait_for(lambda: self.reg()["jobs"][job]["status"] not in forks.RUNNING, timeout=30, step=0.2)
|
||||||
|
self.assertTrue(final)
|
||||||
|
e = self.reg()["jobs"][job]
|
||||||
|
self.assertEqual((e["status"], e["reason"]), ("killed", "orphaned"))
|
||||||
|
self.assertIn("orphaned", e["flags"])
|
||||||
|
self.assert_tree_dead(tree)
|
||||||
|
self.assertTrue(wait_for(lambda: not forks.record_alive(rt["supervisor"]), 15))
|
||||||
|
self.assertEqual([x["flags"] for x in self.ledger() if x["job"] == job], [e["flags"]])
|
||||||
|
|
||||||
|
def test_expire_kills_running_jobs(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
job = self.fork_marked("hang=1")["job"]
|
||||||
|
rt = self.until_running(job)
|
||||||
|
tree = forks.process_tree(rt["supervisor"])
|
||||||
|
out = self.cmd("expire", "--base", "B1")
|
||||||
|
self.assertEqual(out["retired"], ["B1"])
|
||||||
|
self.assertEqual([k["job"] for k in out["killed"]], [job])
|
||||||
|
self.assertTrue(out["ok"])
|
||||||
|
self.assert_tree_dead(tree)
|
||||||
|
r = self.finish(job)
|
||||||
|
self.assertEqual((r["status"], r["reason"]), ("killed", "expire"))
|
||||||
|
self.assertIn("expired", r["flags"])
|
||||||
|
self.assertEqual(self.reg()["bases"]["B1"]["status"], "retired")
|
||||||
|
self.fork(code=2) # retired: refused without --force
|
||||||
|
r = self.finish(self.fork(extra=("--force",))["job"])
|
||||||
|
self.assertEqual(r["status"], "ok")
|
||||||
|
self.assertEqual(self.cmd("expire", "--all")["retired"], [])
|
||||||
|
|
||||||
|
def test_wait_exit_codes(self):
|
||||||
|
self.setup_run()
|
||||||
|
self.train("B1")
|
||||||
|
self.assertEqual(self.wait()[0], 0) # settles the training job
|
||||||
|
a = self.fork("die=1")["job"]
|
||||||
|
b = self.fork("sleep=2", "turns=5")["job"]
|
||||||
|
wait_for(lambda: self.reg()["jobs"][a]["status"] == "failed", timeout=30)
|
||||||
|
code, out = self.wait(timeout=1) # mixed: one failed, one running
|
||||||
|
self.assertEqual(code, 3, out)
|
||||||
|
self.assertEqual([x["job"] for x in out["done"]], [a])
|
||||||
|
self.assertEqual(out["done"][0]["status"], "failed")
|
||||||
|
self.assertEqual([x["job"] for x in out["running"]], [b])
|
||||||
|
self.assertEqual(set(out["running"][0]), {"job", "block", "elapsed_s", "last_event_s_ago", "turns"})
|
||||||
|
self.assertEqual(out["pings"], [])
|
||||||
|
code, out = self.wait(timeout=60) # condition met, one failed -> 4
|
||||||
|
self.assertEqual(code, 4, out)
|
||||||
|
self.assertEqual([x["job"] for x in out["done"]], [b]) # a is not returned twice
|
||||||
|
self.assertEqual(out["done"][0]["status"], "ok")
|
||||||
|
code, out = self.wait(timeout=5) # nothing left
|
||||||
|
self.assertEqual((code, out["done"], out["running"]), (0, [], []))
|
||||||
|
self.assertEqual(self.wait(b)[0], 0)
|
||||||
|
self.assertEqual(self.wait(a)[0], 4)
|
||||||
|
# --any
|
||||||
|
c = self.fork("die=1")["job"]
|
||||||
|
d = self.fork("sleep=1", "turns=3")["job"]
|
||||||
|
code, out = self.wait(timeout=60, any_=True)
|
||||||
|
self.assertEqual(code, 4, out)
|
||||||
|
self.assertEqual([x["job"] for x in out["done"]], [c])
|
||||||
|
code, out = self.wait(timeout=60)
|
||||||
|
self.assertEqual(code, 0, out)
|
||||||
|
self.assertEqual([x["job"] for x in out["done"]], [d])
|
||||||
|
# usage errors
|
||||||
|
for argv in (["wait", "--run", self.run_log, "--all", "--timeout", "571"],
|
||||||
|
["wait", "--run", self.run_log],
|
||||||
|
["wait", "--run", self.run_log, "j0999"],
|
||||||
|
["wait", "--run", self.run_log, "--all", "j0001"]):
|
||||||
|
self.assertEqual(call(argv)[0], 2, argv)
|
||||||
|
|
||||||
|
def test_spawn_failed(self):
|
||||||
|
self.setup_run()
|
||||||
|
failed = {"ok": False, "method": forks.SPAWN_METHOD, "detail": "Win32_Process.Create returned 9"}
|
||||||
|
with mock.patch.object(forks, "spawn_supervisor", lambda *a: failed):
|
||||||
|
t = self.train("B1")
|
||||||
|
code, out = self.wait(t["job"], timeout=5)
|
||||||
|
self.assertEqual(code, 4, out)
|
||||||
|
r = out["done"][0]
|
||||||
|
self.assertEqual((r["status"], r["reason"]), ("failed", "spawn_failed"))
|
||||||
|
self.assertEqual(r["tokens"], {k: None for k in forks.TOKEN_KEYS})
|
||||||
|
self.assertEqual(self.reg()["bases"]["B1"]["status"], "failed")
|
||||||
|
self.assertEqual(self.reg()["jobs"][t["job"]]["reason_detail"], failed["detail"])
|
||||||
|
self.assertEqual([x["reason"] for x in self.ledger()], ["spawn_failed"])
|
||||||
|
self.assertEqual(stub_calls(self.stub), [])
|
||||||
|
|
||||||
|
def test_refusals(self):
|
||||||
|
# launch-field overrides on fork and continue (refused before anything is read)
|
||||||
|
values = {"model": "claude-x-1", "effort": "low", "permission_mode": "default", "tools": "Read",
|
||||||
|
"allowed_tools": "Read", "disallowed_tools": "Bash", "agent": "a", "cwd": self.tmp,
|
||||||
|
"binary": self.stub}
|
||||||
|
for name in forks.LAUNCH_FIELDS:
|
||||||
|
flag = "--" + name.replace("_", "-")
|
||||||
|
for argv in (["fork", "--base", "B1", "--block", "1", "--brief", "b.md"],
|
||||||
|
["continue", "--job", "j0001", "--brief", "b.md"]):
|
||||||
|
with self.subTest(field=name, cmd=argv[0]):
|
||||||
|
out = self.cmd(*argv, flag, values[name], code=2)
|
||||||
|
self.assertEqual(out["fields"], [name])
|
||||||
|
self.assertFalse(os.path.exists(self.state))
|
||||||
|
self.setup_run()
|
||||||
|
# N2: parallel needs --paths
|
||||||
|
out = self.cmd("fork", "--base", "B1", "--block", "1", "--brief", self.brief(), "--mode",
|
||||||
|
"parallel", code=2)
|
||||||
|
self.assertIn("--paths", out["error"])
|
||||||
|
self.fork(base="B9", code=2) # unknown base
|
||||||
|
self.train("B1")
|
||||||
|
self.finish("j0001")
|
||||||
|
# parallel cap (2): two hanging parallel forks, a third launch is refused
|
||||||
|
j1 = self.fork_marked("hang=1", extra=("--mode", "parallel", "--paths", "a.py"))["job"]
|
||||||
|
j2 = self.fork_marked("hang=1", extra=("--mode", "parallel", "--paths", "b.py", "c.py"))["job"]
|
||||||
|
out = self.fork(code=2)
|
||||||
|
self.assertEqual(out["error"], "parallel cap reached")
|
||||||
|
self.assertEqual(out["running"], [j1, j2])
|
||||||
|
out = self.cmd("continue", "--job", j1, "--brief", self.brief(), code=2) # still running
|
||||||
|
self.assertIn("running", out["error"])
|
||||||
|
# N3: launching refused inside a fork; list and wait stay allowed
|
||||||
|
os.environ[forks.CHILD_MARKER] = "1"
|
||||||
|
try:
|
||||||
|
for argv in (["train", "--base", "B3", "--brief", self.brief()],
|
||||||
|
["fork", "--base", "B1", "--block", "2", "--brief", self.brief()],
|
||||||
|
["continue", "--job", j1, "--brief", self.brief()]):
|
||||||
|
out = self.cmd(*argv, code=2)
|
||||||
|
self.assertIn("inside a fork", out["error"])
|
||||||
|
self.assertEqual(self.cmd("list")["running"], [j1, j2])
|
||||||
|
code, out = self.wait(j1, timeout=1)
|
||||||
|
self.assertEqual(code, 3)
|
||||||
|
finally:
|
||||||
|
os.environ.pop(forks.CHILD_MARKER, None)
|
||||||
|
for j in (j1, j2):
|
||||||
|
self.until_running(j)
|
||||||
|
self.assertTrue(self.cmd("kill", "--job", j)["ok"])
|
||||||
|
code, out = self.wait(j1, j2)
|
||||||
|
self.assertEqual(code, 4)
|
||||||
|
self.assertEqual({x["job"]: x["paths"] for x in out["done"]}, {j1: ["a.py"], j2: ["b.py", "c.py"]})
|
||||||
|
# missing binary: the stored path is gone
|
||||||
|
moved = self.stub + ".moved"
|
||||||
|
os.replace(self.stub, moved)
|
||||||
|
try:
|
||||||
|
out = self.fork(code=2)
|
||||||
|
self.assertEqual(out["reason"], "binary_missing")
|
||||||
|
out = self.cmd("train", "--base", "B4", "--brief", self.brief(), code=2)
|
||||||
|
self.assertEqual(out["reason"], "binary_missing")
|
||||||
|
finally:
|
||||||
|
os.replace(moved, self.stub)
|
||||||
|
self.cmd("fork", "--base", "B1", "--block", "1", "--brief", os.path.join(self.tmp, "nope.md"), code=2)
|
||||||
|
|
||||||
|
|
||||||
|
def tearDownModule():
|
||||||
|
"""No supervisor or stub of this suite may survive it (checked by command line, never by name)."""
|
||||||
|
if not IS_WINDOWS or not STATE_DIRS:
|
||||||
|
return
|
||||||
|
ps = ("$p = 'forks' + '-test-'; Get-CimInstance Win32_Process | "
|
||||||
|
"Where-Object { $_.CommandLine -and $_.CommandLine.Contains($p) } | "
|
||||||
|
"ForEach-Object { '{0}|{1}' -f $_.ProcessId, $_.CommandLine }")
|
||||||
|
r = subprocess.run(["powershell", "-NoProfile", "-NonInteractive", "-Command", ps],
|
||||||
|
capture_output=True, timeout=120, creationflags=0x08000000)
|
||||||
|
lines = [x for x in r.stdout.decode("utf-8", "replace").splitlines() if x.strip()]
|
||||||
|
for x in lines:
|
||||||
|
with contextlib.suppress(ValueError, OSError):
|
||||||
|
forks.terminate(forks.pid_record(int(x.split("|", 1)[0])))
|
||||||
|
if lines:
|
||||||
|
raise AssertionError("leftover test processes (killed now): %s" % lines)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main(verbosity=2)
|
unittest.main(verbosity=2)
|
||||||
|
|||||||
Reference in New Issue
Block a user