reliquary/docs/workpad/reports/2026-10-15-22-spire-vs-superpowers-plan-and-build-experiment.md

# Experiment: Spire vs. superpowers, planning and building the same change

**Date**: 2026-10-15
**Model**: Opus 5.5, model id `claude-opus-5-5` (author of this report)
**Method**: read-only synthesis of one session's journal, the experiment folder (plans, run logs, reviews, judge reports, audits, test summaries) and the subagent transcripts. Token totals were recomputed from the transcripts where the audit snapshot was incomplete. Nothing was run against the codebase for this report.
**Anonymization**: the source is a private Django codebase. Product, domain and tool names are replaced by generic terms. Agent, plugin and model names are kept.

## Summary

- **Plans.** A blind judge preferred the superpowers plan (B) over the raw Spire plan (A). After the Spire attacker revised A into a build plan (C), the judge preferred C over B, by a narrow margin. B was ten times longer than A or C, and most of the extra length was code that checked out against the repository. Only B found a documented layering rule: "a repository never calls another repository".
- **Builds.** Spire (plan C in the Athanor, a fresh Opus reviewer per phase) produced the implementation the blind judge chose to merge. Superpowers (plan B with subagent-driven-development) was correct on visibility. It had two Important gaps: two permission queries added on four routes and never recorded, and a parity assertion deleted where it should have been pinned. Superpowers scored better on convention fidelity and produced less code.
- **Where Spire's edge came from.** Two things: characterization pins written before any behaviour flipped, including route-level query counts (this structure came from the attacker), and an independent reviewer per phase. The decisive catches came from comparing old and new SQL against those count pins. Plan code had nothing to do with them.
- **Thesis "code in the plan adds nothing to the outcome".** Supported here, with n = 1 and confounded by the review structure.
- **Rule discovery.** Partly luck: one `cat` of a nested `CLAUDE.md`. Partly the wording of the instruction: "docs they might need to check" versus "read the code". Partly a systematic factor the planner brief can fix: agents read almost everything through Bash, which (to our understanding) bypasses Claude Code's on-demand loading of nested `CLAUDE.md` files.

## 1. Setup

### The change

The change was a mid-sized refactor in a large Django codebase, stacked on a parent branch that had introduced the pattern. The parent was itself an earlier change in a series that moves a module onto an architecture decision record introducing a repository/service/policy layer:

- Move create and update of **data-quality checks** out of a request-bound legacy module into the check service, using frozen input dataclasses (`Draft`, `Changes` with an `UNSET` sentinel for PATCH semantics). Delete the legacy module.
- Move the 10 raw ORM queries in the UI routes for data-quality checks onto repositories. Two new core repositories (workspace lookup, previewable extract), a column-info repository and a baseline repository.
- Switch list scoping from lenient to strict for **hierarchical workspaces (tenancy)**. Under lenient scoping, a path workspace the user cannot reach returns everything visible; under strict, it returns nothing.
- Fix a visibility bug: reads of **a per-source baseline record** were not scoped to the **data sources** the user can see.
- Put the route folder under the semgrep rules that enforce the layer (no raw `.objects` outside repositories, no model writes outside services).

A public management API endpoint, planned for a later change, will be built on the new service signatures, so those signatures matter.

### The shared design

A brainstorm with the user (plain Claude Code with superpowers `brainstorming`) produced a design with seven decisions, D1 to D7, plus behaviour changes and follow-ups. Every planner got the same sanitized copy, with links to epic and prep documents removed. After planning, a **decisions addendum** settled the open points the planners had raised. Both builders received it, and it overrode their plans. Its seven items:

1. Semgrep scope: top-level route files only.
2. The workspace-change check uses the actor's visibility, pinned first and then flipped.
3. Explicit `null` in a PATCH means "unchanged", pinned first and then flipped.
4. Creator and updater are passed explicitly.
5. The public datasource-linking method is replaced by a private one.
6. The baseline repository must not call another repository.
7. Stale docs stay untouched.

Item 6 matters for reading the results: the repository rule that only one planner found was handed to **both** builders. The planning miss therefore did not carry into either implementation.

### Isolation protocol

- **Sealed inputs.** Each agent had its own folder with an `input/` subfolder holding copies of the design, a branch-setup note and, for builders and the attacker, the plan it was given. The brief told each agent not to read the ticket folder beyond its inputs.
- **Separate worktrees** for the two builds, created with the team's worktree tool. Each had its own app database, its own test databases (one per xdist worker), its own Redis pair, its own ports and its own run lock. The Postgres server was shared.
- **Read rules.** No reads of the other run's worktree or build folder, no branch switches, no stash, no `rm -rf`, no push.
- **Transcript audit with a positive control.** A script went over every `tool_use` block of every attributed subagent transcript and checked paths, commands, patterns and globs against the allowed set. As a positive control, the checker was run against each run with the other run's allowed set: it flagged 283 and 407 calls, so it does fire when the rules are wrong.
  - Planning: all four agents clean. One grey case: the superpowers planner ran a repository-wide `grep -rn` that included `.worktrees/` and filtered it out with `grep -v` before reading.
  - Builds: no hard-rule violation on either side. Minor and grey items: Spire read one local tooling file from the main checkout. Both runs used the main checkout's virtualenv binaries (a shared-venv worktree has none of its own). Both used the harness scratchpad. A superpowers child read a script in the tooling repository.

### Blind judges

- **Plan judge** (Fable 5.1). Plans were anonymized as A (Spire planner), B (superpowers `writing-plans`) and C (the Spire attacker's revision of A). Skill and model header lines were stripped. C was described only as "a revision of A by an agent that checked it against the code". The mapping was kept outside the judge's read area. The judge verified claims by reading the repository at the shared base commit.
- **Implementation judge** (Fable 5.1). Inputs: the design, the addendum, `commits.txt`, `stat.txt` and the full diff against the base for each implementation, anonymized as impl-1 (superpowers) and impl-2 (Spire), plus a test-results summary. It read the base code from the main checkout and was forbidden from opening the worktrees. A leak scan of the inputs came back clean.

### Models per role

| Role | Spire | superpowers |
|---|---|---|
| Planner | `spire:planner`, Opus 5.5 (1M context) | general agent running `writing-plans`, Opus 5.5 |
| Plan revision | `spire:attacker`, Opus 5.5 (1M) | none |
| Build controller | `spire:athanor`, Opus 5.5 (1M) | general agent running `subagent-driven-development`, Opus 5.5 |
| Implementers | 4 Sonnet 5.5 children (B1.2, B1.3, B2.2, B2.3); B1.1, B1.4, B2.1 built inline by the Athanor on Opus | 8 implementers: Sonnet for tasks 1, 2, 5, 8+9 and the fix wave; Opus for tasks 3, 4, 6, 7 |
| Reviews | fresh `spire:reviewer` per phase and per re-check, all Opus 5.5 (4 agents, dispatched by the root session) | Sonnet task reviewer per task (8), Opus final reviewer, Sonnet scoped re-review |
| Judges, audits, helpers | Fable 5.1 judges; Sonnet audit and failure investigator; Haiku for worktree creation and the final test runs | same |

The asymmetry in review models (Opus phase reviewers vs. Sonnet task reviewers) is a confound. It follows each method's defaults.

## 2. Planning phase

### What each planner produced

| | Plan A (Spire planner) | Plan B (superpowers) | Plan C (Spire attacker) |
|---|---|---|---|
| Lines / words | 256 / 3,349 | 2,675 / 10,331 | 253 / 3,960 |
| Lines inside code blocks | 26 | 1,917 (72 %) | 26 |
| Structure | 6 phases (0 to 5), about 11 tasks; contracts for inputs, service methods, exceptions and repository methods; invariants; scope fence | 9 TDD tasks with full test and production code, file map, "Global Constraints", "Review Focus", self-review | 2 phases, 7 blocks with owner model, skills, gate and "what may stay broken"; contracts kept from A; U1 to U5 user decisions; risk list; 16 attack findings |
| Design issues raised | 8 | 9 | A's 8, plus 16 revision findings |
| Wall time | 8.2 min | 18.8 min | 8.6 min (after A) |
| Tool calls (Bash / Read / Skill) | 56 (51 / 4 / 0) | 79 (69 / 3 / 4) | 55 (50 / 4 / 0) |
| Tokens, all four columns | 5.31 M | 10.35 M | 5.93 M |
| Tokens, excluding cache reads | 210 k | 658 k | 220 k |

Spire's sequential planning (A, then C) took 16.8 min, 11.25 M tokens in all and 429 k excluding cache reads. Superpowers took 18.8 min, 10.35 M and 658 k. The harness reported 164 k, 174 k and 292 k at completion; those are not sums over the transcript, and the table uses recomputed sums.

The Spire planner and attacker have no Skill tool (their `tools:` list omits it), so they read the project skills' `SKILL.md` files with `cat`. The superpowers planner invoked `writing-plans` and three project skills through the Skill tool. None of those skills contains the repository rule.

### Design issues found

All of these were confirmed real by the judge unless marked otherwise.

| Issue | A | B | C |
|---|---|---|---|
| D7's recursive glob pulls subfolders with 16 raw queries and 2 writes into an ERROR rule, so the acceptance criterion cannot pass | yes | yes | yes, raised as a user decision |
| Create path authorizes "change" (latent: add and change share one predicate) | yes | no | yes |
| Actor exposes no user; creator and updater need explicit parameters | yes | yes | yes |
| "Existing tests check the error bodies" is false; several routes have no tests | yes | yes | yes |
| Actor-scoped workspace-change check differs from today's superuser check | yes | yes (also changes which names are listed) | yes, pinned before the flip |
| 404 bodies differ between `Http404` and the UI handler | yes | yes | yes |
| Baseline model has no protected manager; `policy.scope` raises | yes | yes | yes |
| **A repository never calls another repository** | no | **yes** | no |
| More stale references than the design lists | yes | no | yes |
| Update name has no max length | yes | yes | yes |
| PATCH `null` semantics undefined (today a 500 or a 404) | yes | yes | yes, pinned first |
| Update transaction adds SAVEPOINT queries; an existing count pin moves | no | yes | yes |
| Create query count drops | no | no | yes |
| A request-level workspace override stops working on list routes | no | no | yes |
| `@paginate` over `@inject` may misbind | yes (wrong) | no | yes (wrong; a precedent exists) |
| A's blocked-workspace pin cannot be built with the named fixtures | n/a | n/a | yes |
| A's between-task import gate fails without `django.setup()` | n/a | n/a | yes |
| Service-side `.values_list` on a repository queryset breaks the layer checklist | n/a | (B does this) | yes |

A design claim that none of the three plans challenged: "the app-search parity test stays green". Both builders later found it false.

### The blind plan judge, two rounds

| Criterion (1 to 5) | A | B | C |
|---|---|---|---|
| Fidelity to the code | 4 | 5 | 5 |
| Coverage of the design | 4 | 4 | 4 |
| Design issues found | 4 | 5 | 5 |
| Commit order and gates | 3 | 4 | 5 |
| Test strategy | 4 | 4 | 5 |
| Executability | 3 | 5 | 4 |

- **Round 1, A vs B: B wins.** B's length is verified code: imports, fixtures, constants and moved query bodies all match the repository. B alone found the repository rule. A bundled four moves into one commit, flipped a behaviour change without a pin, had an import gate that cannot pass, and had an unbuildable pin.
- **Round 2, C vs B: C wins narrowly.** C fixed A's gating and pin problems and won on commit discipline and pin-before-flip. It lost on the repository rule, which C still breaks, and on one over-cautious risk. C has no code, so the implementer writes what B already wrote.
- **Judge's merge recommendation:** C's skeleton and gates, B's verified code lifted in where the contracts agree, and B's baseline-repository expression.

### What the attacker bought

The judge verified 15 of 16 revision findings with correct line numbers. Its split:

- **Real corrections.**
  - A buildable blocked-workspace pin using a root-level admin, plus a separate pin recording behaviour change 3 before it flips. The judge called this the most valuable change.
  - A's bundled commit split into three, with behaviour changes 3 and 4 pinned and then flipped in the update commit.
  - The broken import gate dropped.
  - The "add without change" test marked impossible with real users (use a stub actor).
  - Unordered results asserted as sets.
  - The existing count pin placed explicitly inside the allowance.
  - The public linking method removed.
  - Semgrep run from the first service commit, since the pre-commit hook would have blocked it.
  - The API spec check made a gate.
  - One repository method for names and coordinates instead of service-side shaping.
  - Three small, correct notes.
- **Cosmetic or neutral.** The semgrep default re-framed as a user decision. One more stale reference. Two phases instead of six. Owner and model stamps.
- **Not fixed.** The repository rule. One risk that a code search would have removed.
- **Not credited by the judge, but decisive in the build.** The attacker added a fallback clause to the repository contract: "if `policy.scope` changes the SQL or the query count of the intersection or aggregate queries, fall back to `actor.protect(...)` and flag it." This clause is in C only. Section 3 shows that it is what turned a measured +2 queries into a fix rather than a shrug.

### Why only the superpowers planner found the rule

The rule lives in one place: the core services package's `CLAUDE.md`, line 24. It is not in the core repositories package's `CLAUDE.md`, and not in the repository-layer skill. The architecture decision record has a related line but was opened by nobody. The full evidence is in section 6. In short: both planners listed the services directory, but only the superpowers planner then printed its `CLAUDE.md`.

## 3. Implementation phase

### Spire

Plan C plus the addendum went into the Athanor, with one phase per turn. After each phase the root session dispatched a fresh `spire:reviewer`. Then came one fix pass by the Athanor and a fresh reviewer re-checking only the named findings.

**Phase 1: pins and the new layer.** 6 commits, 12 files, +2,305/−3; 317 tests passed.

- B1.2 (Sonnet) and B1.3 (Sonnet) ran in parallel on disjoint files. The children did not commit (index-lock races) or run pytest; the Athanor did both.
- B1.1 (Athanor) added 35 characterization tests on unchanged production code:
  - full error bodies;
  - lenient pins for the workspace anchor and invisible-baseline cases, flipped later;
  - null-PATCH pins;
  - **observed route query counts**: list 8, filters 4, create 16, blocked workspace change 15, three column routes 4/4/5, baseline GET 4, preview 4, PUT 11.
- B1.4 (Athanor) built the service.
- The public linking method was kept until Phase 2, because its only caller still served production in a "no behaviour change" phase.

**Phase 1 review** (Opus, 8 min): 0 Critical, 1 Important, 6 Minor.

- **I1**: the common-source column route had no non-superuser visibility pin and no count pin before its move. The gap came from the plan's B1.1 list.
- The reviewer confirmed that the moved SQL matches the old routes and that the lenient pins will fail when flipped.
- **Fix pass**: 6 commits, one per finding. Phase total 341 passed.
- **Re-check** (Opus, 3 min): I1 resolved, no weakened tests.
- One finding (M5) was logged without fixing: a red commit sits in the history because the Athanor committed a child's block before calibrating its count. History was not rewritten.

**Phase 2: routes onto the layer, enforcement, delivery.** 10 commits, 11 files, +220/−593; 370 passed. Ruff, semgrep (rule tests 6/6) and the spec check were all clean.

- **B2.1** (Athanor): writes moved onto the service in three commits. Behaviour changes 3 and 4 flipped in the update commit. Old and new SQL were compared for each count change:

| Route | Before | After | Cause |
|---|---|---|---|
| Create | 16 | 14 | No lazy load of the workspace; non-polymorphic id read |
| Baseline PUT | 11 | 10 | |
| Update with linking | 13 | 15 | SAVEPOINT and RELEASE |
| Blocked workspace change | 15 | 13 | Real queries 15 → 10, plus three transaction statements (corrected after review) |

- **B2.2** (Sonnet child): reads moved onto the repositories.
  - The child's own old-vs-new SQL comparison showed the four column routes going from 4/4/4/5 to 6/6/6/7. The generated default policy's `can_*` flags made Django's `ModelBackend` load the permission cache, because the "add" permission for that model is `always_deny` in the rules backend.
  - The Athanor applied the attacker's fallback clause (`actor.protect`) and the counts returned to baseline.
  - The child weakened the app-search parity test (dropped the sibling case). The Athanor replaced that with an explicit pin: the strict list is empty, and lenient search still finds the user's own checks.
- **B2.3** (Sonnet): semgrep scope and changeset.

**Phase 2 review** (Opus, 11 min): 0 Critical, 1 Important, 3 Minor.

- **I1**: the recorded cause of list +1 and filters +1 ("anchor lookup") was wrong. The old path also loaded the anchor. The extra query is the `auth_group` lookup that `policy.scope`'s flags trigger through `rules` on routes with no prior permission check.
- **M1**: the dropped request-level override belongs in the PR body.
- **M2**: changeset wording.
- **M3**: count explanation.
- The reviewer also produced a positive semgrep proof in a scratch copy: the top-level file is caught, a subfolder is not. The builder had none.

**Fix pass** (2 commits): after checking that no schema, generated UI spec or route consumes `can_*`, the list reads use `actor.protect`, and the counts are back to 8 and 4. The changeset wording was fixed too.

**Re-check** (Opus, 4 min): I1 resolved, no weakened tests. Two new Minors, logged only:

- N1: one PR-note sentence is wrong.
- N2: the visible-checks helper should call `self.policy.visible`, not `actor.protect`, so a future policy is not skipped.

The re-check also noticed that the builder's changeset-check log was 0 bytes, so its "exits 0" had no evidence. The reviewer reran it and it passed.

**Spire totals:** 24 commits, 19 files, +2,613/−512, most of it tests.

### superpowers

Plan B plus the addendum went to `subagent-driven-development`. It ran a pre-flight scan of task pairs against the addendum, which produced rulings R1 to R8 (later R9 to R11). After that, each task got a fresh implementer, a task review (spec compliance, then code quality), and at the end an Opus whole-branch review, one fix dispatch and a scoped re-review. The run stopped before `finishing-a-development-branch`.

| Task | Implementer | Result |
|---|---|---|
| 1 Characterization tests | Sonnet | Pins incl. null-PATCH and invisible-assigned-source; counts list 8, filters 4, create 16, baseline save 14. Worked around a pre-existing row in the fresh test DB. |
| 2 Core repositories | Sonnet | Counts 1/1 as planned |
| 3 Create/update in service | Opus | R9: the public linking method kept until its caller goes |
| 4 Routes onto service, delete legacy module | Opus | Flipped pins; update 13→15, create 16→14 (cause inferred, not measured) |
| 5 Strict list scoping | Sonnet | List 8→9, filters 4→5 recorded as "anchor lookup" (wrong cause). App-search sibling case dropped from the parity test (R10). |
| 6 Column-info repository | Opus | 2 commits; the reviewer "compared query by query" and found no behaviour difference. Repository tests pin 9 queries on a cold actor. **No route-level count pins.** |
| 7 Scoped baseline reads | Opus | Baseline save 14→8 |
| 8+9 Semgrep scope, changeset | Sonnet (one dispatch) | Planted-probe check of the glob; the CI semgrep over the whole tree failed on 3 findings. The implementer blamed all 3 on the parent branch; `git blame` showed one came from this branch. |

- **Task reviews:** all nine passed with no Critical or Important findings. About 18 Minor findings were deferred to the ledger.
- **Final review** (Opus): "ready with fixes". There were no Critical or Important findings beyond the semgrep issue, and every deferred Minor was triaged as "leave" except the changeset wording.
- **Fix wave**: 3 commits (the semgrep fix, which also touched the parent branch's lines; changeset reorder; reuse of a shared JSON helper). The scoped re-review was clean.
- **Results:** 526 passed and 3 skipped on its affected set; ruff, semgrep and the spec check were clean.
- **Totals:** 13 commits, 17 files, +1,946/−523.

### Same final test run on both

The same command (three test trees, `-p no:randomly`, 7 xdist workers) ran in parallel on both worktrees.

| | Passed | Failed | Skipped / xfail / xpass | Time |
|---|---|---|---|---|
| Spire | 7,569 | 1 | 9 / 3 / 1 | 1,008.9 s |
| superpowers | 7,520 | 1 | 9 / 3 / 1 | 1,002.8 s |

The one failure is the same test on both, an environment-dependent instance-settings test unrelated to the change. It was ignored and the reruns were cancelled. Two full suites in parallel on two worktrees ran without collisions.

## 4. Results

### Blind implementation judge: merge Spire

| Criterion (1 to 5) | impl-1 superpowers | impl-2 Spire |
|---|---|---|
| Correctness and visibility | 4 | 5 |
| Fidelity to design and addendum | 4 | 4 |
| Behaviour changes deliberate and pinned | 3 | 5 |
| Tests | 3 | 5 |
| Architecture conventions | **4** | 3 |
| Commit history | 3 | 4 |
| Code quality and size | 4 | 4 |

Both implementations are correct on visibility, with no leak. Response bodies, status codes and visibility semantics are identical; the differences are query counts, transaction shape and what is recorded.

**Superpowers' two Important gaps:**

1. **Two queries added on the four column routes and not recorded.** The cause is `policy.scope` on the column-info model, where an `always_deny` "add" permission falls through to `ModelBackend`, which loads the permission cache. There was no route pin, and the design required count changes to be recorded.
2. **App-search parity assertion removed for the sibling case** instead of pinned. A regression in either direction is no longer caught.

Its Minors: the wrong cause for list/filters +1; repository count tests that measure actor warm-up (9 queries) rather than the method; a redundant second query and a dead guard in the workspace-change check; a no-op test edit; missing commit bodies where behaviour flips; no rollback test.

**Spire's findings were all Minor:**

- Two documented convention deviations: the list helper bypasses the policy, and the baseline write uses `Model.objects` in the service under `nosemgrep`.
- A new core repository method where the design named an existing one (justified by one query fewer).
- Comments in tests, against a project rule.
- One typing cast in the wrong place.
- A query expression duplicated verbatim.
- Commit churn: 24 commits, including self-corrections.

**Where superpowers was better:**

- **Convention fidelity.** It followed the letter: `policy.scope` everywhere and writes through a repository queryset. The judge noted that the letter is exactly what cost the extra queries.
- **A one-query workspace-change check.** Spire also has one query, but it resolves the payload through a second lookup. Superpowers compares UUID sets with no payload query, and the judge's merge item 1 ports this into Spire.
- **Smaller code.** A shared helper replaces a duplicated expression, the existing selection value object is reused, the typing casts are cleaner, and there is one extra `null` pin.

### Time and tokens

Tokens are summed over assistant messages and deduplicated by message id, across all four usage columns. "Excl. cache read" is input plus cache creation plus output.

| | Agents | Wall time | Tokens, all | Tokens, excl. cache read |
|---|---|---|---|---|
| Spire builder (Athanor + 4 children) | 5 | Athanor active ≈ 97 min (32 + 8 + 46 + 11) | 61.89 M | 2.08 M |
| Spire phase reviewers (2 reviews, 2 re-checks) | 4 | 26 min (8.2 + 3.2 + 10.7 + 3.9) | 16.15 M | 0.61 M |
| **Spire, total** | 9 | ≈ 119 min elapsed, start to last re-check | **78.05 M** | **2.69 M** |
| superpowers, without its reviewers | 10 | | 44.94 M | 3.15 M |
| superpowers reviewers (8 task, final, re-review) | 10 | | 9.63 M | 0.97 M |
| **superpowers, total** | 20 | 108 min (controller, start to end) | **54.57 M** | **4.12 M** |

Accounting caveats:

- The build audit was a snapshot taken while Spire's last fix pass was still running, and it excluded Spire's reviewers, which the root session dispatched as separate agents. It reported Spire at 61.0 M / 2.08 M against superpowers at 54.6 M / 4.17 M. The table above adds the final fix pass and the four reviewer transcripts.
- Spire's total is dominated by cache reads in the Athanor's 1M-context session: 53 M of its 55.6 M. Excluding cache reads, Spire used about 35 % fewer tokens than superpowers. Including them, it used about 43 % more. The audit cannot turn this into cost without per-column pricing.
- The root session's own orchestration (Igor dispatching reviewers, writing briefs and the addendum) is counted on neither side.
- The Athanor did not record tokens for the blocks it built inline.

### Process incidents

1. **Transient failures on a fresh test DB.**
   - Spire's first pytest on its new worktree had 16 failures. 15 never recurred; the 16th was a real count bug, fixed.
   - An investigator ruled out cross-worktree interference: no shared DB, Redis, tmp or lock, ten later overlapping runs passed, and the Postgres log was clean.
   - Most likely cause (inferred, not proven): xdist worker DBs are template copies whose `public` schema still holds dev rows until the first teardown flush. The failing tests were the ones that read all rows or reach the root workspace.
   - Superpowers saw the same effect on its first run (a pre-existing column row) and renamed its fixture to work around it.
   - A tooling request was filed.
2. **Worktree tooling gaps.**
   - The tool derived the worktree slug from the ticket key, so a second worktree for the same change was refused. It also had no base option and fetched from origin for a new branch.
   - The tool's owner added `--slug` and `--base` during the session.
   - The tool's "terminate connections to the main DB" confirmation did not appear (stdin was closed), so whether connections were terminated is unknown.
   - Shared-venv worktrees have no `.venv`, so both builders ran the main checkout's binaries.
   - Separately, the user's local "one test runner per machine" rule was stale: it predated per-worktree test DBs, and parallel runs worked.
3. **An unexplained `.venv`.** An empty virtualenv (Python 3.13.3, prompt named after the worktree) appeared in the Spire worktree root mid-run. The builder did not create it. While it existed, the local ruff wrapper failed with "Failed to spawn"; tests were unaffected. Its origin is unknown.
4. **Test output cut by `tail`.** The builder piped the first failing run through `tail -60`, which cut every assertion message, so the cause of the 16 failures could only be inferred. New rule for both builders: pytest output goes to files in `logs/`, and a fresh-worktree first-run failure gets one rerun. Superpowers applied the rule from task 5 on.
5. **Gates that checked nothing.**
   - The local semgrep (1.156.0) scanned 0 files when given explicit file paths, so a Phase 1 pre-commit gate may have checked nothing. The Phase 1 reviewer flagged this.
   - Spire's 0-byte changeset log backed a claim with no evidence.
   - Superpowers' planted-probe check and the Phase 2 reviewer's scratch-copy proof are the good patterns here.
6. **Cross-branch side effect.** Superpowers fixed two semgrep findings belonging to the parent branch. The parent's CI stays red until it gets the same fix, and a trivial rebase overlap is expected.

## 5. Analysis for Spire

### Where the edge came from

Following each decisive catch back to its source:

| Catch | Who caught it | What made it catchable |
|---|---|---|
| Column routes +2 (policy flags) | Spire B2.2 child, comparing old and new SQL; fixed by the Athanor | Route count pins from B1.1 (attacker's pins-before-flip block), plus the attacker's fallback clause, which made a count change a trigger for action |
| List/filters +1, wrong cause | Fresh Spire Phase 2 reviewer | An independent reviewer with an instruction to answer every flag and to trace scenarios; the builder had recorded a plausible but wrong cause |
| App-search sibling assertion | Spire Athanor, replacing its child's weakening | Athanor keeper: "a child's 'tests pass' is a claim: read the diff for loosened assertions" |
| Common-source route had no pin | Fresh Spire Phase 1 reviewer | Pins listed per route in the plan, so a missing route was visible |

Superpowers' per-task Sonnet reviewers and its final Opus reviewer passed all three of the costs above. Its task-6 reviewer even "compared query by query", but without a route-level count pin the compared SQL looked equivalent: the extra queries are permission loads outside the data query. Superpowers measured counts only where plan B told it to.

So the edge comes from three things, in this order:

1. Route-level count pins written on unchanged code before anything moved.
2. A rule that turns a count change into a decision.
3. A reviewer who did not build the phase.

All three are structural, and two of them came from the attacker, not the planner.

### Does "code in the plan adds nothing to the outcome" hold?

Supported, with n = 1. The evidence for it:

- B's 1,917 lines of code were verified correct against the repository, and its implementation still lost on the same dimensions where B's code carried a blind spot: `policy.scope` everywhere, no route pins for the column routes, and trust in the design's parity claim. The code transmitted the plan's assumptions verbatim, and the implementers and task reviewers checked conformance to that code.
- C contained no code and produced a better result. Its Sonnet children wrote the bodies without trouble, and their count guesses were right except for one warm-up issue.
- Planning cost was similar in wall time: 16.8 min for A plus C, 18.8 min for B. B used 1.5 times the non-cache tokens.

Confounds that keep it at "supported" rather than "shown":

- **Review structure.** Spire used Opus phase reviewers with a mandate to answer every flag. Superpowers used Sonnet task reviewers, per its model-selection guidance, and one Opus final review.
- **Pin discipline** came from the attacker. A code-bearing plan with the same pins-before-flip structure might do as well.
- **The Athanor built three of seven blocks itself** on a 1M-context Opus that had read the whole trunk.
- **The addendum** equalized the biggest planning difference (the repository rule) and forced pin-then-flip for behaviour changes 3 and 4 on both sides.
- **One change, one codebase, one run each.**

### What Spire missed

- **The convention rule.** Neither the planner nor the attacker read the services package's `CLAUDE.md`, and the attacker had the plan's baseline repository contract in front of it and still let it through. See section 6.
- **Plan-level gaps the attacker had to fix.** A had an unbuildable pin, a silent behaviour flip, a gate that cannot pass, and a bundled commit. The attacker caught all of them, but they are basic planner errors.
- **Gaps both planner and attacker left.** No common-source route pin (the Phase 1 reviewer caught it). The design's parity claim was never checked (the builder caught it). One risk was contradicted by an existing precedent.
- **Convention fidelity in the build.** The Athanor deviated from documented conventions twice for query-count reasons and documented each. The reviewers accepted the documentation as sufficient. The judge scored Spire 3 against 4 on conventions and called the deviations "worth a core-team conversation".
- **No Skill tool** for the planner, attacker and reviewer. All three read `SKILL.md` with `cat` and said so. The Athanor (tools `*`) invoked skills properly, 19 times.
- **Hygiene.** A red commit in history (children were told not to run tests; the Athanor committed before calibrating). A commit body with a wrong cause that stays, because history was not rewritten. A 0-byte evidence log. 24 commits where the judge thought 14 to 16 would do.

### Proposals

**Planner (`agents/planner.md`)**

1. Replace "Read the code the plan touches" with "Read the code **and the local conventions** the plan touches". For every directory the plan creates files in or changes, and every layer those files call into, open that directory's `CLAUDE.md` or `AGENTS.md` and any ADR it cites.
2. Require a short **Conventions** section in the plan: each rule the plan relies on or could break, with file and line. This makes a missed rule visible to the attacker.
3. Tell the planner to open convention files with the Read tool or an explicit `cat`, not to rely on directory listings. See section 6 for why Bash reads are not enough.
4. When the design asserts that an existing test "stays green" or "already checks X", read that test and say whether the claim holds.
5. For every query or route that moves, require a route-level visibility pin and a route-level count pin on unchanged code. Repository-level count tests hide per-request permission queries, because a warm actor has already loaded them.
6. Add `Skill` to `tools:`, or preload the project's routing skill through `skills:`. Reading `SKILL.md` with `cat` works, but it skips whatever the project wires to skill activation.

**Attacker (`agents/attacker.md`)**

1. Add a **conventions pass** to the attack: for each new class or method in the contracts, find the docs that govern its layer and check the contract against them. In this run the attacker read the repositories package's `CLAUDE.md` but not the services one.
2. Every risk on the risk list needs a precedent search. If the repository already does the risky thing somewhere, drop the risk and cite the precedent.
3. Keep, and make explicit in the procedure, what worked: pins before flips; a "what may stay broken" line per block; **fallback clauses with a measurable trigger** ("if the count of X changes, do Y and flag it"); tool invocations named exactly (C said "semgrep is not on PATH" without saying how to run it).
4. Add `Skill` to `tools:`.

**Athanor briefs (`agents/athanor.md`, `skills/implementation`)**

1. Pytest output goes to a file under the run's `logs/`, never through `tail` or `head`. After a fresh worktree or a migration, treat the first run as a warm-up and rerun once before judging a failure.
2. Children that do not run tests must have their block calibrated by the Athanor **before** the commit. Otherwise a red commit lands, as M5 did.
3. A count-change cause written into a commit body must come from an old-vs-new SQL diff saved to `logs/`, not from inference.
4. Evidence files must be non-empty. A gate with no output is not evidence.
5. Every enforcement gate (lint rule, semgrep scope) needs a positive control, such as a planted probe that must be caught.
6. Record tokens for inline blocks, so per-block cost is comparable to child blocks.
7. The fork agent type was not available. The Athanor built three blocks inline instead. The spawning reference should say what to do when forks are missing.

**Reviewer (`agents/reviewer.md`)**

1. Add `Skill` to `tools:`. All four reviewers noted its absence.
2. For every recorded count or behaviour change, verify the **stated cause**, not only the number. This is how I1 in Phase 2 was found; make it a rule, not luck.
3. A deviation from a documented convention is a finding (at least Minor) that names the convention file, even when the builder documented it. "Documented" means the reviewer knows about it, not that it is accepted.
4. Ask for route-level pins wherever only repository-level count tests exist.

**Igor / cycle**

1. Token and time reports for a build must include the phase reviewers that Igor dispatches. Otherwise Spire looks cheaper than it is when compared with methods that count their reviewers internally.
2. Comparisons need sealed inputs and a transcript audit with a positive control. Briefs alone do not prove isolation.

## 6. The rule-discovery question

Why did only the superpowers planner find "a repository never calls another repository"?

**What the instructions say.**

- Superpowers `writing-plans` asks the planner to document "which files to touch for each task, code, testing, **docs they might need to check**, how to test it", and to "**follow established patterns**" in existing codebases. Both phrases point at conventions and documentation.
- The Spire planner is told: "**Read the code the plan touches** before you write it." That points at code, not docs.

**What the transcripts show.**

- **Both planners listed the services directory** with `ls` and so saw its `CLAUDE.md` in the output.
  - The Spire planner did this in one compound command (call 11) that also `cat`-ed the **repositories** package's `CLAUDE.md`. It never opened the services one.
  - The superpowers planner listed `core/services` in call 45, while reading the exception base classes and the public-API error mapping. Its very next call (46) was `cat` of the services `CLAUDE.md`. Line 24 of that file holds the rule.
- **Neither opened the architecture decision record.** The attacker only `grep`-ed it for stale references to the old module. The attacker read the repositories `CLAUDE.md` and never listed or read the services one.
- **Both read code almost entirely through Bash.**

| Agent | Bash | Read | Skill |
|---|---|---|---|
| Spire planner | 51 | 4 | 0 |
| superpowers planner | 69 | 3 | 4 |
| Spire attacker | 50 | 4 | 0 |

  Their Read calls went to the input files, one route file and one YAML file.
- **The rule is in neither skill set's material**: not in the repository-layer skill, not in the repositories `CLAUDE.md`, not in `writing-plans`. The documentation gap is real: it sits on the services side, where repository authors do not look.

**The harness factor.** To our understanding, Claude Code loads a subdirectory's `CLAUDE.md` on demand when the Read tool opens a file in that directory. Reads through Bash (`cat`, `sed -n`, `grep`) do not trigger this. A planner that reads `core/repositories/*.py` with `cat` therefore never gets the nested `CLAUDE.md` files of the directories it works in, unless it opens them deliberately. This describes how the harness behaves as we understand it; it was not verified in this run. The session ran in bypass-permissions mode, whose system prompt explicitly encourages Bash for reading ("read files with cat, head, or sed -n … instead of the dedicated Read … tools"). We assume, without having verified it, that the subagents saw the same nudge. The tool mix is consistent with that.

**Conclusion.** The discovery was:

- **partly luck**: a single `cat` call, made right after an `ls` while the planner was reading error conventions next door;
- **partly a nudge from the instruction wording**: "docs they might need to check" and "follow established patterns" invite opening a `CLAUDE.md`, while "read the code the plan touches" does not;
- **partly a systematic factor that the planner brief can fix**: Bash-heavy reading bypasses nested `CLAUDE.md` loading, so the brief has to make reading local convention files an explicit step (planner proposals 1 to 3, attacker proposal 1).

The journal records the exploration difference as 79 against 56 tool calls. Breadth helps, but the transcript shows the deciding moment was one specific read, not volume.

## Gaps and open points

- **Commit counts.** The implementation judge's text says 14 (superpowers) and 22 (Spire); its own `commits.txt` inputs list 13 and 24. This report uses 13 and 24.
- **Phase 2 fix-pass duration.** The journal says 5 min; the run log says 20:58 to 21:09 local, which is 11 min. This report uses 11 min.
- **Harness-reported planner tokens.** The numbers in the journal (164 k, 174 k, 292 k) are not sums over the transcripts, and what they measure is unclear. This report uses recomputed sums.
- **Bypass-mode nudge for subagents.** That subagents saw the bypass-mode Bash nudge, and that Bash reads bypass nested `CLAUDE.md` loading, are both assumptions, not observations from this run.
- **Unexplained `.venv`.** Its origin is unknown.
- **Cause of the first-run failures.** Inferred, not reproduced. The assertion text was lost to `tail`.
- **Pricing.** Cache reads, cache writes and output tokens are priced differently, and this report does not compute cost.