dirigence/reliquary
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

📝 docs(workpad): Spire temp-file clutter report

Browse Source
This commit is contained in:
g4borg
2026-10-20 11:51:02 +02:00
parent 544b69ccb7
commit 94d085f04f
1 changed files with 141 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/workpad/reports/2026-10-20-spire-temp-file-clutter.md
+141
View File
@@ -0,0 +1,141 @@
# Spire temp-file clutter in the workpad
Date: 2026-10-20. Trigger: Spire runs on dirigent ticket #32 (2026-10-19/20) left about 56 MB of untracked throwaway files in `docs/workpad/reports/`, mixed in with real reports. Those files have been deleted since, so this report works from the plugin sources, from the reports that link to them, and from the coordinator's inventory.
Sources read: the installed Spire 0.3.0 (`~/.claude/plugins/cache/reliquary/spire/0.3.0/`). For every file quoted below I checked that it is byte-identical to `origin/main` of reliquary. Note that the local `main` checkout is 46 commits behind and has no `gremlins` skill yet. Also read: superpowers 7.0.0 and 6.4.1 (`~/.claude/plugins/cache/claude-plugins-official/superpowers/`), and the dirigent workpad and `.superpowers/` directory. No web pages were fetched.
## 1. What was left behind, and who asked for it
| Leftover (dirigent) | Size | Written because of |
|---|---|---|
| `…-run-phase-{a,b,c}-gremlins/raw/*.jsonl`, `raw/*.err` | 37 MB, 33 transcripts | the gremlins helper code (no agent decides this) |
| `…-gremlins/briefs/`, `jobs/`, `bases/`, `notes/`, `logs/`, `run.json`, `preamble.md`, `mcp-empty.json`, `ledger.jsonl` | about 110 to 160 files per run folder | the gremlins helper and preamble |
| `~250 .txt` gate and test logs | 3.9 MB | the gremlins preamble and spawn template |
| `…-phase-*-walk/`, `…-rewalk/`, `2026-10-20-ticket-32-c-d-walk/` screenshots | 207 PNG, 18 MB | an ad hoc Igor brief (no Spire skill covers walks) |
| walk scripts: phase-a-walk 5 `.py`, phase-a-rewalk 13 `.py`, phase-b-walk 5 `.py` plus `__pycache__/*.pyc` plus `state.json`, phase-c-walk 8 `.py` plus `.pyc` plus `state.json` | small | the walker agent itself; nothing in Spire says where scripts go |
### 1a. Gremlins: hard-coded state dir next to the run log, inside `reports/`
- `skills/implementation/references/run-log.md:3` puts the run log in the workpad's `reports/`: "The run log is the record of a build: one file per build plan, in the workpad's `reports/` folder".
- `skills/gremlins/grm/store.py:17-22` derives the state dir from that path: `"""<x>/run.md -> <x>/run-gremlins (a sibling of the run log)."""` … `return Path(p[:-3] + "-gremlins")`. So the state dir always lands in `reports/`, and no option changes that.
- `skills/gremlins/grm/store.py:10`: `SUBDIRS = ("bases", "jobs", "raw", "notes", "logs", "briefs")`, created by `init_state` (`:25-30`).
- `skills/gremlins/grm/cli.py:134-135` writes the full stream for every call to `state / "raw" / (job + ".jsonl")` and the full prompt to `state / "briefs" / (job + ".md")`.
- `skills/gremlins/SKILL.md:42-50` documents the layout: "## State dir — `<run log without .md>-gremlins/`, next to the run log", including "`raw/<J>.jsonl` (the stream) and `raw/<J>.err`; `briefs/<J>.md` (the full prompt sent)" and "`notes/<J>.md` (the gremlins' handoff notes); `logs/` (their gate and test output)".
- `skills/gremlins/preamble.md:8`: "Write gate and test output to `.txt` files under the state dir's `logs/` folder (the brief names the state dir)."
- `skills/gremlins/templates/spawn.md:14-15`: "Write your handoff note to `${note_path}`." / "Write gate and test output to `.txt` files under `${logs_dir}`."
- No Spire file says anything about cleanup, retention or gitignore for this dir. The dirigent run logs mark it by hand: `2026-10-19-ticket-32-run-phase-a.md:13`: "Gremlin state: `docs/workpad/reports/2026-10-19-ticket-32-run-phase-a-gremlins/` (not committed)". They then link into it, e.g. `:75` "Logs: `…-gremlins/logs/A2-*.txt`" and `:388` "logs `…-gremlins/logs/phase/`".
`raw/` is mostly a duplicate. The helper launches `claude -p --output-format stream-json` with `--session-id` or `--resume … --fork-session` (`grm/launch.py:125,173,177`), so Claude Code also keeps each session's transcript in its own store under `~/.claude/projects/<cwd-slug>/`. That has to be the case, because `--resume` only works on a stored session. The stream-json copy is a second copy, in a slightly different format. I did not check whether the two copies match line for line.
### 1b. Athanor and implementation: trunk notes "next to the run log"
- `agents/athanor.md:32`: "Write short trunk notes next to the run log so fresh agents, and a successor Athanor, can start from them."
- `agents/athanor.md:60` and `skills/igor/SKILL.md:114`: "Every subagent gets an explicit output file path in the workpad." This means every artifact an agent produces gets a workpad path by default, and no other place is defined.
- `skills/implementation/SKILL.md:72`: "the run log and review reports to `reports/`."
Trunk notes are real documents, so they belong in the workpad. The rule matters here because it is where the "everything next to the run log" habit starts.
### 1c. Workpad skill: it pulls scratch into the workpad
- `skills/workpad/SKILL.md:8`: "It is authoritative for placement: if another skill suggests its own scratch directory (a plugin-named folder, a sibling `notes/` dir, the repo root), use the workpad instead."
- `skills/workpad/SKILL.md:3` (description): it fires "when another skill is about to drop a markdown file somewhere ad-hoc (e.g. … the repo root, `/tmp`, …). If a workpad exists, the workpad wins."
- `skills/workpad/SKILL.md:78`: dated subfolders for "multiple related files". This is the pattern the walk folders followed (`reports/<date>-<slug>/`).
- `skills/workpad/SKILL.md:41`: `tickets/` is described as "per-ticket scratch space". So the word "scratch" already means something tracked in the workpad.
As written, the skill is about markdown documents. Its wording ("use the workpad instead" of a scratch directory, and "the workpad wins" over `/tmp`) still steers an agent away from any scratch location it might have chosen. It has no category for non-document artifacts (logs, transcripts, images, scripts, bytecode).
### 1d. Reviewer, attacker, planner and walks
`agents/reviewer.md`, `agents/attacker.md` and `agents/planner.md` contain nothing about screenshots, logs or walks. Spire 0.3.0 has no browser-walk skill or agent at all. The walks came from Igor briefs in dirigent, for example `docs/workpad/journals/2026-10-19-11-ticket-32-import-screen.md:34`: "Sonnet UI walker (`just serve` on dev dirs + Playwright, screenshots) → `docs/workpad/reports/2026-10-19-ticket-32-phase-a-walk.md`". At `:36` it adds "(+ ~70 screenshots, untracked)". The walker put screenshots and scripts next to its report, following the workpad subfolder rule. See `reports/2026-10-19-ticket-32-phase-a-walk.md:4`: "Screenshots: `2026-10-19-ticket-32-phase-a-walk/` … Scripts: `2026-10-19-ticket-32-phase-a-walk/scripts/`." The findings tables link each bug to its PNG, e.g. `:21` `[10-sort-name.png](2026-10-19-ticket-32-phase-a-walk/10-sort-name.png)`. Those links are now dead.
Counter-example in the same project: the CSS pipeline plan sent its harness output out of the repo. `docs/workpad/plans/2026-10-19-css-pipeline-plan.md:26`: "The harness writes its runs, traces and screenshots under `~/.cache/dirigent-tmp/web-measure/`, never into the repo." It worked because the plan said so explicitly. Nothing in Spire makes this the default.
### 1e. Python bytecode in the workpad
The walk scripts imported each other or a shared helper. Python then wrote `__pycache__/*.pyc` beside them. Python always does this unless `PYTHONDONTWRITEBYTECODE=1` is set or `-B` is passed. Reliquary's `.gitignore:13` ignores `__pycache__/`. Dirigent does not (its `.gitignore` has `**/*.tmp` but no pycache rule), so the `.pyc` files there were untracked and visible. The installed gremlins skill also ships its own `grm/__pycache__/` in the plugin cache. That is harmless, but it is the same mechanism.
### 1f. What in the gremlin and walk folders is worth keeping
| Artifact | Kind | Worth keeping after the phase review? |
|---|---|---|
| `ledger.jsonl` | summary: model, tokens, cost per call | **Yes.** Small, and it is the cost record. It could also be rendered into the run log |
| `notes/<J>.md` | handoff notes | **Yes, as text.** The Athanor copies them into the run log (`agents/athanor.md:40`), so the file itself is redundant once copied |
| `run.json`, `bases/*.json` | launch fingerprint, base session ids | Short-lived: needed while forks may still run, useless after the phase |
| `jobs/*.json` | per-call status | Short-lived |
| `briefs/<J>.md` | full prompt sent (template, delta, brief) | Short-lived. Useful for debugging a bad gremlin, and the brief part could be rebuilt from the run log |
| `raw/*.jsonl`, `raw/*.err` | stream transcript | Scratch (the transcript is duplicated in `~/.claude/projects/`). Keep only for Spire evaluation, and outside the repo |
| `logs/*.txt` | gate and test output | Scratch. The run log quotes the result line, which is the evidence |
| `preamble.md`, `mcp-empty.json` | frozen copies | Scratch |
| walk `*.py`, `state.json`, `__pycache__/` | throwaway driver code | Scratch. A walk script worth reusing should be promoted to `scripts/` or a test, not kept in `reports/` |
| walk PNGs | evidence | A few chosen images per finding are worth keeping. The rest is scratch |
## 2. What superpowers actually does
The directory is **not** `.sdd/`. It is **`<repo-root>/.superpowers/sdd/<plan-basename>/`**, created by the subagent-driven-development (SDD) skill. Verified in 7.0.0 and 6.4.1:
- **Location and resolver.** `skills/subagent-driven-development/scripts/sdd-workspace` is the "single source of truth for the workspace location" (`:27-28`). It puts the workspace at `base="$root/.superpowers/sdd"` (`:49`), with one directory per plan, `"$base/$slug"`, where the slug is the plan file's basename (`:41,73`). A `plan-path` marker file records which plan owns a directory, and a name clash gets a suffix (`:61-82`). `task-brief` and `review-package` write their default output there (`task-brief:7`: "Default OUTFILE: `<repo-root>/.superpowers/sdd/<plan-basename>/task-<N>-brief.md`").
- **Why in the working tree.** `sdd-workspace:21-23`: "The workspace lives in the working tree (not under .git/) because Claude Code treats .git/ as a protected path and denies agent writes there". Release notes (`RELEASE-NOTES.md:231`) confirm it moved out of `.git/sdd/` for this reason.
- **Gitignore.** It is self-ignoring, and no tracked file is changed: `sdd-workspace:86` `[ -e "$base/.gitignore" ] || printf '*\n' > "$base/.gitignore"`. A `*` in `.superpowers/sdd/.gitignore` hides everything below it, including the `.gitignore` itself. Since 7.0.0 the file is written only when it is missing, so a repo can commit its own version (`:84-85`, `RELEASE-NOTES.md:32`). In 6.4.1 it was overwritten on every run.
- **Contents.** "every artifact for THIS plan: ledger, briefs, reports, review packages" (`SKILL.md:138-140`). The ledger is `<workspace>/progress.md`.
- **Lifecycle.** It is deleted at the end, by the controller. `SKILL.md:482-485`: "When the final whole-branch review is clean and its fixes are merged, delete this plan's workspace (`rm -rf <workspace>`) — the git history is the record now. Sibling directories belong to other plans; leave them alone." There is the same step in `skills/executing-plans/SKILL.md:300`. Before deleting, decisions are copied out: `SKILL.md:473-480` says "collect every ledger line containing `Ruling:` … into your final message … A ruling that dies with the workspace was a decision made in secret."
- **Loss risk is stated, not solved.** `SKILL.md:153`: "`git clean -fdx` will destroy the workspace (it's git-ignored scratch); if that happens, recover from `git log`."
- **Other superpowers locations.** The brainstorm visual companion uses `.superpowers/brainstorm/<session>/` with `--project-dir`, and `/tmp/brainstorm-<id>` without it (`skills/brainstorming/scripts/start-server.sh:9-10,117,123`). Its stop script deletes only `/tmp` sessions (`stop-server.sh:112-114`). For `.superpowers/` it only asks the user to gitignore it (`visual-companion.md:58`: "Remind the user to add `.superpowers/` to `.gitignore`"), so the brainstorm part does **not** ignore itself. `diagnosing-superpowers` writes outside the repo, to `~/.superpowers/diagnosing-superpowers/<session-id>/` (`SKILL.md:33`).
**How well it works, judged from dirigent.** `/home/g4b/Projects/dirigent/.superpowers/sdd/` still holds 69 entries (40 directories plus loose files from the old flat layout), **91 MB** in total, from plans dated 2026-09-18 onwards. Only one file is newer than 2026-10-01. The ignore file works: none of it ever showed up in `git status`. The "delete at the end" step was skipped every time. So superpowers keeps the scratch out of git but does not keep it from piling up. Nothing ages it out, and the files are invisible, so nobody notices them.
## 3. Options for Spire
### 3a. Where the scratch dir lives
| Option | Pros | Cons |
|---|---|---|
| **A. `<repo>/.spire/scratch/<run>/`**, ignoring itself (`*` written on first use, like superpowers) | Nothing tracked changes. Agents can write there (not `.git/`, inside the project, so no permission prompt). Easy to find from the report. It moves along with the repo | Gets copied into every worktree checkout. `git clean -fdx` deletes it. Grows unseen, as dirigent's 91 MB `.superpowers/sdd/` shows. File walkers that skip ignored files hide it (as expected), but tools that do not skip them see it |
| **B. Under `.claude/spire/scratch/`** | Sits beside the existing Spire config (`.claude/spire/project.md`). Init already writes `.claude/.gitignore` (`skills/init/SKILL.md:293-302`), so init could add `spire/scratch/` | `.claude/` holds configuration, and mixing 50 MB of transcripts into it is surprising. Claude Code may treat parts of `.claude/` as sensitive in some modes (not verified). In a personal install, `spire/` is already ignored as a whole |
| **C. Outside the repo: `${XDG_CACHE_HOME:-~/.cache}/spire/<repo-slug>/<run>/`** | Can never reach git or worktrees, and no ignore rule is needed. Fits the host's "keep big output outside any checkout" rule (user `CLAUDE.md`, TMPDIR at `~/.cache/<project>-tmp`). Dirigent's CSS plan already did this by hand. Easy to age out globally | Writes outside the project may trigger permission prompts or sandbox denials for subagents and headless gremlins. Links from reports become absolute paths outside the repo. Harder to find |
| **D. `/tmp` or the session scratchpad** | Cleans itself up | On this host `/tmp` is a 7.7 GB tmpfs shared by all projects (user `CLAUDE.md`). Each headless gremlin gets its own session scratchpad, so the Athanor cannot reach the files. Gone at reboot, which is before the phase review |
| **E. Keep the workpad location, add an ignore rule** (e.g. ignore `docs/workpad/**/*-gremlins/`, `**/raw/`, `*.png` under walk folders) | Smallest change | Tracked and untracked files stay mixed in the same folders, which is exactly the complaint. The rule must be added in every project. Images worth keeping become hard to commit |
Whichever location is chosen, make it a single resolver, as superpowers does with `sdd-workspace`. One option is a `gremlins.py scratch --run R` command, or a small shared script. Then `store.state_dir`, the preamble, the spawn template and walk briefs all get the same path, and a project can override it in `.claude/spire/project.md` (for example a "Scratch:" line beside "Workpad:").
**Per-run subfolders.** Key them by the run log's slug, which is what gremlins already does (`<run>-gremlins`). The walk would then sit under the same run: `<scratch>/<run-slug>/gremlins/…`, `<scratch>/<run-slug>/walk-phase-a/…`. Superpowers' per-plan scoping, with its ownership marker, exists to stop one plan from reading another's ledger. Spire could use the same marker to prevent a slug clash.
### 3b. What stays in the workpad
- Run log, trunk notes, review and walk report as markdown, as now.
- The gremlins `ledger.jsonl`, or a rendered table of it in the run log. Small, and the only cost record. Options: (i) the helper writes the ledger to the scratch dir and `status` prints a summary the Athanor pastes into the run log; (ii) only the ledger stays beside the run log, everything else moves to scratch. Option (ii) keeps one machine-readable file in `reports/`.
- Handoff notes go into the run log (already required). The `notes/` files can then be scratch.
- Evidence images: a few chosen ones, committed under a dedicated folder, e.g. `reports/<date>-<slug>/evidence/NN-name.png`, one per finding at most, cropped or downscaled. Nothing else.
- Walk scripts that are worth keeping get promoted into the project's `scripts/` or test suite by an explicit decision. Throwaway scripts belong in scratch. Setting `PYTHONDONTWRITEBYTECODE=1` (or using `python -B`) in walk and gremlin briefs keeps `.pyc` out wherever the scripts run.
### 3c. Retention and cleanup: who deletes, and when
| Option | Pros | Cons |
|---|---|---|
| **Athanor deletes at phase end** | Close to the files | Too early: the reviewer, the fix pass and Igor read logs and briefs after hand-back (`agents/athanor.md:45`) |
| **Igor deletes when the build closes** (after the last phase review and the re-check), like superpowers' "final review clean, delete" | Matches the review cycle. Igor owns the run | The dirigent `.superpowers/sdd/` shows that a manual end-of-run delete is often skipped (91 MB left over). Needs a fixed step in Igor's close procedure, plus a "copy out first" rule like superpowers' `Ruling:` sweep |
| **TTL sweep** (e.g. delete run dirs older than N days, run at Igor or `spire:project` boot, or by a `just`/script recipe) | Works even when someone forgets. Matches the user's "interesting for a while, then should disappear" | It deletes on its own, so it needs a clear rule (only under the scratch root, only dirs with a Spire marker), or the user's consent for each sweep. Note that Claude Code's own transcripts in `~/.claude/projects/` already age out under its `cleanupPeriodDays` setting (from memory, not checked this session) |
| **Curator** | Already the "tidy the workpad" skill | `skills/curator/SKILL.md` compresses refinement sections in files, only when a human triggers it. Deleting files would be a different job; better as a separate method or command |
| **Pin to keep** | A run under study (Spire evaluation, e.g. gremlin cost research) can be exempted from TTL with a `keep` marker file | One more convention |
These can be combined. Igor deletes at build close, with a TTL as backstop and a keep marker for runs under study. The deletion step should list what it removes, and it must never touch anything outside the scratch root.
### 3d. Reports that link to evidence that will vanish
- **Quote, don't only link.** The run log already quotes the gate's result line (`run-phase-a.md:37`). A rule could say that a link to scratch is only a pointer, and the claim must stand on the quoted text alone.
- **Mark scratch links visibly**, e.g. `scratch: <path> (deleted after build close)`, or with a fixed prefix so a reader knows a dead link is expected. The dirigent run logs did this by hand with "(not committed)".
- **Promote before delete.** At build close, Igor (or the walker, when it writes its report) copies the evidence images it cites into `evidence/` and rewrites those links. Images that are not cited are deleted.
- **Point at durable ids instead of files** where they exist: commit hashes, the gremlin `session_id` from the ledger (resumable in `~/.claude/projects/` while Claude Code keeps it), test names.
### 3e. Places in Spire to change, whatever option is picked
- `skills/gremlins/grm/store.py:17-22` (state dir path), `SKILL.md:42-50`, `preamble.md:8`, `templates/spawn.md:15`.
- `skills/workpad/SKILL.md:3,8`: say that non-document artifacts (logs, transcripts, screenshots, scripts) go to the scratch dir, so the "workpad wins" rule stops pulling them in.
- `skills/implementation/references/run-log.md` and `agents/athanor.md:40`: render the ledger and copy notes into the run log; mark scratch links.
- `skills/igor/SKILL.md`: a build-close cleanup step, and a brief rule for walkers and other evidence-producing agents (scratch path, `PYTHONDONTWRITEBYTECODE=1`, the evidence limit).
- `skills/init/SKILL.md` Phase 5, only if the scratch dir lives in the repo but does not ignore itself.
## Gaps
- I did not inspect the deleted dirigent files themselves (already gone). Counts and sizes come from the coordinator's brief and follow-up.
- I did not check whether `raw/*.jsonl` matches Claude Code's stored transcript line for line, or how long `~/.claude/projects/` keeps forked headless sessions.
- I did not test whether writes to `~/.cache/…` from headless gremlins or Agent-tool subagents cause permission prompts under the project's current settings. That decides whether option C works.