📝 Add Dirigent process reports as input for the superpowers replacement

Retrospective of the ChatGPT importer run, the lean process report and the lean-finish handoff, copied from dirigent/docs/workpad/reports.

docs/workpad/reports/dirigent/2026-10-14-chatgpt-importer-retrospective.md

@@ -0,0 +1,161 @@
+# Retrospective: why the ChatGPT importer took a day, not 1–2 hours
+
+**Date**: 2026-10-14 · **Author**: researcher agent, model `claude-opus-5-5` · **Scope**: research (2026-10-13 21:00) to Phase 7/8 running (2026-10-14 21:53)
+**Sources**: git author times (`e6aa0532..HEAD`, branch `chatgpt-importer-phase8`), both journals, run log, handoff, design, plans, reviews, `spire` plugin skills (`~/.claude/plugins/cache/reliquary/spire/0.1.0/skills/igor/SKILL.md`), memory `feedback_lean_gates.md`.
+**Clock caveat**: the second journal's times (23:00–00:35) do not match git (the same events are at 19:22–21:49) or the memory file's mtime (19:49Z = 21:49 local). All times below come from git.
+
+## Verdict
+
+The day went to three things, roughly equally. (1) **Scope**: a second-generation importer was built where a first one was needed. It has union merge with variants and revisions, six generic importer seams, a protocol part, hostile-input hardening, versioned side entries and approximate library matching. (2) **The verification loop**: every phase ran review, then a fix pass, then a re-check, then carry-over "Block 0" fixes. That took about 5 h, about as long as writing the features (about 6 h). (3) **Dead time**: about 9.6 h of the 24.9 h wall clock was waiting. That covers the night, waiting on the history-scrub answer, and about 3 h lost after the spend-limit kill. The heavy rules came from Igor and the plan/design agents, not from the user. Igor's own skill prescribes less than Igor ran.
+
+## 1. Timeline (git author times, local +0200)
+
+| Stage | Wall clock | Span | Notes |
+|---|---|---|---|
+| Research (format census, 2024 compare, desktop app, dedup, E2/E3) | ~1 h 25 m | 21:00–22:26 | 10-13. Desktop-app pipe / `read_thread` work is not importer scope |
+| Design (independent Opus) | 23 m | 22:05–22:28 | 414 lines, 5 open questions |
+| Plan (independent Opus) | ~17 m | 22:21–22:38 | 321 lines, 8 phases, 17 deviations |
+| *Gap*: user Q&A, plan attack, decisions | 2 h 41 m | 22:38–01:19 | Attack 194 lines + reshaped build plan 637 lines; active share unknown (likely 30–45 m) |
+| P1 fixtures: build / review / fix / re-check | 45 / 14 / 15 / 5 m | 01:19–02:40 | Generator 2,610 LOC; review C1 = leak in fixtures |
+| *Idle*: night + scrub decision | 4 h 52 m | 02:40–07:32 | Scrub itself ~5 m |
+| P2 v1 reader | 22 / 9 / 12 / 6 m | 07:32–08:21 | Includes Block 0 (P1 re-check R1–R3) |
+| P3 mapping + citations + protocol | 63 / 13 / 23 / 6 m | 08:21–10:07 | Block 0 = P2 re-check N1–N5 |
+| P4 seams, union, importer | 128 / 10 / 58 / 5 m | 10:07–13:28 | Block 0 = P3 re-check; fix pass incl. profiling + fsync batching |
+| P5 v0 + union | 45 / 9 / 33 / 6 m | 13:28–15:01 | Union smoke 137–176 s per run |
+| P6 assets | 60 / 7 / 18 m (part A) | 15:01–16:26 | Killed mid-fix by API spend limit (429) |
+| *Dead*: spend-limit stop to handoff | 2 h 56 m | 16:26–19:22 | |
+| P6 fix part B | 34 m | 19:22–19:56 | All suites + 4 mutations + 2 smoke runs (Igor's brief, later retracted) |
+| P7 UI (main) ‖ P8 tests+audit (worktree) | ~1 h 55 m each, in parallel | 19:58–21:53 | P7: 6 commits + 1,021 uncommitted lines; P8: 16 commits, 3,970 lines |
+| **Total** | **24 h 53 m wall** | | **~15.3 h active, ~9.6 h idle/dead** |
+
+## 2. Where the active time went (~15.3 h)
+
+| Category | Time | Share | Evidence |
+|---|---|---|---|
+| Pre-build docs (research, design, plan, attack, decisions) | ~2.9 h | 19 % | 6 docs; attack added 38 findings and 3 user decisions |
+| Feature build blocks P1–P6 (incl. their gates and smoke) | ~6.0 h | 39 % | 363 m of block time |
+| ↳ of which fixture generator + leak gate (P1) | ~0.75 h | | Exists only because fixtures were derived from private exports |
+| Review → fix → re-check loop + carry-over Block 0s | ~5.2 h | 34 % | Reviews 62 m, fix passes 193 m, re-checks 28 m, Block 0s ~27 m |
+| P7 + P8 so far (parallel, counted once) | ~1.9 h | 12 % | Partly more hardening (audit R1–R6, perf) |
+| Incidents | ~0.1 h active, ~7.8 h waiting | | Scrub: ~5 m of work, but it held the build overnight. Spend limit: 2 h 56 m lost |
+
+The loop's fix passes alone (193 m) took more than half of the feature block time. The fix passes also fixed most **Minors** (for example P4 M1–M5 and M9–M12), although Igor's skill says "Lows stay in the report; never loop for lows" (`igor/SKILL.md` §Development Cycle).
+
+**Verification inside the blocks** is an estimate, because gate runs are not timed individually:
+- full `dirigent_archivist --no-fail-fast` runs (857+ tests, 13 known failures) in almost every block and every fix pass;
+- 37 mentions of mutation checks and 44 of smoke runs in the run log;
+- release smoke runs: union test 137 s → 176 s → 192 s → 203 s, plus each release compile;
+- two extra worktrees with their own fresh target dirs: a baseline at `e6aa0532` to prove failures were pre-existing, and a Phase-5 build to measure a converter bump.
+
+A conservative estimate is 25–35 % of block time.
+
+**Output.** 124 commits since `origin/main`, 71 of them docs-only. Code added: 17.1k lines of src, 8.8k lines of tests, 2.6k lines of generator and 29.4k lines of fixtures, with 340 new `#[test]` attributes on main. That excludes P8's 3,970 lines. Workpad docs for this importer come to about 7.2k lines:
+
+| Doc | Lines |
+|---|---|
+| Run log | 1,329 |
+| Build plan | 637 |
+| Design | 414 |
+| Plan | 321 |
+| Attack | 194 |
+| Reviews and re-checks | 1,684 (11 docs) |
+| Research | 1,027 |
+
+The archivist `CLAUDE.md` changed by 654 lines, and `dirigent_chatgpt/CLAUDE.md` grew into a 61-line spec.
+
+## 3. Scope: built vs. what a first importer needed
+
+**Needed for a first importer.**
+- Read the user's own v1 export (folder or ZIP) and the v0 `conversations.json`.
+- Turn each node into a record with `parent_id` and `active_leaf_id`, so branches survive.
+- Map the basic kinds.
+- Store images in the file store.
+- Make imports idempotent by node id through the existing Incremental mode.
+- Make the second export add the node ids that are missing. This satisfies the CLAUDE.md archive refinement: never silently lose archived data.
+- Commit a small hand-written fixture.
+- Run one smoke test on the real export.
+
+**Built beyond that need.**
+
+| Feature / hardening | Where | Why it was beyond need |
+|---|---|---|
+| Hostile-input hardening of a user's own OpenAI export: zip-bomb ratios, traversal, symlinks, nested-ZIP lazy spill with sub-range reader, manifest fast path for 3–20 GiB parts, `max_conversations` 250k, `max_nodes` 100k, output-amplification budget `REPEAT_SLACK`, cancellable aligner, P8 audit R1–R6 | `crates/dirigent_chatgpt/src/source.rs` (1,001), `tests/hostile_inputs.rs` (747), `src/citations.rs` (1,216), design §2.1 limits table | The input is trusted (the user downloads it). Real max is 210 nodes per conversation and 1,632 conversations. A `zip` size cap plus `enclosed_name()` would have covered the real risk. The P2–P3 re-checks became a "rolling audit" (journal 10-13, P3 re-check entry) |
+| Union with variants, winner rule, metadata key union, parent rule, content hash vs raw hash, revisions, converter version in the digest, cross-order hash convergence asserts | `archivist/src/import/variants.rs` (493), `sources/chatgpt/convert.rs` (1,032), smoke `tests/import_chatgpt_real_export_smoke.rs` (1,573) | 303 shared conversations needed "add missing node ids". The full machinery produced 4,124 representational revision lines (+26.7 MB) on the real union |
+| Six generic importer seams (opt-in prior, revisions, revision fold, digest, owned metadata, discovery context) | `archivist/src/import/mod.rs` (+519), `seam_tests.rs` (512) | Needed only for the union semantics above, and it put Claude and Codex paths at risk (a 4A gate and a review priority to prove them unchanged) |
+| Fixture generator with an anonymiser and an independent leak gate, run on private exports | `scripts/ops/chatgpt_fixtures.py` (2,610), `tests/fixture_hygiene.rs` (393) | Caused the leak (P1 C1), the history scrub and the overnight block. A synthetic hand-written fixture had no leak surface |
+| Versioned side entries (shares, feedback, model_comparisons, library records) and approximate library placement | `sources/chatgpt/library.rs` (689); P5/P6 fix passes | 10 shares, 0 unattached entries and 33 library files on real data. Plain storage would have done |
+| Intra-build back-compat: converter bumps 2→3→4 measured from older builds; placeholder wording "fixed before more archives" | run log P6 6C, part A | No real archive holds ChatGPT data from this unpushed build (imports went only to temp archives). It protected nothing |
+| `MessagePart::Sources` protocol part and a type-aware citation aligner (E2 rules) | `dirigent_protocol/src/conversation.rs` (+359), `citations.rs` | Nice to have. Text with inline links would have done for a first version |
+| 8 phases, phase 4 split into 5 blocks | build plan | Each phase boundary triggered the full loop. 3 phases would have done |
+
+**Worth keeping from the "extra"**: the fsync batching (`49c733e3`, first import 137 s → 28.6 s; it also benefits live writes), branches as `parent_id`, raw `_source` retention (user decision 1), and the archived-asset fallback (attack BL1).
+
+## 4. Who drove the blow-up
+
+| Decision | Origin | Evidence |
+|---|---|---|
+| Design → plan before build; subagent-driven; "review between tasks" | **User** (brief + root `CLAUDE.md` §Superpowers) | journal 10-13 22:00 |
+| Accept all recommendations (raw storage, full switcher, automatic revisions, agent runs generator on private exports) | **User**, on recommendations from the designer and attacker | journal "use recommendations" |
+| Build on `main`, scrub history, approve limits 1–4, lean gates | **User** | journals |
+| Union-of-everything, raw `_source`, safety-limits table, five seams, `MessagePart::Sources` | **Designer** (from the research's "Merge, not replace", overview §12) | design §1, §2.1, §6 |
+| 8 phases; per-phase gates of full archivist + api + rest suites and `check-all`; `hostile_inputs.rs`; real-export smoke; field ledger | **Planner** | plan 10-13 l.14, l.83; build plan l.351–353, l.430–434 |
+| Plan attack stage; extra blocks (4A1–4B2), converter version in digest, more leak-gate rules, decisions 6–8 | **Igor** dispatched it; the **attacker** added the content | Not in Igor's six-stage cycle (`igor/SKILL.md` l.95). No earlier attack doc exists in the repo |
+| Fresh reviewer after **every** phase, plus a fix pass, plus a **re-check**; folding re-check findings into Block 0s; fixing Minors | **Igor** | journal 10-13: "Plan per phase: … reviewer → at most one fix pass", first re-check at P1. The skill says one review stage and "never loop for lows" |
+| Mutation / fail-first checks of tests | **Builders**, via superpowers TDD habit, with reviewers grading "fails on mutation" | run log l.347, l.510 (`local/break_4a1.py`); in no plan or brief |
+| Full suites + 4 mutations + repeated release smoke after the user approved cuts | **Igor** | journal 10-14 course-correction entry: "my rules, not theirs"; memory `feedback_lean_gates.md` |
+| Baseline worktree to prove 13 + 9 failures were pre-existing; converter-bump worktree | **Igor**, **reviewer**, **builder** | journal P3 review and P4 entries; run log 6C |
+
+The user's claim holds. No user message sets per-phase reviews, re-checks, mutation runs, full-suite gates or hostile-input depth. The one user-sourced rule that pushes toward rigour is the archive-safety refinement. It justifies "add missing nodes, never drop". It does not justify revisions of representational differences or hostile-ZIP limits.
+
+## 5. Changes for builds of this size
+
+**Policy**
+
+| Area | Rule |
+|---|---|
+| Stages | Skip design + plan + attack when research already names the format and the seams. Write one combined design-plan of 100–150 lines instead. Run an attack only when a plan touches central storage formats *and* the user asks for one. |
+| Phases | At most 3: reader, mapping + import, UI. No sub-block splits under about 300 LOC. |
+| Review | One fresh review at the end, scoped to data loss and correctness. One fix pass for Critical/Important only. Minors are logged, never looped. No re-check unless a fix touches a Critical. |
+| Gates | `cargo check -p` between blocks. Changed-crate tests once per phase. `check-all` once at the end. No mutation runs unless a Critical needs proof. Real-export smoke once, at the end. No baseline or measurement worktrees unless the user asks. |
+| Hardening | A trusted local input gets the library's default safety (zip size cap, `enclosed_name`). It gets no adversarial audit unless the input crosses a trust boundary (upload, network). |
+| Back-compat | Not before any real archive holds the data. Temp-archive data written by an unpushed build needs no migration story. |
+| Fixtures | Hand-written synthetic fixtures. Never derive them from private data for a first version. |
+| Docs | A run log of at most 1 line per block plus deviations (≤ 80 lines total). No per-phase trunk notes. A crate `CLAUDE.md` stays a nav map of 20–50 lines. |
+| Spend | Check the API budget before a long serial run. A kill cost 3 h. |
+
+**Target budget for a comparable importer (≈ 2 h)**
+
+| Stage | Budget |
+|---|---|
+| Research (only if the format is unknown; here it already existed) | 0–15 m |
+| Combined design-plan + user questions | 15 m |
+| P1 reader + synthetic fixture | 25 m |
+| P2 mapping, import, union-by-node-id, assets | 35 m |
+| P3 UI (thread order, branch switcher) | 20 m |
+| Final review + one fix pass | 20 m |
+| Real-export smoke + docs | 10 m |
+| **Total** | **~2 h** |
+
+**My view**: 1–2 h is realistic for the scope in §3 "Needed". The scope that was actually agreed (v0+v1 union with revisions, assets, library, Sources, full switcher) is closer to 3–4 h even with lean gates. Holding the 1–2 h line therefore needs the scope cut, not only the process cut.
+
+## 6. Phases 7 and 8: finish vs. drop
+
+| Item | Call | Reason |
+|---|---|---|
+| P7 7B thread view, branch switcher, Sources footer, missing-asset notes (committed 20:39–21:34) | **Keep** | Done; user-visible |
+| P7 7A Import Hub (1,021 uncommitted lines across api/web/claude/codex importers) | **Finish, then stop** | Commit it after `just check-all`, and confirm the Claude and Codex edits are wiring only |
+| P7 full phase review | **Merge into one final review** | Cover P6 part B + P7 + the P8 merge. Scope: data loss and display correctness. No hostile-input scope. No re-check |
+| P8 e2e suite, per-node golden, last-session cancel fix (`763bbbd1`, `95817c6c`, `db4c8c37`) | **Keep, merge** | Already done; cheap regression value |
+| P8 audit fixes R1–R6 and mapping-table perf (`8fe18a8f`) | **Keep (done), stop there** | Sunk cost. Add no further attack classes |
+| Hostile-input audit report document | **Drop or ≤ 30 lines** | The fixes are the record |
+| Remaining P8 perf/memory items (16k-session extrapolation, the 38.6 s vs 29.8 s re-time, asset-byte memory, shard cache) | **Drop to backlog** | Real account is 1.6k sessions; first import takes about 30 s |
+| ZIP-form real export test | **Drop**; one manual import of a zipped copy if wanted | Synthetic ZIP tests already cover the code path |
+| Further converter-bump measurements | **Drop** | No real archive to protect |
+| Close: `dirigent_chatgpt/CLAUDE.md` back to a nav map; spec to `docs/architecture/chatgpt_importer.md` | **Keep, ≤ 20 m** | Already decided; keeps future agents from re-growing the spec |
+| Pre-existing failures (13 archivist, 9 api, 8 web test-target errors) | **Separate ticket** | Not this build |
+
+## Gaps
+
+- The time from the plan commit to the attack commit (22:38–01:14) cannot be split into user time and agent time from the sources.
+- Compile and test durations are not logged per run, so the verification share inside the blocks is an estimate.
+- The research session's own brief is not in these journals, so the origin of the desktop-app/pipe research is not attributed.