reliquary/docs/workpad/reports/dirigent/2026-10-14-chatgpt-importer-retrospective.md

Retrospective: why the ChatGPT importer took a day, not 1–2 hours

Date: 2026-10-14 · Author: researcher agent, model claude-opus-5-5 · Scope: research (2026-10-13 21:00) to Phase 7/8 running (2026-10-14 21:53) Sources: git author times (e6aa0532..HEAD, branch chatgpt-importer-phase8), both journals, run log, handoff, design, plans, reviews, spire plugin skills (~/.claude/plugins/cache/reliquary/spire/0.1.0/skills/igor/SKILL.md), memory feedback_lean_gates.md. Clock caveat: the second journal's times (23:00–00:35) do not match git (the same events are at 19:22–21:49) or the memory file's mtime (19:49Z = 21:49 local). All times below come from git.

Verdict

The day went to three things, roughly equally. (1) Scope: a second-generation importer was built where a first one was needed. It has union merge with variants and revisions, six generic importer seams, a protocol part, hostile-input hardening, versioned side entries and approximate library matching. (2) The verification loop: every phase ran review, then a fix pass, then a re-check, then carry-over "Block 0" fixes. That took about 5 h, about as long as writing the features (about 6 h). (3) Dead time: about 9.6 h of the 24.9 h wall clock was waiting. That covers the night, waiting on the history-scrub answer, and about 3 h lost after the spend-limit kill. The heavy rules came from Igor and the plan/design agents, not from the user. Igor's own skill prescribes less than Igor ran.

1. Timeline (git author times, local +0200)

Stage	Wall clock	Span	Notes
Research (format census, 2024 compare, desktop app, dedup, E2/E3)	~1 h 25 m	21:00–22:26	10-13. Desktop-app pipe / read_thread work is not importer scope
Design (independent Opus)	23 m	22:05–22:28	414 lines, 5 open questions
Plan (independent Opus)	~17 m	22:21–22:38	321 lines, 8 phases, 17 deviations
Gap: user Q&A, plan attack, decisions	2 h 41 m	22:38–01:19	Attack 194 lines + reshaped build plan 637 lines; active share unknown (likely 30–45 m)
P1 fixtures: build / review / fix / re-check	45 / 14 / 15 / 5 m	01:19–02:40	Generator 2,610 LOC; review C1 = leak in fixtures
Idle: night + scrub decision	4 h 52 m	02:40–07:32	Scrub itself ~5 m
P2 v1 reader	22 / 9 / 12 / 6 m	07:32–08:21	Includes Block 0 (P1 re-check R1–R3)
P3 mapping + citations + protocol	63 / 13 / 23 / 6 m	08:21–10:07	Block 0 = P2 re-check N1–N5
P4 seams, union, importer	128 / 10 / 58 / 5 m	10:07–13:28	Block 0 = P3 re-check; fix pass incl. profiling + fsync batching
P5 v0 + union	45 / 9 / 33 / 6 m	13:28–15:01	Union smoke 137–176 s per run
P6 assets	60 / 7 / 18 m (part A)	15:01–16:26	Killed mid-fix by API spend limit (429)
Dead: spend-limit stop to handoff	2 h 56 m	16:26–19:22
P6 fix part B	34 m	19:22–19:56	All suites + 4 mutations + 2 smoke runs (Igor's brief, later retracted)
P7 UI (main) ‖ P8 tests+audit (worktree)	~1 h 55 m each, in parallel	19:58–21:53	P7: 6 commits + 1,021 uncommitted lines; P8: 16 commits, 3,970 lines
Total	24 h 53 m wall		~15.3 h active, ~9.6 h idle/dead

2. Where the active time went (~15.3 h)

Category	Time	Share	Evidence
Pre-build docs (research, design, plan, attack, decisions)	~2.9 h	19 %	6 docs; attack added 38 findings and 3 user decisions
Feature build blocks P1–P6 (incl. their gates and smoke)	~6.0 h	39 %	363 m of block time
↳ of which fixture generator + leak gate (P1)	~0.75 h		Exists only because fixtures were derived from private exports
Review → fix → re-check loop + carry-over Block 0s	~5.2 h	34 %	Reviews 62 m, fix passes 193 m, re-checks 28 m, Block 0s ~27 m
P7 + P8 so far (parallel, counted once)	~1.9 h	12 %	Partly more hardening (audit R1–R6, perf)
Incidents	~0.1 h active, ~7.8 h waiting		Scrub: ~5 m of work, but it held the build overnight. Spend limit: 2 h 56 m lost

The loop's fix passes alone (193 m) took more than half of the feature block time. The fix passes also fixed most Minors (for example P4 M1–M5 and M9–M12), although Igor's skill says "Lows stay in the report; never loop for lows" (igor/SKILL.md §Development Cycle).

Verification inside the blocks is an estimate, because gate runs are not timed individually:

• full dirigent_archivist --no-fail-fast runs (857+ tests, 13 known failures) in almost every block and every fix pass;
• 37 mentions of mutation checks and 44 of smoke runs in the run log;
• release smoke runs: union test 137 s → 176 s → 192 s → 203 s, plus each release compile;
• two extra worktrees with their own fresh target dirs: a baseline at e6aa0532 to prove failures were pre-existing, and a Phase-5 build to measure a converter bump.

A conservative estimate is 25–35 % of block time.

Output. 124 commits since origin/main, 71 of them docs-only. Code added: 17.1k lines of src, 8.8k lines of tests, 2.6k lines of generator and 29.4k lines of fixtures, with 340 new #[test] attributes on main. That excludes P8's 3,970 lines. Workpad docs for this importer come to about 7.2k lines:

Doc	Lines
Run log	1,329
Build plan	637
Design	414
Plan	321
Attack	194
Reviews and re-checks	1,684 (11 docs)
Research	1,027

The archivist CLAUDE.md changed by 654 lines, and dirigent_chatgpt/CLAUDE.md grew into a 61-line spec.

3. Scope: built vs. what a first importer needed

Needed for a first importer.

• Read the user's own v1 export (folder or ZIP) and the v0 conversations.json.
• Turn each node into a record with parent_id and active_leaf_id, so branches survive.
• Map the basic kinds.
• Store images in the file store.
• Make imports idempotent by node id through the existing Incremental mode.
• Make the second export add the node ids that are missing. This satisfies the CLAUDE.md archive refinement: never silently lose archived data.
• Commit a small hand-written fixture.
• Run one smoke test on the real export.

Built beyond that need.

Feature / hardening	Where	Why it was beyond need
Hostile-input hardening of a user's own OpenAI export: zip-bomb ratios, traversal, symlinks, nested-ZIP lazy spill with sub-range reader, manifest fast path for 3–20 GiB parts, max_conversations 250k, max_nodes 100k, output-amplification budget REPEAT_SLACK, cancellable aligner, P8 audit R1–R6	crates/dirigent_chatgpt/src/source.rs (1,001), tests/hostile_inputs.rs (747), src/citations.rs (1,216), design §2.1 limits table	The input is trusted (the user downloads it). Real max is 210 nodes per conversation and 1,632 conversations. A zip size cap plus enclosed_name() would have covered the real risk. The P2–P3 re-checks became a "rolling audit" (journal 10-13, P3 re-check entry)
Union with variants, winner rule, metadata key union, parent rule, content hash vs raw hash, revisions, converter version in the digest, cross-order hash convergence asserts	archivist/src/import/variants.rs (493), sources/chatgpt/convert.rs (1,032), smoke tests/import_chatgpt_real_export_smoke.rs (1,573)	303 shared conversations needed "add missing node ids". The full machinery produced 4,124 representational revision lines (+26.7 MB) on the real union
Six generic importer seams (opt-in prior, revisions, revision fold, digest, owned metadata, discovery context)	archivist/src/import/mod.rs (+519), seam_tests.rs (512)	Needed only for the union semantics above, and it put Claude and Codex paths at risk (a 4A gate and a review priority to prove them unchanged)
Fixture generator with an anonymiser and an independent leak gate, run on private exports	scripts/ops/chatgpt_fixtures.py (2,610), tests/fixture_hygiene.rs (393)	Caused the leak (P1 C1), the history scrub and the overnight block. A synthetic hand-written fixture had no leak surface
Versioned side entries (shares, feedback, model_comparisons, library records) and approximate library placement	sources/chatgpt/library.rs (689); P5/P6 fix passes	10 shares, 0 unattached entries and 33 library files on real data. Plain storage would have done
Intra-build back-compat: converter bumps 2→3→4 measured from older builds; placeholder wording "fixed before more archives"	run log P6 6C, part A	No real archive holds ChatGPT data from this unpushed build (imports went only to temp archives). It protected nothing
MessagePart::Sources protocol part and a type-aware citation aligner (E2 rules)	dirigent_protocol/src/conversation.rs (+359), citations.rs	Nice to have. Text with inline links would have done for a first version
8 phases, phase 4 split into 5 blocks	build plan	Each phase boundary triggered the full loop. 3 phases would have done

Worth keeping from the "extra": the fsync batching (49c733e3, first import 137 s → 28.6 s; it also benefits live writes), branches as parent_id, raw _source retention (user decision 1), and the archived-asset fallback (attack BL1).

4. Who drove the blow-up

Decision	Origin	Evidence
Design → plan before build; subagent-driven; "review between tasks"	User (brief + root CLAUDE.md §Superpowers)	journal 10-13 22:00
Accept all recommendations (raw storage, full switcher, automatic revisions, agent runs generator on private exports)	User, on recommendations from the designer and attacker	journal "use recommendations"
Build on main, scrub history, approve limits 1–4, lean gates	User	journals
Union-of-everything, raw _source, safety-limits table, five seams, MessagePart::Sources	Designer (from the research's "Merge, not replace", overview §12)	design §1, §2.1, §6
8 phases; per-phase gates of full archivist + api + rest suites and check-all; hostile_inputs.rs; real-export smoke; field ledger	Planner	plan 10-13 l.14, l.83; build plan l.351–353, l.430–434
Plan attack stage; extra blocks (4A1–4B2), converter version in digest, more leak-gate rules, decisions 6–8	Igor dispatched it; the attacker added the content	Not in Igor's six-stage cycle (igor/SKILL.md l.95). No earlier attack doc exists in the repo
Fresh reviewer after every phase, plus a fix pass, plus a re-check; folding re-check findings into Block 0s; fixing Minors	Igor	journal 10-13: "Plan per phase: … reviewer → at most one fix pass", first re-check at P1. The skill says one review stage and "never loop for lows"
Mutation / fail-first checks of tests	Builders, via superpowers TDD habit, with reviewers grading "fails on mutation"	run log l.347, l.510 (local/break_4a1.py); in no plan or brief
Full suites + 4 mutations + repeated release smoke after the user approved cuts	Igor	journal 10-14 course-correction entry: "my rules, not theirs"; memory feedback_lean_gates.md
Baseline worktree to prove 13 + 9 failures were pre-existing; converter-bump worktree	Igor, reviewer, builder	journal P3 review and P4 entries; run log 6C

The user's claim holds. No user message sets per-phase reviews, re-checks, mutation runs, full-suite gates or hostile-input depth. The one user-sourced rule that pushes toward rigour is the archive-safety refinement. It justifies "add missing nodes, never drop". It does not justify revisions of representational differences or hostile-ZIP limits.

5. Changes for builds of this size

Policy

Area	Rule
Stages	Skip design + plan + attack when research already names the format and the seams. Write one combined design-plan of 100–150 lines instead. Run an attack only when a plan touches central storage formats and the user asks for one.
Phases	At most 3: reader, mapping + import, UI. No sub-block splits under about 300 LOC.
Review	One fresh review at the end, scoped to data loss and correctness. One fix pass for Critical/Important only. Minors are logged, never looped. No re-check unless a fix touches a Critical.
Gates	cargo check -p between blocks. Changed-crate tests once per phase. check-all once at the end. No mutation runs unless a Critical needs proof. Real-export smoke once, at the end. No baseline or measurement worktrees unless the user asks.
Hardening	A trusted local input gets the library's default safety (zip size cap, enclosed_name). It gets no adversarial audit unless the input crosses a trust boundary (upload, network).
Back-compat	Not before any real archive holds the data. Temp-archive data written by an unpushed build needs no migration story.
Fixtures	Hand-written synthetic fixtures. Never derive them from private data for a first version.
Docs	A run log of at most 1 line per block plus deviations (≤ 80 lines total). No per-phase trunk notes. A crate CLAUDE.md stays a nav map of 20–50 lines.
Spend	Check the API budget before a long serial run. A kill cost 3 h.

Target budget for a comparable importer (≈ 2 h)

Stage	Budget
Research (only if the format is unknown; here it already existed)	0–15 m
Combined design-plan + user questions	15 m
P1 reader + synthetic fixture	25 m
P2 mapping, import, union-by-node-id, assets	35 m
P3 UI (thread order, branch switcher)	20 m
Final review + one fix pass	20 m
Real-export smoke + docs	10 m
Total	~2 h

My view: 1–2 h is realistic for the scope in §3 "Needed". The scope that was actually agreed (v0+v1 union with revisions, assets, library, Sources, full switcher) is closer to 3–4 h even with lean gates. Holding the 1–2 h line therefore needs the scope cut, not only the process cut.

6. Phases 7 and 8: finish vs. drop

Item	Call	Reason
P7 7B thread view, branch switcher, Sources footer, missing-asset notes (committed 20:39–21:34)	Keep	Done; user-visible
P7 7A Import Hub (1,021 uncommitted lines across api/web/claude/codex importers)	Finish, then stop	Commit it after just check-all, and confirm the Claude and Codex edits are wiring only
P7 full phase review	Merge into one final review	Cover P6 part B + P7 + the P8 merge. Scope: data loss and display correctness. No hostile-input scope. No re-check
P8 e2e suite, per-node golden, last-session cancel fix (763bbbd1, 95817c6c, db4c8c37)	Keep, merge	Already done; cheap regression value
P8 audit fixes R1–R6 and mapping-table perf (8fe18a8f)	Keep (done), stop there	Sunk cost. Add no further attack classes
Hostile-input audit report document	Drop or ≤ 30 lines	The fixes are the record
Remaining P8 perf/memory items (16k-session extrapolation, the 38.6 s vs 29.8 s re-time, asset-byte memory, shard cache)	Drop to backlog	Real account is 1.6k sessions; first import takes about 30 s
ZIP-form real export test	Drop; one manual import of a zipped copy if wanted	Synthetic ZIP tests already cover the code path
Further converter-bump measurements	Drop	No real archive to protect
Close: dirigent_chatgpt/CLAUDE.md back to a nav map; spec to docs/architecture/chatgpt_importer.md	Keep, ≤ 20 m	Already decided; keeps future agents from re-growing the spec
Pre-existing failures (13 archivist, 9 api, 8 web test-target errors)	Separate ticket	Not this build

Gaps

• The time from the plan commit to the attack commit (22:38–01:14) cannot be split into user time and agent time from the sources.
• Compile and test durations are not logged per run, so the verification share inside the blocks is an estimate.
• The research session's own brief is not in these journals, so the origin of the desktop-app/pipe research is not attributed.