[guijacquemet]

PP-12 of the v0.4.0 plan (design v0.4 §1 and §8): per-file staging and resume, crash-consistent publication, and the two source-provenance issues.

Fixes #86, fixes #90

Summary (by module)

• inventory (new, io; the move is its own commit): source_fingerprint and LABELS_GROUP moved out of manifest (re-exported). source_signature returns size, newest mtime and, for an OME-Zarr folder, the SHA-256 of its inventory: each file's relative path, size and mtime_ns, sorted by path, with the root labels group left out. It is still metadata only (design §1). A folder inside the image that cannot be listed now raises. Links to folders inside the image are followed, as the reader follows them, and links that reach one folder twice are refused (review).
• fingerprint, manifest, mask_record, conversion_record, sources ([Sad Catapella B5][Scientific integrity] Aggregate OME-Zarr fingerprint misses changed chunks when newest mtime is unchanged #86): Fingerprint.inventory, plus FileEntry.inventory_sha256, which is written for OME-Zarr images only. Label images and copies keep the digest in their records too, and check_sources compares it. An entry recorded without it (v0.4.0rc3) counts as changed, so its rows are never reused. file_entry_with builds an entry from a fingerprint that was already taken.
• run.staging (new, io): each image's rows are staged in .staging as soon as the image is read. The rows go to <id>.csv (the normalization writer), then <id>.json names the image, the rows' SHA-256 and the staging key: the RowKey, the pixel type, the schema version and the versions of preprocessing and normalization.
• run.resume (new, io): for each file that needs new rows, the run either resumes it (same staging key and fingerprint) or reads it and stages it. All other staged rows are discarded and the reason is reported, by image. A full run reuses no staged rows. It also holds source_changed_during_run ([Sad Catapella B5][Scientific integrity] Source files may change between fingerprint capture and histogram publication #90).
• run.generation (new, io): the publication protocol (below), finish_publication, published_path and read_stable.
• run.publish: only write_report remains. The backup-and-restore protocol (and its .run- folders) is removed.
• run.pipeline / run.steps: the run's preflight binds each header to its fingerprint (bind_headers=True). The rows come from resume, and the run stops on source_changed_during_run. Every image is fingerprinted again inside the publication, just before the commit. The generation is published through staged_generation, and .staging is removed after the commit.
• run.lock: once the lock is held, an interrupted publication is finished (if committed) or discarded (if not). This applies to run, check, masks, design, calibrate and the notebook's writes.
• run_inputs: reads the table and the design by the SHA-256 the manifest names (from the output folder, or from .publish while a publication is unfinished). It reads again if the manifest changed during the read.
• log_target (after merging Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128): --log is refused inside .publish and .staging (the .run- folders no longer exist).
• discovery_report / RunResult / report / notebook_run: new resumed and staged_discarded fields in the report, RunResult.resumed, a summary line and a notebook tile.
• conversion_record (owner decision): copy_changed_after_conversion is now an error. The run stops after the header checks and writes only its report. The message says to convert the originals again into a new folder.
• run_inputs_text (new, pure; a move-only commit): RunInputs' refusal texts, moved unchanged so run_inputs.py stays under 600 lines.
• Docs: contract §9.4 (new), §8, §9, §9.2 and §17; the user guide; plan §10 (PP-12, with its follow-ups) and decided question 69 (numbered after main's 64 and 65 from Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128, 66 from Micrometre conversions are always positive and finite; check and design exit 1 on errors (#93) #133, 67 and 68 from same_objects is a one-to-one matching; size and plane-count warnings follow the confirmed roles (#67, #74) #131); development.md and AGENTS.md. Version stays 0.4.0rc3.

Decisions

The publication design (guarantee). Manifest-last commit, the simplest design that keeps the documented files as plain files on every system:

1. The whole generation (table, design, manifest, report) is written into .publish and flushed. Then publication.json is written, with each file's SHA-256 and the files to remove (mask_relationships.json).
2. The one atomic step: os.replace(.publish/run_manifest.json, run_manifest.json). Nothing in the output folder changes before this.
3. The other files are moved in, the removed files are deleted, then .publish and then .staging are deleted.

What each reader sees:

• Readers through the manifest (RunInputs) see exactly one generation, old or new, never a mix, also after a kill at any step.
• A plain reader of plane_histograms.csv always reads one whole table of one generation: the old one until it is moved.
• A reader that opens several files without the manifest can pair two generations during an interrupted step 3. The contract tells such readers to go through RunInputs or check the SHA-256 values.

Rejected: a generation folder with an atomic symbolic link (Windows without Developer Mode, Google Drive in Colab and zipped or copied output folders do not keep links); generation-named files with the manifest as pointer (changes the documented path or doubles the disk space).

On Windows the protocol is the same. os.replace fails while another program has a file open: at the commit nothing has changed; after it, the run raises PublicationIncompleteError and the next run finishes the moves. Folders are not fsynced on Windows, and the SIGKILL tests are POSIX-only.

Resume. Design §1 says the staging folder is removed "at the start of the next run". It is kept until a run publishes, so a second interruption loses nothing; every run re-validates every staged row at its start (accepted by the owner and coordinator on that condition; question 69).

#90. A file's fingerprints must all agree: the two taken around its header read (the manifest entry records that fingerprint); the one taken just after its pixels are read (only then are its rows staged); the one taken for every analysed image just before the commit. Any difference is source_changed_during_run (an error): nothing is published, the earlier generation is kept, and the other images' staged rows stay for the next run.

Notes

• v0.4.0rc3 runs on OME-Zarr images must be run again (no inventory: recomputed, and RunInputs refuses them).
• A chunk rewritten with the same size and its time set back is not detected; only content hashing would catch it, which stays off (design §1, owner's call). Documented in contract §9.
• Follow-ups recorded in plan §10: masks are fingerprinted when the manifest is written, not bound to their header fingerprints during the run; a run killed while an atomic write's temporary file is open leaves a hidden .<name>.<random>.tmp in the output folder (on main too).
• Main merged in four times (no rebase): OME-Zarr axes and calibration: nothing guessed, nothing silent (#72, #73, #94) #129/Experiment design identity and counts: excluded folders, duplicate columns, output folder (#82, #83, #98) #130; Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128 (conflicts in contract §8, plan §10/§12 and run/publish.py, resolved hunk by hunk); Micrometre conversions are always positive and finite; check and design exit 1 on errors (#93) #133 (plan §10/§12 only); same_objects is a one-to-one matching; size and plane-count warnings follow the confirmed roles (#67, #74) #131 (plan §12 only).

Measured (fingerprint cost)

OME-Zarr store with 20,484 files, shared machine (load ~5, noisy): one fingerprint 186–203 ms on main vs 252–279 ms here; 3 walks per reused image and 4 per image read (1 on main); no-op restart 0.25–0.28 s vs 0.91–0.95 s; first-run difference below the noise; table byte-identical (5a6211b2…). No performance claim is made.

Tests

Each fails on main: test_inventory.py (#86 counterexample, links to folders), test_resume.py (SIGKILL while reading then resumed byte-identically; discards with reasons; #90 during the read, before the commit, after the header, an OME-Zarr chunk), test_publication.py (failures before/after the commit and at clean-up; SIGKILL before each of the 24 file-changing steps), test_convert_run.py / test_convert_integrity.py (the new error).

Full gate on the final head (Linux, Python 3.13, .[dev,notebook,zarr], as root): ruff check passed; ruff format --check 201 files formatted; tools/check.py --rules passed (baseline only shrank: test_run.py 982 → 862; no # noqa added); pytest 1997 passed, 2 skipped (test_discovery.py:665, root ignores file modes; test_cli_log.py:165, from #128, needs a case-insensitive file system).

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

---

Generated by Claude Code

[guijacquemet]

Independent review: merged as af8f769 with a merge commit. The branch was auto-deleted and has been restored on f7ccf98.

Review fixes (small commits)

• 7e23fda ([Sad Catapella B5][Scientific integrity] Aggregate OME-Zarr fingerprint misses changed chunks when newest mtime is unchanged #86): the OME-Zarr inventory skipped folders reached through a link. os.walk does not enter a linked folder, so a chunk changed behind one, which the reader does follow, left the fingerprint unchanged. Links are now followed, and links that reach one folder twice are refused. This matches main's [Sad Catapella B4][Discovery] Symbolic links can silently import images outside the experiment root #81 link rule. Two tests were added, and both fail without the fix.
• 15ffdd5: some staged rows were discarded under the wrong reason. A different pixel type, or an image changed after its header was read, was reported as "axis roles, labels, folder or row settings changed". Unreadable staged rows were reported under the hashed record name instead of the image. The copy_changed_after_conversion text said "Nothing was written", which breaks PP-8c because the report is written. Test added; it fails without the fix.
• abaacc6: the plan records the decision to keep .staging until a run publishes. The condition for it was checked here: every run re-validates every staged row at its start. Plan §10 now lists two follow-ups: binding masks to their header fingerprints, and the hidden .tmp files an atomic write leaves after a kill (this also happens on main).
• Main was merged three times, with no rebase:
  • c91dfee (Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128): --log now protects .publish and .staging, because the .run- folders are gone.
  • 63333e1 (Micrometre conversions are always positive and finite; check and design exit 1 on errors (#93) #133).
  • f7ccf98 (same_objects is a one-to-one matching; size and plane-count warnings follow the confirmed roles (#67, #74) #131).
  • This PR's plan question is now 69, and every reference to it is updated.

Probes, run independently of the PR's tests

• SIGKILL at every file-changing step. An audit hook killed a real full run at each of its 95 file-changing operations (open for write, rename, chmod, mkdir, remove, rmtree) in the output folder, across staging, .publish, the commit, the moves and the clean-up. After each kill:
  • a plain read of plane_histograms.csv was always one whole table, the old or the new one;
  • RunInputs always opened exactly one generation;
  • the next locked run always finished a committed publication or discarded an uncommitted one; the final table was byte-identical to an uninterrupted run, the manifest matched the table, mask_relationships.json was removed, and no .publish or .staging folder was left.
  • The only leftovers were the provisional report's .tmp files, at kill points 3, 4, 6 and 7. This happens on main too and is a recorded follow-up.
• Resume. A restart with changed bins was killed at each of its 94 steps. Each next run resumed exactly the files with a complete staged record (0 to 8) and gave a byte-identical table.
• Discard reasons. Each of these was discarded with its reason and the table still matched: touched image, changed version, corrupted rows, changed pixel type, and a full run (which reuses nothing).
• [Sad Catapella B5][Scientific integrity] Source files may change between fingerprint capture and histogram publication #90. An image changed during its read, or after the table and manifest were written but before the commit, gave source_changed_during_run. The outputs stayed unchanged, .publish was removed, and the other images' staged rows were kept. The next run resumed 7 images and read only the changed one.
• [Sad Catapella B5][Scientific integrity] Aggregate OME-Zarr fingerprint misses changed chunks when newest mtime is unchanged #86. Each of these changes the fingerprint while total size and newest time stay the same: a retimed chunk, a renamed chunk, an added empty chunk, a removed chunk, and a resized chunk. Label images are left out. rc3 entries count as changed (documented).

§9 figures (files touched)

• Largest file: manifest.py, 592 lines.
• Longest function: manifest.py:from_dict, 58 executable lines.
• Highest complexity: check_sources, 15.
• Most statements: 37.
• Most parameters: 15 for run_experiment. This is already in the baseline and unchanged; apart from it the maximum is 7.
• Layers are declared for the new modules.
• Baseline: only shrank (test_run.py 982 to 862). No # noqa was added or removed.

Gate on the final head
Linux, Python 3.13, .[dev,notebook,zarr], run as root.

• ruff check: passed.
• ruff format --check: 201 files formatted.
• tools/check.py --rules: passed.
• pytest: 1997 passed, 2 skipped. The skips are test_discovery.py:665 (root ignores file modes) and test_cli_log.py:165 (from Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99) #128; needs a file system that ignores case).
• CI: green on all 7 jobs. macOS 3.11 was cancelled once by the runner and passed on rerun.

#86 and #90 are closed.

---

Generated by Claude Code