Skip to content

Turn content hashing of the sources on by default - #189

Merged
guijacquemet merged 3 commits into
mainfrom
claude/content-hash-default
Nov 7, 2026
Merged

guijacquemet merged 3 commits into
mainfrom
claude/content-hash-default

Conversation

@guijacquemet

Copy link
Copy Markdown
Contributor

What changed and why

Fixes #159 as decided by the owner (2026-11-07): content hashing ON by default, the cost benchmarked on a realistic folder, and an explicit opt-out recorded in the manifest.

Without content hashing, an image rewritten in place with its size and modification time kept counted as unchanged and its earlier rows were reused.

  • On by default. A run with no verify_content choice (RunMode() default, run_experiment default, CLI without a flag) records each image's content SHA-256 and writes manifest version 3. The notebook's "Verify each image's content" box is ticked by default (and set back to ticked, not unticked, when filled values are reset).
  • Explicit opt-out. RunMode(name, verify_content=False), --no-verify-content, or unticking the box. The manifest records it: settings.verify_content is now written by every version 2 and 3 manifest (true or false). Version 1 manifests are unchanged. A manifest written before this change has no key and still reads as not verified.
  • Existing output folders keep the choice of their earlier run ("as in the earlier run, else on"). A legacy folder is therefore not silently recomputed; to start hashing, run once with --verify-content (this recomputes every row, as turning it on always did). A first run, or a rebuild from a version 1 manifest, hashes.
  • The standalone masks command is unchanged; its default still follows the recorded run.
  • Docs updated: docs/contract.md, docs/user_guide.md, CHANGELOG.md, CLI help.

Benchmark

60 OME-TIFF images (2 conditions x 3 repeats x 10; 3 channels, 2048 x 2048, uint16; 25.2 MB each, 1.51 GB), generated locally, 4 vCPU, page cache dropped before every timed run, 3 repetitions (run_experiment, confirmed design, restart mode):

hashing off hashing on overhead
fresh run (read all 60) 16.8 / 15.6 / 15.3 s 20.3 / 19.7 / 19.4 s +3.5 / +4.0 / +4.1 s (+21 to +27 %)
rerun, every row reused 5.4 / 5.5 / 5.3 s 8.8 / 8.2 / 7.9 s +3.4 / +2.7 / +2.6 s (+48 to +63 %)

Raw SHA-256 of the same files, cold: 1.43 s (about 1,060 MB/s). Reading about 1.5 GB takes roughly 1.5 to 4 s extra, so the cost is about 2 to 3 seconds per GB of sources on this machine. It is not a bound for slower storage: on a network drive the extra reads would take longer. The benchmark script is not part of the PR.

Tests

New tests/test_content_hash_default.py, all 4 failing on the old code and passing now:

  • test_a_run_given_no_mode_hashes_every_source_and_records_it
  • test_a_default_run_reads_an_image_rewritten_with_its_metadata_kept
  • test_an_opt_out_is_recorded_as_false_and_the_rewrite_is_not_seen
  • test_the_cli_hashes_by_default_and_no_verify_content_opts_out

Existing tests that relied on the old default now opt out explicitly (version 1 and 2 manifest fixtures, "default run does not see a rewrite" blind-spot tests, the notebook box test, the byte-identity test against the pre-version-3 code, which now allows the one new key). test_the_cli_turns_content_verification_on and similar already used a hashing run and are unchanged.

Checks

  • ruff check . and ruff format --check .: clean.
  • python tools/check.py --rules: passes.
  • pytest: 3525 passed, 4 skipped, 0 failed (main: 3521 passed, 4 skipped).
  • Not run: tools/check.py as a whole, and other Python versions.

Downstream: bricks reading a run through RunInputs already read version 3 manifests (RunInputs.open checks every SHA-256 once per open, which is now the default cost). This was not tested against MorphoFeature in this PR.


Generated by Claude Code

Without it an image edited in place under the same size and modification time was reused as
unchanged. A run now records each image's content SHA-256 unless it opts out
(RunMode(verify_content=False), --no-verify-content, the notebook's box unticked). The manifest
always records the choice (settings.verify_content, true or false). A first run hashes; an
existing output folder keeps the choice of its earlier run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm

# Conflicts:
#	CHANGELOG.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm

# Conflicts:
#	CHANGELOG.md
@guijacquemet
guijacquemet merged commit aec2251 into main Nov 7, 2026
7 checks passed
@guijacquemet
guijacquemet deleted the claude/content-hash-default branch November 7, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Content hashing of sources is off by default

2 participants