[guijacquemet]

Bucket-A integrity fixes, statistics side: refuse corrupt pixel blocks, counts, ranges, tails and plane identity. The other half (parameters, scope column, unchecked channels) is #62. The two PRs merge cleanly into each other.

Fixes #25, fixes #26, fixes #31, fixes #42, fixes #43, fixes #44, fixes #45, fixes #46, fixes #47, fixes #48

Changes

• Blocks ([Sad Catapella N1][Correctness] Big-endian int8/int16 histogram blocks are reinterpreted as native-endian pixel values #25, [Sad Catapella N1][Correctness] Block histogram accepts dtype-mismatched pixel arrays and can silently change pixel cardinality #26): integrity.checked_blocks is used in every pass of histogram.py. A non-empty block of another pixel type (kind, width or signedness) raises ValueError naming the block and both types. A block in the other byte order is converted by value. >u2/>i2 planes were previously read as native-endian bytes, which gave min/max 256/768 for the values 1..3.

• Histogram invariants ([Sad Catapella N5][Scientific integrity] CSV accepts negative and int64-overflowing histogram bin counts #45, [Sad Catapella N5][Correctness] Nonempty histogram with all intensity limits missing passes CSV validation #47), new integrity.py: PlaneHistogram.__post_init__ checks the following:

  • counts are a 1-D int64 array of bin_count non-negative values;
  • total and valid are whole numbers from 0 to 2**63-1, with valid ≤ total;
  • bins + tails == valid, using an exact sum that cannot wrap ([2**62]*4+[3] is caught);
  • with no valid pixels, all limits are blank and the tails empty;
  • with valid pixels, all four limits are finite and ordered, and a limit differs from its extremum exactly when a tail lies beyond it.

• Tails ([Sad Catapella N5][Scientific integrity] CSV parser accepts NaN, infinity and impossible nonempty tail moments #46): HistogramTail.__post_init__ refuses:

  • a NaN sum;
  • a negative or NaN sum of squares;
  • an infinite sum with a finite sum of squares;
  • n·ssq < sum².

  PlaneHistogram also checks that each tail's mean and sum of squares are possible within its range (relative slack 1e-9). A float64 overflow of the sum of squares ([Sad Catapella N2][Scientific correctness] Tail second moments and pooled standard deviation can overflow despite finite true spread #33) is accepted only where the range makes it possible.

• CSV:

  • a header-only v4 table is refused ([Sad Catapella N5][Data integrity] Header-only schema-v4 CSV is accepted as an empty experiment #48);
  • a repeated plane_id row is refused, naming both lines ([Sad Catapella N4][Scientific integrity] Duplicate plane_id rows are accepted and silently double-counted in normalization #43);
  • text counts above int64 now raise CsvFormatError instead of an uncaught OverflowError;
  • errors name the line and channel.

  The old _consistent_range has moved into the histogram invariants, and _record is down from 72 to 56 lines.

• Repeated planes ([Sad Catapella N4][Scientific integrity] Duplicate plane_id rows are accepted and silently double-counted in normalization #43): write_plane_histograms, scope_groups, fit and image_summaries also refuse repeated plane_ids (records.duplicate_planes).

• Coordinates ([Sad Catapella N4][Data integrity] PlaneRecord and CSV reader accept negative plane coordinates; direct records also accept fractional coordinates #44): PlaneRecord z/t must be whole numbers ≥ 0. None still becomes 0; bool, float (including 1.0), text and negative values are refused. The CSV reader refuses -1.

• Pooling ([Sad Catapella N2][Correctness] PooledHistogram loses exact pixel counts through float64 weighted bincount #31): PooledHistogram merges counts with integer np.add.at, not float64 bincount weights, so counts above 253 stay exact. A group above 263-1 pixels is refused. The float policy of WeightedHistogram is now documented.

• Repeat groups ([Sad Catapella N4][Scientific identity] scope_groups accepts two biological repeats of one condition in the same paired group #42): scope_groups refuses two biological repeats of one condition mapped to one repeat-group label, and empty or non-text labels. It does not normalize labels. Unused mapping entries are still ignored.

• Docs: contract §1.2.2 invariants, §4.2 pairing, §6.2, §9.1 rows; AGENTS map.

• One existing test changed: test_failed_write_leaves_previous_file_untouched used to write the same plane twice, which is the [Sad Catapella N4][Scientific integrity] Duplicate plane_id rows are accepted and silently double-counted in normalization #43 defect. It now writes two different planes.

Evidence

• Regression tests: tests/test_integrity.py and two tests in test_scopes.py. Run against main (f784c95), 70 of the 91 integrity tests and all 4 new scope tests fail. The remaining 21 are controls that check valid inputs are still accepted.
• Bitwise comparison: a dump of 5,769 outputs gives the same SHA-256 f7604ef1…c091417 on main, this branch and main + both PRs. It covers:
  • histograms for 10 dtypes × 3 bin counts × min-max/robust, whole, blockwise and with the range check;
  • CSV bytes and read-back;
  • pooled statistics, the linear percentile and WeightedHistogram;
  • fit, apply and from_columns round trips for 4 methods × 4 scope/pooling/per-time combinations;
  • pixel and patch fits, RGB, summaries and checks.
• Gate:
  • ruff check . passes;
  • ruff format --check . passes;
  • ruff check --ignore-noqa . reports 3 E402 in tests/data/make_v2_fixture.py, the same 3 as on main;
  • pytest: 569 passed, 2 skipped (csbdeep not installed locally; the CI csbdeep job passed), 0 failed.
  • There is no tools/check.py or tools/code_rules.py in this repo.
• Code standards:
  • histogram.py was already over the limit at 863 lines and is now 864. The _small_integer_histogram refactor offsets most of the new lines.
  • New and changed functions are ≤ 60 lines, C901 ≤ 15 and ≤ 8 parameters.
• Performance: PooledHistogram of 200 uint16 planes takes 483 ms here against 509 ms on main (best of 5). This is a sanity check, not a performance claim.
• Consumers: the full test suites of CellPatcher 438e6d4, MorphoFeature 9e66ca6 and morphobricks-preprocessing 83ce987, run against main + both PRs, have the same pass/fail sets as against main. The pre-existing failures are dependency-pin and installer tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc