[guijacquemet]

Ready for review. Do not merge until the owner has reviewed it. On 11 November 2026 the owner decided to start Stage 3 now. Different channel counts across repeats are to be enabled as soon as possible, and nobody uses production yet.

This PR is rebased on main 0b35530, which includes #91 (numerics) and #92 (validation). The owner decided two points the same day:

• Centred tail moments are stored in schema version 5 (N65, commit 9). This was Fixes #28, fixes #33, fixes #41 #91's question.
• Partial channels are refused by default (N66), and the notebook will ask the user explicitly.

The remaining open questions of the draft are resolved below, each on the side the Spec states or the safer one; please confirm or overrule them. The version is not bumped and nothing is tagged. Everything is under CHANGELOG "Unreleased".

I read MorphoBricks-Spec main b4b1008. It differs from the vendored pin cc9820f only in docs/pins.md, so the pin stays at cc9820f (vendor.py --check: identical). Documents read: ADR 0003 HV1–HV4, roadmap Stage 3, conformance §4 "Histogram table v5", experiment-v1 §8 (A_SCOPE_INCOMPLETE) and ADR 0001 SI1.

Commits

1. aacebec plane_histograms.csv schema version 5: channels by canonical channel ID (ADR 0003). This is the draft, rebased.
2. e4df1c2 Tests on described trees: the owner's extra-stain case and Spec fixtures v04, v05, v06, v13, v17, v22. Rebased.
3. faea256 Docs: contract §1.5, §6, §7–§9; plan N61–N64; CHANGELOG. Rebased.
4. 5df10aa Rebase fixes: the tests from Fixes #28, fixes #33, fixes #41 #91 and Fixes #35, fixes #39, fixes #23 #92 use channel IDs.
5. 6c4243a Q3 / N66: refuse partial presence inside a fitted group unless the subset is chosen.
6. e0b7c1d Q4: the parameter columns get their own version.
7. 91f4e75 Q5: a failed measurement removes an earlier table.
8. 5a22848 Docs for the resolutions; tools/evidence_v4_parity.py is deleted.
9. 1a0109d N65: schema v5 tail columns hold centred moments (count, mean, std), computed exactly.

Commit 9: centred tail moments (owner decision N65)

What is stored. Each tail of a robust histogram is written as Histogram_{Underflow,Overflow}Count_<ID>, …Mean_<ID> and …Std_<ID>. These replace the raw float64 …Sum_<ID> and …SumOfSquares_<ID>. In code, HistogramTail(count, mean, std) replaces HistogramTail(count, sum, sum_of_squares).

Why this minimal set. The mean and the population standard deviation are each the float64 nearest their exact value.

• Both are finite for every finite pixel: the mean lies in the tail's range, and the std is at most half that range. So no power-of-two scale column is needed.
• A stored M2 (sum of squared deviations) could still overflow, for example for tails of ±1e200 values, and would then need such a column. The std never does.
• Pooling uses count·std². That is exact up to one rounding, with the same relative accuracy as a stored M2.

How it is computed. The new pure module tail_moments.py takes every tail pixel at its exact value:

• a float as its 53-bit significand times a power of two;
• an integer as a Python int, which is exact for 64-bit values beyond 2**53.

It keeps the sum and the sum of squares as exact rationals across blocks. The mean is rounded once. The std is taken by integer square root with a sticky bit, then rounded once. As a result:

• nothing cancels, where the raw formula did for large values with a small spread;
• nothing overflows, where the raw squares did beyond about 1.3e154;
• a plane gives the same bits read whole or in blocks of any order. Before, block order could change the last bits.

Pooling. Each tail adds count·std² plus count·(mean − pooled mean)². Both terms are centred, and the existing power-of-two rescaling handles overflow (#33). When the tail products in a pooled mean overflow, that mean is rescaled the same way.

What this replaces from #91. #91 had named refusals for an infinite tail sum of squares and for opposite infinite tail sums. Those tails now give their finite mean and std, and zscore works on them.

Validation. integrity.check_tail checks that the mean is finite and inside the tail's range, and that the std is at most half the range.

Tests (tests/test_tail_moments.py), all against an exact Fraction oracle:

• Inputs covered:
  • large values with a small spread;
  • squares beyond float64 (1e200);
  • subnormals;
  • both float64 limits;
  • a constant tail;
  • float32 values;
  • int64 and uint64 values beyond 2**53.
• Expected bits: in every case, the mean and std equal the oracle rounded once.
• Cancellation: on 1e9 + U(0,10), the draft's raw formula is wrong by more than the spread itself; the moments are exact.
• Blocks: any split and any order gives identical bits.
• Square root: correctly rounded.
• Histogram tails: both tails, read whole and in 9 blocks, equal the oracle of their pixels.
• Pooled tails: exact given the stored moments (rel 1e-9).

Tests of the old sums in test_robust_range, test_integrity, test_pooling and test_csv_io were rewritten for the moments. assert_same_histogram now requires blocks to equal whole planes bit for bit.

Cost. The exact accumulation runs at Python speed over the tail pixels only. On a 2048² float32 plane, robust (0.01, 99.99) took 0.194 s against 0.189 s before. With wide quantiles (5, 95), which put 400k pixels in the tails, it took 0.214 s against 0.167 s. This is one local run, not a benchmark.

Contract and docs. Contract §1.2.2, §1.2.3 table, §1.4, §6.2 and §9.1 are updated. Plan §12 records N65 and N66. CHANGELOG and AGENTS are updated. tools/dump_outputs.py dumps (count, mean, std), or the old fields on older commits.

Rebase conflicts and how they were resolved

Both sides' science is kept.

• rgb.py: Fixes #35, fixes #39, fixes #23 #92's RGB_DTYPES and this PR's RGB_CHANNELS / check_rgb_channels are both kept.
• pooling.py: merged automatically. Fixes #28, fixes #33, fixes #41 #91's std rescaling stays; its tail refusals are superseded by N65 as described above.
• from_columns: it moved to parameters.py. It still compares against Fixes #35, fixes #39, fixes #23 #92's percentile_name.
• [Sad Catapella N1][Scientific precision] 'Exact' int64/uint64 histogram becomes inexact when representatives are stored as float64 #28 64-bit representatives: untouched, and their tests now use channel IDs.
• Contract §9.1: Fixes #35, fixes #39, fixes #23 #92's rows are kept.
• CHANGELOG: both sides' entries are kept.
• Code-rules baseline: merged and shrunk. It only shrinks; the last change is histogram.py going from 861 to 835 lines.

Resolutions (for the owner)

#	Question	Resolution	Where
Q1	Which IDs go in the header	The description's declared channels (its [channels] table), never used_ids.channels, which also holds retired IDs. A declared but unused channel (v22) is blank everywhere. The calling brick passes the list. A test checks that a retired ID stays out of the header.	contract §6.1
Q2	Sort order and place of channels	Code point order. channels is the last plane column, before every per-channel column.	contract §6.1, §6.2
Q3	Partial presence inside one fitted group	Owner confirmed (N66): refused by default. fit raises ChannelScopeIncompleteError with the message "A_SCOPE_INCOMPLETE: in group 'R002', channel PHASE is present in k of n planes (absent from …)"; .channels maps each such ID to the planes that lack it. The opt-in is NormalizationSettings(partial_channels="planes_with_channel"). to_dict stores it when it is chosen, and each fit reports the channel_partially_present warning. RGB resets the setting.	contract §1.5, §2, §8.3, §9
Q4	v4-style Normalization_Offset_C0	Each row gets normalization_parameters_version = 2. from_columns refuses a row without it or with another version ("fit again").	contract §8.1, §8.2
Q5	HV3 failure with an existing file	The earlier table is removed, and ChannelMeasurementError.removed names it. Other errors leave an existing file untouched.	contract §6.2, §9.1
Q6	Summary columns by ID now	Kept.	plan §12
Q7	RGB parameter order	Sorted (B, G, R), one rule for every per-channel table. Source order R, G, B applies to a plane's channels cell, pixel arrays and apply.	contract §8.1
Q8	Spec pins.md row	Not edited here (no change to other repositories).	—
#91	Centred tail moments	Done (N65), commit 9.	contract §1.4

Evidence (final head 1a0109d)

Numbers without tails do not change.

• Fully present planes: v13, and v13 with a third channel, give 3600 fitted channels with every histogram cell and fit as float.hex. The output is byte-identical on main 0b35530, on the pre-N65 head 5a22848 and on 1a0109d: sha256 d89c140f…c24c99. v0.4.0 gave the same digest earlier.
• 400 golden fits: pass unchanged. They use the default min-max range, so there are no tails.

tools/dump_outputs.py --corpus (4764 lines):

• main and 5a22848: 8bb7c3c1…72256c.
• 1a0109d: 0111f8bd…6009e.
• Exactly 330 lines differ, and they differ only in the stored tail fields: sum and sum of squares became mean and std. Not one tail mean moves compared with the old sum/count, and no count, limit, representative or bound changes.

tools/dump_outputs.py (1788 lines):

• main: 1723cafb….
• 5a22848: 0b6f6feb…. Against main, that is 600 lines: 60 table digests, 464 lines differing only by the added normalization_parameters_version, 60 lines where Issue.channel is 'C0', and 16 messages.
• 1a0109d: 032b395b…. Against 5a22848, 192 lines differ:
  • 60 table digests (the new tail column names);
  • 100 histogram lines that differ only in the stored tail fields;
  • 32 lines, all on planes with tails, where numbers move by at most 1.4e-14 relative. There are 56 such numbers:
    • the max_representative_error of the int64 and uint64 tight robust planes (whole and blocks now agree bit for bit; before, they differed);
    • their pooled means and high percentiles, by 1 ulp;
    • 14 fits: zscore offsets by 1–2 ulps and int64 percentile scales by 1 ulp. The tail mean is now rounded once from exact values, and the pooled mean is one correctly rounded sum.

Gate and CI.

• Gate python tools/check.py: exit 0, 1754 passed, 0 skipped (CSBDeep 0.8.2 installed, so the two CSBDeep tests ran). Without CSBDeep, those two skip, as on main.
• CI on 1a0109d: 9/9 green (lint, spec-pin, csbdeep, and tests on ubuntu, macOS and Windows with Python 3.11 and 3.13).

What the consumers need (S3.3 and later; not done here)

preprocessing

• Build PlaneRecord(..., channels=) from the unit's confirmed list.
• Call write_plane_histograms(records, path, channel_ids=<declared [channels] IDs>, failures=…), giving every shard the same IDs.
• On ChannelMeasurementError, report .failures and .removed.
• In RunInputs, read with read_plane_histogram_table.
• Drop the N43 refusals.
• Pass channel IDs to range_check_issues.
• Show ChannelScopeIncompleteError as A_SCOPE_INCOMPLETE, and offer partial_channels only as an explicit user choice.
• The new tail columns are written by the library; no change is needed beyond the version.

CellPatcher

• Use record.channel_histogram(id), fit(..., channels=), and apply / select by ID.
• Patch columns are Normalization_Offset_<ID> / Normalization_Scale_<ID> plus normalization_parameters_version. Old tables are refused.
• Handle ChannelScopeIncompleteError.

MorphoFeature (histogram_fit)

• Map catalogue names to record.channels.
• Expect ChannelScopeIncompleteError unless the settings opt in.

All bricks

• Issue.channel is now an ID.
• summary_columns(channel_ids).
• channel_partially_present appears only with the opt-in.
• New: ChannelScopeIncompleteError, NormalizationSettings.partial_channels, the normalization_parameters_version column (COLUMNS has 5 entries), ChannelMeasurementError.removed, records.check_universe.
• HistogramTail(count, mean, std); .sum and .sum_of_squares are gone.
• New module tail_moments.
• Every brick should refuse v5 explicitly until it is updated.

Open questions

1. Release v0.5.0 is not bumped here. The repository's convention is a separate release PR from main after the merge (for example Release v0.4.0 #89 for v0.4.0).
2. Opt-in granularity. partial_channels is one choice per run. A per-channel or per-group choice would be a small follow-up if the notebook needs it.
3. Spec pins.md needs a normalization row at cc9820f.
4. Pooled mean of a fully binned plane whose products overflow (no tails, values near the float64 limit) still gives inf or nan, which the methods refuse by name, unchanged from main. The rescaling that this PR adds for tails could be extended to it as a separate fix; it is not done here to keep untailed numbers and behaviour unchanged.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc

[guijacquemet]

Independent science review of head 1a0109d: approve and merge

I reviewed against ADR 0003 HV1–HV4 (Spec b4b1008), plan §12 N61–N66, the contract, and the owner decisions of 11 November. I found no defects, so this review adds no commits.

Tail moments (N65): checked with my own exact oracle. I wrote the oracle independently of tests/test_tail_moments.py:

• It uses exact Fraction sums.
• The mean is checked as correctly rounded by comparing it with both float neighbours.
• The std is the nearest float to sqrt, checked by squaring the midpoints exactly, with ties to even. No Decimal is involved.

Results:

• 46 tail sets: mean and std match the oracle bit for bit in every set. Each set was also added in splits of 2, 3, 7 and n blocks, in 3 shuffled orders each, and gave the same bits. The sets cover:
  • 1e15 + U(0, 0.1) and 1e9 + U(0, 10);
  • ±1e308 and ±DBL_MAX;
  • subnormals, including an exact sqrt tie at 2.5e-324, which correctly gives 0;
  • constant tails, including 1e300;
  • int64 beyond 253, the int64 extremes and uint64 near 264;
  • float32, including FLT_MAX;
  • mixed magnitudes, -0.0;
  • 30 random sets spanning 2^-1074 to 2^1000.
• 11 robust histograms through the public API (f64, f32, i64, u64, u16, u8, i32, NaN-containing, subnormal-only): both tails equal the oracle of the tail pixels, and a 9-block shuffled read is bit-identical to the whole plane.
• Pooling: the pooled variance, taken about the library's own mean, is within 3.4e-16 relative of the exact value from the stored moments. That holds in every case, including tails of ±DBL_MAX.
• Rescaled mean path: on the ±DBL_MAX tails, the pooled mean has an absolute error of 1.2e-17 pooled std, against an exact pooled-std of 3.9e307. That is negligible, and no exactness is claimed for that path.
• No inf or NaN reaches a fit: I checked min-max, percentile, zscore and robust z-score on ±DBL_MAX planes. Each either gives finite parameters or is refused by name (open question 4, unchanged from main).
• check_tail / _check_tail_range: the mean is checked inside the float tail range, which is sound because rounding is monotone. The std bound is half the range plus slack, so a correctly rounded std is never falsely refused.

v5 table (HV1, HV2, HV4)

• I used the IDs PAX_488, DNA_00001, DNA, Zeta, a. The header is in code-point order (Zeta before a), and the bytes are identical whatever order the channel_ids are given in.
• The header is rebuilt from the Histogram_ValidPixelCount_<ID> columns and compared exactly, so the _NNNNN suffixes cannot be ambiguous.
• In an absent channel's cells, the values 0, nan, NaN, , 0.0, -0 and '' are all refused, in both bin and limit cells.
• A channel listed as present but with blank cells is refused.
• A table with schema_version 4 is refused, and the message names version 5 and says to recompute.

Fits by ID (N66)

• Integer channels are refused, in fit and in channel_histogram.
• A pooled channel uses only that ID's histograms.
• Partial presence raises A_SCOPE_INCOMPLETE by default, with .channels mapping each channel to the planes that lack it.
• The opt-in is stored by to_dict and read back by from_dict. Each partial channel gets a channel_partially_present warning listing its planes.
• RGB resets the opt-in to refuse.

Parameter columns (version 2) and HV3

• from_columns refuses a row without the version column, and rows with version '1', '2.0' and True. A round trip reads back equal.
• HV3 removes the earlier table and names it in .removed.

Evidence: re-run here, all digests match the PR

• I re-ran the deleted evidence_v4_parity.py from 5a22848^: 3600 fitted channels, d89c140f…c24c99, identical on v0.4.0, main 0b35530 and 1a0109d.
• dump_outputs --corpus:
  • main and 5a22848: 8bb7c3c1…;
  • head: 0111f8bd….
  • 330 lines differ, and all of them only in the tail tuples. No tail mean differs from the old sum/count.
• dump_outputs (1788 lines):
  • main: 1723cafb…;
  • 5a22848: 0b6f6feb… (600 lines differ);
  • head: 032b395b….
  • Between 5a22848 and head, 192 lines differ: 60 csv digests, 100 tail-only lines, 18 lines with moved numbers and 14 fits. The largest relative move is 1.36e-14, within the claimed 1.4e-14.

Gate and CI

• Gate (python tools/check.py, Python 3.13, numpy 2.5.4, CSBDeep 0.8.2): 1754 passed, 0 skipped, exit 0.
• CI: 9/9 green.

🤖 Generated with Claude Code

---

Generated by Claude Code