diff --git a/AGENTS.md b/AGENTS.md index 7001c12..602900b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,6 +34,8 @@ src/morphobricks_normalization/ bin_centres.py float_bin_centres: bin centres that do not overflow near the float64 limits bin_reachability.py float_bin_indices, reachable_float_bins, centres_map_back: which float bins a value can fall in and whether centres map back (#86) + integer_representatives.py integer bin representatives (nearest float64 midpoint) and + their exact bound for 64-bit values float64 cannot hold (#28) ranges.py HistogramSettings, RangeCheck, range_check_issues (Milestone 5) integrity.py invariants every PlaneHistogram and HistogramTail checks when made (counts, limits, tail sums) and the pixel type of every block diff --git a/CHANGELOG.md b/CHANGELOG.md index 1317811..a7163e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,59 @@ Release notes and migration notes of `morphobricks-normalization`. Numbers are p issues of [CellMigrationLab/morphobricks-normalization](https://github.com/CellMigrationLab/morphobricks-normalization). The decisions are in [docs/plan.md](docs/plan.md) §12. +## Unreleased + +Numerical-correctness fixes. Each says which outputs change; every other output is +bit-identical (`tools/dump_outputs.py` and `--corpus` lines, and the 400 golden fits). + +- **64-bit integers float64 cannot hold (#28).** A bin's representative is the float64 nearest + its midpoint, rounded once (it was rounded twice, `binning_min` first, which could miss a + representable value by 2 above `2**53`). `max_representative_error` of an integer plane + with a value of magnitude `2**51` or more is the exact largest distance between its + integers and the float64 standing in for them, rounded up, so a histogram is exact only + when every representative equals its value: `[2**53 + 1]` is no longer reported exact with + error 0 (its representative is `2**53`; error 1), and `normalization_exact` and + `Histogram_Exact_C` are false for it. Only int64 and uint64 planes with such values + change; bin counts and limits never do. + + **Who is affected, and by how much.** Only int64 and uint64 images with a pixel of + magnitude `2**51` (about `2.25e15`) or more. Every other pixel type (8-, 16- and 32-bit + integers, float32, float64), and 64-bit images whose values all stay below `2**51`, give + bit-identical numbers. For the affected images: + - representatives move by at most one float64 step (at most 1536 for int64 and 2048 for + uint64 values near `9e18` and `1.8e19`, about `1e-16` relative); + - `max_representative_error` is the true largest distance: for bins of one value float64 + cannot hold it grows from 0 to at most half a float64 step (for example 1 for + `[2**53 + 1]`, 170 and 341 for the constant planes `3074457345618258602` and + `6148914691236517205`), so `is_exact`, `normalization_exact` and + `Histogram_Exact_C` change from true to false; for wider bins it changes in its last + bits only; + - pooled means, standard deviations and percentiles, and fitted offsets and scales, move by + at most about `3e-14` relative in the `tools/dump_outputs.py` cases (a zscore offset + `9618469625711740` becomes `9618469625711942`, nearer the exact model mean + `9618469625711938`). + + Statistics files need not be recomputed: they store counts and limits, which do not change, + and representatives and bounds are derived when the file is read, so an existing + `plane_histograms.csv` gives the new numbers. Parameters fitted with v0.4.0 from such + images differ from a refit by the amounts above. +- **Standard deviations of large finite values (#33).** `PooledHistogram.std` and + `WeightedHistogram.std` summed squared deviations that overflow for values beyond about + `1e154` (`[-1e200, 1e200]` gave `inf`, so `zscore` was refused, although its standard + deviation is `1e200`; a tail could raise a bare `OverflowError`). When the plain sum is not + finite but the mean and every value are, it is redone with the values divided by a power of + two, which is exact; every finite result is unchanged bit for bit. A tail whose stored sum + of squares is infinite (schema version 4 keeps raw sums of squares) is refused by name + instead of giving `inf`: its spread is not known. No dump output changes. Not changed: the + raw sums of squares of the file lose precision by cancellation for tails of large values + with a small spread; a centred representation needs a new schema version. +- **Missing conditions per time point (#41).** With `scope="biological_repeat"` and + `per_time_point=True`, `repeat_missing_conditions` checked each repeat group over all its + time points, not the groups that are fitted, so a group such as `r1::t=0` with control + images only gave no warning when treated images of `r1` existed at another time point. + The groups that are fitted are now checked, and the warning's items and message name them + (`r1::t=0`). Without `per_time_point` the warning is unchanged. No dump output changes. + ## v0.4.0 (2026-11-10) Breaking release: `PlaneRecord.position_id` is required (NM-04) and the readers of old table layouts diff --git a/docs/contract.md b/docs/contract.md index 90ab53d..47b8abd 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -120,8 +120,14 @@ Notation in this section: `n` = `bin_count`, `lo` = `binning_min`, `hi` = `binni - If `W > n`: bin `i` holds the values `lo + ceil(i·W/n)` to `lo + ceil((i+1)·W/n) − 1`. Limits are whole numbers, consecutive bins touch without gaps or overlaps, and widths are `floor(W/n)` or `ceil(W/n)` values. -- Representative value of a bin: `(first value + last value) / 2`. +- Representative value of a bin: `(first value + last value) / 2`, the float64 nearest it. - Bound: every pixel is within `(widest bin − 1) / 2` of its representative. +- Planes with a value of magnitude `2**51` or more (64-bit types, #28): float64 cannot hold + every value there (`2**53 + 1` is stored as `2**53`), so the bound is the exact largest + distance between a bin's (or tail's) integers and the float64 standing in for them, rounded + up. A bin is exact only when its representative equals its value, so `normalization_exact` + is false for a group with such a value. Counts and bin limits stay exact integers. Below + `2**51` representatives and bound are exact float64 arithmetic, unchanged. ##### Float planes @@ -192,7 +198,7 @@ tail's mean and its count. | Statistic | Definition | Accuracy | |---|---|---| | min, max | Smallest `pixel_min`, largest `pixel_max` | Exact | -| mean, std | Count-weighted mean and population std (÷N) of representatives; tails count with their exact sums and sums of squares. The weighted sums are correctly rounded sums of the products (`math.fsum`), so they never depend on the CPU thread count or BLAS library (#67) | Exact for exact histograms; otherwise very close (errors cancel) | +| mean, std | Count-weighted mean and population std (÷N) of representatives; tails count with their exact sums and sums of squares. The weighted sums are correctly rounded sums of the products (`math.fsum`), so they never depend on the CPU thread count or BLAS library (#67). When the squares overflow float64 although every value is finite (`[-1e200, 1e200]`), the std is computed with the values divided by a power of two, exactly, so a finite spread is never reported as infinite (#33) | Exact for exact histograms; otherwise very close (errors cancel) | | percentile q | Smallest representative with cumulative count ≥ ceil(q/100 · N) (numpy `inverted_cdf`) | Within the group's largest per-plane bound | | median | 50th percentile | As percentiles | | MAD | Median of \|representative − median\| | Within twice the bound | @@ -254,6 +260,10 @@ pixels keeps its extrema as limits; a 512 × 512 plane has 26 pixels in each tai - Percentiles, median and MAD: each non-empty tail stands as one value, its mean, with its count. - Mean and standard deviation use the tails' sums: as exact as without tails. +- A tail whose sum of squares is not finite (its pixels' squares overflowed float64 when the + file was written, values beyond about `1.3e154`) has no known spread: the standard + deviation, and so `zscore`, is refused with a message naming the tail (#33). Mean, + percentiles, median and MAD do not use it. - Min-max parameters use the exact extrema (`pixel_min`, `pixel_max`). - `max_representative_error` is the larger of the bins' bound and, for each non-empty tail, the larger distance from the tail's mean to its two ends (`max(mean − pixel_min, @@ -456,7 +466,9 @@ y_c = (x_c − offset) / scale for c = R, G, B forms a group with the other conditions' `single` rows, or with the repeat it is paired with. - **Incomplete groups:** a repeat group without rows from every condition of the statistics - gives the warning `repeat_missing_conditions` (§9.2) and is still used. + gives the warning `repeat_missing_conditions` (§9.2) and is still used. With + `per_time_point=true` each time point's group is checked on its own and named in the + warning as `