diff --git a/AGENTS.md b/AGENTS.md index cde7114..f6b991d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,8 @@ Read it, and the documents below, before changing anything. |---|---| | `docs/plan.md` | Goals, methods, scopes, milestones, the owner's decisions N1–N60 (§12) and what is still to be measured (§13) | | `docs/contract.md` | Rules other bricks rely on: methods and formulas, interface, scopes and repeat groups, image types, the statistics file, intensity checks, parameter columns, validation | -| `docs/followup_preprocessing.md`, `docs/followup_cellpatcher.md` | What the two calling bricks change once this brick is released | +| `CHANGELOG.md` | Release notes and migration notes | +| `docs/archive/` | Historical record: the two done follow-ups for the calling bricks | ## 1. What the brick is @@ -113,16 +114,13 @@ range check. 2. **Ask before deciding.** Decisions N1–N60 are final (those marked withdrawn in plan §12 no longer apply); do not re-open them. A choice the plan and the contract leave open, and any change of product behaviour or file format, is the owner's: ask with 2–4 options, the recommended one first. -3. **Milestone 1 moves code unchanged.** The histogram, bin-count, record, CSV, pooling - and normalization modules come from morphobricks-preprocessing with only the package - name changed, with their tests, and give bit-identical outputs (contract §1). New - behaviour starts in Milestone 2; Milestone 2b changed only the table layout (contract - §1.3), and `tests/test_schema_v2_equivalence.py` keeps the numbers bit-identical. +3. **Outputs stay bit-identical** unless a change says otherwise. `tests/test_schema_v2_equivalence.py` keeps the + numbers of the version-2 table layout bit-identical (contract §1.3). 4. **Tests** use simulated data generated in the tests; real data only as an optional check. Each formula is tested against a direct implementation (plan §11). 5. **Style** as in morphobricks-preprocessing: Python 3.11+ (MorphoBricks decision D01), `ruff` (line length 100, NumPy docstrings), small modules with one job each, comments that explain why. 6. **Done means** `python tools/check.py` passes (Ruff's findings go through the shrink-only - baseline, so plain `ruff check .` still lists the v0.3.3 debt) and the docs are updated. Report honestly what was verified and what was not. + baseline, so plain `ruff check .` still lists the baselined debt) and the docs are updated. Report honestly what was verified and what was not. 7. **One milestone per branch and PR** against `main`. Do the task asked; list unrelated problems as follow-ups. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..6b36f3f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,237 @@ +# Changelog + +Release notes and migration notes of `morphobricks-normalization`. Numbers are pull requests and +issues of [CellMigrationLab/morphobricks-normalization](https://github.com/CellMigrationLab/morphobricks-normalization). +The decisions are in [docs/plan.md](docs/plan.md) §12. + +## Unreleased (after v0.3.4) + +Merged to `main` since the `v0.3.4` tag; from the commit history, not yet released. Each is a +new refusal or a numerical-range fix (see the pull requests for the exact messages): + +- Named refusals for a non-finite z-score scale and for opposing infinite tail sums in the + pooled mean (#72). +- Percentiles are interpolated between finite extremes without overflowing the gap (#74). +- Float bin centres are computed without overflowing near the float64 limits (#76). +- Image summaries whose channels disagree, pooled summaries with different percentile + definitions, and images whose Z/T planes have different pixel types are refused (#77). +- RGB: channel count and pixel type are checked before histograms are built, RGB groups with + different pixel types are refused before pooling, `Parameters.select` of a strict subset of + R, G and B is refused, and `apply_rgb` uses one float32 copy (#78). +- `range_check_issues` refuses malformed channels and mixed settings (#79). + +## Release notes moved from the README + +The sections below were the README's "Status" section; they are kept as they were written. + +Pre-alpha. Milestone 0 (the contract) is done. Milestone 1 moved the existing code here +from morphobricks-preprocessing unchanged, with identical outputs: + +- Supported pixel types and bin counts (`dtypes`) +- Per-plane histograms, also of planes read in blocks (`histogram`) +- Plane records, plane identifiers and grouping by labels (`records`) +- Reading and writing `plane_histograms.csv` (`csv_io`) +- Group statistics pooled from histograms (`pooling`) +- Min-max, percentile, z-score and robust z-score parameters (`normalization`) + +Milestone 2 added the settings, the scopes and the fitting interface for fluorescence +images: + +- `NormalizationSettings` (image type, method, percentiles, consistency, scope, pooling + weight, per-time-point groups), stored with `to_dict`/`from_dict`, and `check_settings` + (`settings`) +- The `biological_repeat`, `image` and `patch` scopes: `group_key`, `group_name` and + `scope_groups`, with repeat groups matched by name or paired by the calling brick + (`scopes`) +- Pooling weighted per pixel or equally per condition (`WeightedHistogram` in `pooling`) +- `fit` from `plane_histograms.csv` rows or from pixels, and `Parameters` with `apply`, + `to_columns` and `from_columns` (`fitting`) +- Warnings returned as `Issue` objects (`issues`) + +```python +from morphobricks_normalization import NormalizationSettings, check_settings, fit, scope_groups +from morphobricks_normalization import group_name, read_plane_histograms + +settings = NormalizationSettings(scope="biological_repeat", pooling="pixels") +warnings = check_settings(settings, statistics_file=True) +records = read_plane_histograms("plane_histograms.csv") +groups, group_warnings = scope_groups(records, settings, repeat_groups=None) +for key, rows in groups.items(): + parameters = fit(rows, settings, group=group_name(key)) + columns = parameters.to_columns() # normalization_* and Normalization_*_C columns +``` + +Milestone 2b put the channels across columns (decisions N41–N47): + +- `plane_histograms.csv` schema version 3: one row per XYC plane (image, Z and T), with + each channel's extrema, pixel counts and bins in CellProfiler-style columns + (`Intensity_MinIntensity_C0`, `Histogram_Count_C1_00042`, ...). Tables of versions 1 and + 2 are refused with a message asking to run the preprocessing brick again; images with + different numbers of channels are an error. +- `PlaneRecord.histograms` holds one histogram per channel; pooling and fitting take the + channel index. Histograms, pooled statistics and parameters are bit-identical to those of + Milestone 2, checked against a version-2 fixture written by that code (`tests/data`). +- Parameters are written as `Normalization_Offset_C` and `Normalization_Scale_C`, one + pair per channel; `Parameters.select([i])` gives the pair of a single-channel patch. + +Milestone 3 added RGB images, with StarDist's percentile normalization (decisions N33, +N39, N40, N48, N49): + +- `NormalizationSettings(image_type="rgb")` allows only `method="percentile"` (default + 1–99.8, written `rgb percentile 1-99.8`); another method raises `MethodNotAllowedError`. +- `fit` pools the R, G and B values of the group together, at every scope, from + `plane_histograms.csv` rows or from pixels: `offset = P(low)`, `scale = P(high) − P(low) + + 1e-20`, with NumPy's linear-interpolation percentiles (`linear_percentile` in `pooling`, + exact for 8- and 16-bit histograms). The three channels share the values, so colour + ratios are kept; a region without spread becomes zeros instead of an error. +- `Parameters.apply` repeats CSBDeep's float32 arithmetic, so the result equals + `csbdeep.utils.normalize(x, 1, 99.8, axis=(0, 1, 2))` bit for bit (`rgb`). CSBDeep is + not a dependency; the tests compare with it when it is installed. +- RGB needs exactly three channels of an integer type; anything else is refused with a + message that suggests checking the axis mapping or choosing `fluorescence`. +- Fluorescence methods are unchanged, bit for bit. + +```python +settings = NormalizationSettings(image_type="rgb", scope="image") +parameters = fit(rgb_pixels, settings, group="img.tif", channel_axis=-1) # (Y, X, 3) +normalized = parameters.apply(rgb_pixels, channel_axis=-1) +``` + +Milestone 4 added the intensity checks, computed from `plane_histograms.csv` rows alone +(decisions N38, N50–N53): + +- `image_summaries(records)` gives one `ImageSummary` per image, all Z and T planes pooled per + channel: median, 1st and 99.8th percentiles, the fraction of pixels at the pixel type's + maximum and whether the histograms were exact. `to_row()` gives the CellProfiler-style + columns (`Intensity_MedianIntensity_C0`, ...) for a table; the library writes no file. +- `intensity_checks(summaries, settings)` returns warnings: images far from the other images + of their condition and biological repeat (`image_intensity_outlier`), biological repeats + shifted from the other repeats of their condition (`repeat_intensity_shift`), saturated + images and images without spread. Images are compared only within a condition, so a + treatment effect is never reported. Each `Issue` names the image or repeat and the channel + (`Issue.channel`). +- The limits are an `IntensityCheckSettings` (default 3 robust deviations, 1 % saturated + pixels, at least 5 images per comparison, provisional). +- `unchecked_channels(summaries)` names each image channel without finite pixels + (`channel_without_finite_pixels`): its summaries are blank, so the checks cannot see it. + +```python +from morphobricks_normalization import IntensityCheckSettings, image_summaries, intensity_checks + +summaries = image_summaries(records) +table = [summary.to_row() for summary in summaries] # e.g. pandas.DataFrame(table) +for issue in intensity_checks(summaries, IntensityCheckSettings(outlier_limit=3.0)): + print(issue.code, issue.items, issue.channel, issue.message) +``` + +Milestones 2–4 are released as `v0.2.0` (27 October 2026) and Milestone 5 as `v0.3.1` +(28 October 2026); the other bricks pin the latest tag. The tag `v0.3.0` holds the same +code but reports version 0.2.0, as its `pyproject.toml` was not updated; use `v0.3.1`. +`v0.3.2` (30 October 2026) refuses a `plane_histograms.csv` row whose `plane_id`, `image_file` +or `bit_depth` disagrees with its other columns (#18). +`v0.3.3` (3 November 2026) refuses corrupt or ambiguous statistics and parameters instead +of using them (issues #25, #26, #31, #36–#38, #42–#48); valid inputs give bit-identical +results. See [Migrating to v0.3.3](#migrating-to-v033). +`v0.3.4` (5 November 2026) refuses a histogram whose missing `pixel_min` or `pixel_max` used +to be filled in (#47, PR #65), makes means, standard deviations and z-score fits independent +of the CPU thread count, changing their last bits (#67, PR #69), and adds the section 9 +code-standards gate (PR #66). See [Migrating to v0.3.4](#migrating-to-v034). + +Milestone 5 added a robust histogram range for histograms that are not exact (float planes, +and integer planes with more values than bins), for issue #12 (decisions N54–N60): + +- By default nothing changes: bins span each plane's minimum to maximum. +- `HistogramSettings(histogram_range="robust")` bins between the plane's 0.01 % and 99.99 % + quantiles (adjustable) and keeps the pixels outside as underflow and overflow counts and + sums, so one hot pixel no longer squeezes the other pixels into a few bins. Exact 8- and + 16-bit histograms never change. +- `plane_histograms.csv` schema version 4 adds `Histogram_UnderflowCount_C`, + `Histogram_UnderflowSum_C`, `Histogram_UnderflowSumOfSquares_C` and the same for + `Overflow`; tables of version 3 are refused with a message asking to run the + preprocessing brick again. +- `histogram_with_range_check` computes a plane's histogram together with a `RangeCheck`: how + many bins its central range (1st to 99.8th percentile) covers, and how many it would cover + with the robust range. `range_check_issues` turns the checks into one warning per channel + for planes below 100 bins, for the preprocessing brick to show. + +```python +from morphobricks_normalization import HistogramSettings, histogram_with_range_check +from morphobricks_normalization import range_check_issues + +settings = HistogramSettings(histogram_range="minmax") # or "robust" +histogram, check = histogram_with_range_check(lambda: (plane,), plane.dtype, settings=settings) +warnings = range_check_issues([("img.tif::z=0,t=0", 0, check)]) +``` + +### v0.3.4 (5 November 2026) + +- **`PlaneHistogram` refuses a missing extremum (#65, issue #47)**: it no longer fills a + missing `pixel_min` or `pixel_max` from `binning_min` or `binning_max`. A caller that + builds a histogram of a plane with valid pixels passes both extrema explicitly, e.g. + `pixel_min=lo, pixel_max=hi` for a min-max histogram. A `plane_histograms.csv` row with + one blank intensity limit and valid pixels is refused with a `CsvFormatError` naming the + column. This change alone leaves every output for valid inputs bitwise unchanged. +- **Means and standard deviations no longer depend on CPU threads (#69, issue #67).** They + summed with `np.dot`, which the BLAS library splits across its threads, so a z-score fit + changed in its last bits with `OMP_NUM_THREADS` or `OPENBLAS_NUM_THREADS`. They now add the + same products with `math.fsum` (the correctly rounded sum): the same bits for any thread + count, BLAS or platform, never less accurate than before. A sum that overflows float64 is + still `nan` or `inf`, refused as before. The last bits change (see the migration note). +- **Section 9 code-standards gate (#66)**: `python tools/check.py` runs the shared code + rules, Ruff and pytest against a shrink-only baseline (`tools/code_rules_baseline.json`, + taken at `v0.3.3`); CI runs its `--rules` part. No library behaviour changes. + +### Migrating to v0.3.4 + +Results are **not** all bit-identical to v0.3.3: + +- **Mean, standard deviation and z-score change in their last bits (#67).** The pooled + `mean` and `std` (`PooledHistogram`, `WeightedHistogram`, `pooled_statistics`) and the + `zscore` offset and scale, at every scope, are now the correctly rounded sums of the same + products, so they can differ from v0.3.3 by a few units in the last place: at most + 3.7e-14 relative on `tools/dump_outputs.py` (134 numbers on 86 of its 1,788 lines; 89 of + them by 1 ulp, the largest by 317 ulps, for int64 planes with values near 2**53). Cached + z-score parameters or features computed from them with v0.3.3 are not bitwise + reproducible with v0.3.4; recompute them rather than mixing the two. Every other output + (histograms, `plane_histograms.csv`, minimum, maximum, percentiles, median, MAD, the + min-max, percentile, robust z-score and RGB methods) is bitwise unchanged. + +One input that was accepted before is now refused: + +- **Histograms with a missing extremum:** a `PlaneHistogram` of a plane with valid pixels + whose `pixel_min` or `pixel_max` is `None` raises `ValueError` (`CsvFormatError` when read + from `plane_histograms.csv`, naming the blank column). Earlier versions filled it from + `binning_min` or `binning_max`. Pass the plane's actual extrema, e.g. + `PlaneHistogram(..., pixel_min=lo, pixel_max=hi)`. A channel without valid pixels may still + leave both blank. + +### Migrating to v0.3.3 + +No valid statistics change: on the same inputs, histograms, `plane_histograms.csv` bytes, +pooled statistics, fitted parameters and normalized pixels are bit-identical to `v0.3.2`. +These inputs are now refused with `ValueError` (`CsvFormatError` when read from the file): + +- **Pixel blocks:** a non-empty block of another pixel type than the plane's. A block in the + other byte order (big-endian TIFF) is now converted by value; 16-bit ones were misread. +- **Histograms:** negative or wrapping bin counts, `valid > total`, missing or disordered + intensity limits on a plane with valid pixels, NaN or impossible underflow/overflow sums. + `PlaneHistogram.counts` must be a one-dimensional int64 array. +- **Tables:** a `plane_histograms.csv` with the header but no rows; two rows with the same + `plane_id` (also refused by `write_plane_histograms`, `scope_groups`, `fit` and + `image_summaries`). +- **Plane coordinates:** `PlaneRecord` `z` and `t` must be whole numbers 0 or above (`None` + is still 0); booleans, floats (even `1.0`), text and negative values are refused. +- **Repeat groups:** `repeat_groups` that put two biological repeats of one condition in one + repeat group, or give a present pair an empty or non-text label. +- **Pooling:** a group of more than 2**63 − 1 finite pixels. Counts above 2**53 are now + exact. +- **Parameters:** a non-finite offset or a scale that is not finite and above 0 + (`Parameters`, `NormalizationParameters`); a channel index that is not a whole number 0 or + above (booleans, `1.0`, `"1"` are no longer truncated). +- **`Parameters.from_columns`:** a `normalization_scope` that is not a scope, or that differs + from `settings.scope` (#37). Pass the settings of the run that wrote the row, scope included, + e.g. `NormalizationSettings(method=..., percentile=..., scope=run_scope)`. +- **Summaries:** `ImageSummary.exact` (`Histogram_Exact_C`) is `None`, not `True`, for a + channel without finite pixels. The new `unchecked_channels(summaries)` returns a + `channel_without_finite_pixels` warning for each such channel; `intensity_checks` still + returns its four codes only, so calling bricks add the new code themselves (#57). diff --git a/README.md b/README.md index 6e1c197..c0e3bfd 100644 --- a/README.md +++ b/README.md @@ -29,29 +29,16 @@ out of scope for now. ## Status -Pre-alpha. Milestone 0 (the contract) is done. Milestone 1 moved the existing code here -from morphobricks-preprocessing unchanged, with identical outputs: +Pre-alpha; the latest release is v0.3.4. The library offers: -- Supported pixel types and bin counts (`dtypes`) -- Per-plane histograms, also of planes read in blocks (`histogram`) -- Plane records, plane identifiers and grouping by labels (`records`) -- Reading and writing `plane_histograms.csv` (`csv_io`) -- Group statistics pooled from histograms (`pooling`) -- Min-max, percentile, z-score and robust z-score parameters (`normalization`) +- supported pixel types, bin counts and per-plane histograms (`dtypes`, `histogram`), with a robust range and a range check; +- plane records, `plane_histograms.csv` reading and writing (schema version 4) and pooling of group statistics (`records`, `csv_io`, `pooling`); +- settings, scopes, repeat groups and `fit`, with `Parameters` (`apply`, `to_columns`, `from_columns`), for fluorescence and RGB images; +- per-image summaries and intensity checks, returned as `Issue` warnings. -Milestone 2 added the settings, the scopes and the fitting interface for fluorescence -images: +Release notes and migration notes are in [CHANGELOG.md](CHANGELOG.md). -- `NormalizationSettings` (image type, method, percentiles, consistency, scope, pooling - weight, per-time-point groups), stored with `to_dict`/`from_dict`, and `check_settings` - (`settings`) -- The `biological_repeat`, `image` and `patch` scopes: `group_key`, `group_name` and - `scope_groups`, with repeat groups matched by name or paired by the calling brick - (`scopes`) -- Pooling weighted per pixel or equally per condition (`WeightedHistogram` in `pooling`) -- `fit` from `plane_histograms.csv` rows or from pixels, and `Parameters` with `apply`, - `to_columns` and `from_columns` (`fitting`) -- Warnings returned as `Issue` objects (`issues`) +Example: ```python from morphobricks_normalization import NormalizationSettings, check_settings, fit, scope_groups @@ -66,181 +53,6 @@ for key, rows in groups.items(): columns = parameters.to_columns() # normalization_* and Normalization_*_C columns ``` -Milestone 2b put the channels across columns (decisions N41–N47): - -- `plane_histograms.csv` schema version 3: one row per XYC plane (image, Z and T), with - each channel's extrema, pixel counts and bins in CellProfiler-style columns - (`Intensity_MinIntensity_C0`, `Histogram_Count_C1_00042`, ...). Tables of versions 1 and - 2 are refused with a message asking to run the preprocessing brick again; images with - different numbers of channels are an error. -- `PlaneRecord.histograms` holds one histogram per channel; pooling and fitting take the - channel index. Histograms, pooled statistics and parameters are bit-identical to those of - Milestone 2, checked against a version-2 fixture written by that code (`tests/data`). -- Parameters are written as `Normalization_Offset_C` and `Normalization_Scale_C`, one - pair per channel; `Parameters.select([i])` gives the pair of a single-channel patch. - -Milestone 3 added RGB images, with StarDist's percentile normalization (decisions N33, -N39, N40, N48, N49): - -- `NormalizationSettings(image_type="rgb")` allows only `method="percentile"` (default - 1–99.8, written `rgb percentile 1-99.8`); another method raises `MethodNotAllowedError`. -- `fit` pools the R, G and B values of the group together, at every scope, from - `plane_histograms.csv` rows or from pixels: `offset = P(low)`, `scale = P(high) − P(low) + - 1e-20`, with NumPy's linear-interpolation percentiles (`linear_percentile` in `pooling`, - exact for 8- and 16-bit histograms). The three channels share the values, so colour - ratios are kept; a region without spread becomes zeros instead of an error. -- `Parameters.apply` repeats CSBDeep's float32 arithmetic, so the result equals - `csbdeep.utils.normalize(x, 1, 99.8, axis=(0, 1, 2))` bit for bit (`rgb`). CSBDeep is - not a dependency; the tests compare with it when it is installed. -- RGB needs exactly three channels of an integer type; anything else is refused with a - message that suggests checking the axis mapping or choosing `fluorescence`. -- Fluorescence methods are unchanged, bit for bit. - -```python -settings = NormalizationSettings(image_type="rgb", scope="image") -parameters = fit(rgb_pixels, settings, group="img.tif", channel_axis=-1) # (Y, X, 3) -normalized = parameters.apply(rgb_pixels, channel_axis=-1) -``` - -Milestone 4 added the intensity checks, computed from `plane_histograms.csv` rows alone -(decisions N38, N50–N53): - -- `image_summaries(records)` gives one `ImageSummary` per image, all Z and T planes pooled per - channel: median, 1st and 99.8th percentiles, the fraction of pixels at the pixel type's - maximum and whether the histograms were exact. `to_row()` gives the CellProfiler-style - columns (`Intensity_MedianIntensity_C0`, ...) for a table; the library writes no file. -- `intensity_checks(summaries, settings)` returns warnings: images far from the other images - of their condition and biological repeat (`image_intensity_outlier`), biological repeats - shifted from the other repeats of their condition (`repeat_intensity_shift`), saturated - images and images without spread. Images are compared only within a condition, so a - treatment effect is never reported. Each `Issue` names the image or repeat and the channel - (`Issue.channel`). -- The limits are an `IntensityCheckSettings` (default 3 robust deviations, 1 % saturated - pixels, at least 5 images per comparison, provisional). -- `unchecked_channels(summaries)` names each image channel without finite pixels - (`channel_without_finite_pixels`): its summaries are blank, so the checks cannot see it. - -```python -from morphobricks_normalization import IntensityCheckSettings, image_summaries, intensity_checks - -summaries = image_summaries(records) -table = [summary.to_row() for summary in summaries] # e.g. pandas.DataFrame(table) -for issue in intensity_checks(summaries, IntensityCheckSettings(outlier_limit=3.0)): - print(issue.code, issue.items, issue.channel, issue.message) -``` - -Milestones 2–4 are released as `v0.2.0` (27 October 2026) and Milestone 5 as `v0.3.1` -(28 October 2026); the other bricks pin the latest tag. The tag `v0.3.0` holds the same -code but reports version 0.2.0, as its `pyproject.toml` was not updated; use `v0.3.1`. -`v0.3.2` (30 October 2026) refuses a `plane_histograms.csv` row whose `plane_id`, `image_file` -or `bit_depth` disagrees with its other columns (#18). -`v0.3.3` (3 November 2026) refuses corrupt or ambiguous statistics and parameters instead -of using them (issues #25, #26, #31, #36–#38, #42–#48); valid inputs give bit-identical -results. See [Migrating to v0.3.3](#migrating-to-v033). -`v0.3.4` (5 November 2026) refuses a histogram whose missing `pixel_min` or `pixel_max` used -to be filled in (#47, PR #65), makes means, standard deviations and z-score fits independent -of the CPU thread count, changing their last bits (#67, PR #69), and adds the section 9 -code-standards gate (PR #66). See [Migrating to v0.3.4](#migrating-to-v034). - -Milestone 5 added a robust histogram range for histograms that are not exact (float planes, -and integer planes with more values than bins), for issue #12 (decisions N54–N60): - -- By default nothing changes: bins span each plane's minimum to maximum. -- `HistogramSettings(histogram_range="robust")` bins between the plane's 0.01 % and 99.99 % - quantiles (adjustable) and keeps the pixels outside as underflow and overflow counts and - sums, so one hot pixel no longer squeezes the other pixels into a few bins. Exact 8- and - 16-bit histograms never change. -- `plane_histograms.csv` schema version 4 adds `Histogram_UnderflowCount_C`, - `Histogram_UnderflowSum_C`, `Histogram_UnderflowSumOfSquares_C` and the same for - `Overflow`; tables of version 3 are refused with a message asking to run the - preprocessing brick again. -- `histogram_with_range_check` computes a plane's histogram together with a `RangeCheck`: how - many bins its central range (1st to 99.8th percentile) covers, and how many it would cover - with the robust range. `range_check_issues` turns the checks into one warning per channel - for planes below 100 bins, for the preprocessing brick to show. - -```python -from morphobricks_normalization import HistogramSettings, histogram_with_range_check -from morphobricks_normalization import range_check_issues - -settings = HistogramSettings(histogram_range="minmax") # or "robust" -histogram, check = histogram_with_range_check(lambda: (plane,), plane.dtype, settings=settings) -warnings = range_check_issues([("img.tif::z=0,t=0", 0, check)]) -``` - -### v0.3.4 (5 November 2026) - -- **`PlaneHistogram` refuses a missing extremum (#65, issue #47)**: it no longer fills a - missing `pixel_min` or `pixel_max` from `binning_min` or `binning_max`. A caller that - builds a histogram of a plane with valid pixels passes both extrema explicitly, e.g. - `pixel_min=lo, pixel_max=hi` for a min-max histogram. A `plane_histograms.csv` row with - one blank intensity limit and valid pixels is refused with a `CsvFormatError` naming the - column. This change alone leaves every output for valid inputs bitwise unchanged. -- **Means and standard deviations no longer depend on CPU threads (#69, issue #67).** They - summed with `np.dot`, which the BLAS library splits across its threads, so a z-score fit - changed in its last bits with `OMP_NUM_THREADS` or `OPENBLAS_NUM_THREADS`. They now add the - same products with `math.fsum` (the correctly rounded sum): the same bits for any thread - count, BLAS or platform, never less accurate than before. A sum that overflows float64 is - still `nan` or `inf`, refused as before. The last bits change (see the migration note). -- **Section 9 code-standards gate (#66)**: `python tools/check.py` runs the shared code - rules, Ruff and pytest against a shrink-only baseline (`tools/code_rules_baseline.json`, - taken at `v0.3.3`); CI runs its `--rules` part. No library behaviour changes. - -### Migrating to v0.3.4 - -Results are **not** all bit-identical to v0.3.3: - -- **Mean, standard deviation and z-score change in their last bits (#67).** The pooled - `mean` and `std` (`PooledHistogram`, `WeightedHistogram`, `pooled_statistics`) and the - `zscore` offset and scale, at every scope, are now the correctly rounded sums of the same - products, so they can differ from v0.3.3 by a few units in the last place: at most - 3.7e-14 relative on `tools/dump_outputs.py` (134 numbers on 86 of its 1,788 lines; 89 of - them by 1 ulp, the largest by 317 ulps, for int64 planes with values near 2**53). Cached - z-score parameters or features computed from them with v0.3.3 are not bitwise - reproducible with v0.3.4; recompute them rather than mixing the two. Every other output - (histograms, `plane_histograms.csv`, minimum, maximum, percentiles, median, MAD, the - min-max, percentile, robust z-score and RGB methods) is bitwise unchanged. - -One input that was accepted before is now refused: - -- **Histograms with a missing extremum:** a `PlaneHistogram` of a plane with valid pixels - whose `pixel_min` or `pixel_max` is `None` raises `ValueError` (`CsvFormatError` when read - from `plane_histograms.csv`, naming the blank column). Earlier versions filled it from - `binning_min` or `binning_max`. Pass the plane's actual extrema, e.g. - `PlaneHistogram(..., pixel_min=lo, pixel_max=hi)`. A channel without valid pixels may still - leave both blank. - -### Migrating to v0.3.3 - -No valid statistics change: on the same inputs, histograms, `plane_histograms.csv` bytes, -pooled statistics, fitted parameters and normalized pixels are bit-identical to `v0.3.2`. -These inputs are now refused with `ValueError` (`CsvFormatError` when read from the file): - -- **Pixel blocks:** a non-empty block of another pixel type than the plane's. A block in the - other byte order (big-endian TIFF) is now converted by value; 16-bit ones were misread. -- **Histograms:** negative or wrapping bin counts, `valid > total`, missing or disordered - intensity limits on a plane with valid pixels, NaN or impossible underflow/overflow sums. - `PlaneHistogram.counts` must be a one-dimensional int64 array. -- **Tables:** a `plane_histograms.csv` with the header but no rows; two rows with the same - `plane_id` (also refused by `write_plane_histograms`, `scope_groups`, `fit` and - `image_summaries`). -- **Plane coordinates:** `PlaneRecord` `z` and `t` must be whole numbers 0 or above (`None` - is still 0); booleans, floats (even `1.0`), text and negative values are refused. -- **Repeat groups:** `repeat_groups` that put two biological repeats of one condition in one - repeat group, or give a present pair an empty or non-text label. -- **Pooling:** a group of more than 2**63 − 1 finite pixels. Counts above 2**53 are now - exact. -- **Parameters:** a non-finite offset or a scale that is not finite and above 0 - (`Parameters`, `NormalizationParameters`); a channel index that is not a whole number 0 or - above (booleans, `1.0`, `"1"` are no longer truncated). -- **`Parameters.from_columns`:** a `normalization_scope` that is not a scope, or that differs - from `settings.scope` (#37). Pass the settings of the run that wrote the row, scope included, - e.g. `NormalizationSettings(method=..., percentile=..., scope=run_scope)`. -- **Summaries:** `ImageSummary.exact` (`Histogram_Exact_C`) is `None`, not `True`, for a - channel without finite pixels. The new `unchecked_channels(summaries)` returns a - `channel_without_finite_pixels` warning for each such channel; `intensity_checks` still - returns its four codes only, so calling bricks add the new code themselves (#57). - ## Installation ```bash @@ -260,8 +72,7 @@ python -m pytest |---|---| | [docs/plan.md](docs/plan.md) | Goals, milestones and the owner's decisions N1–N60 | | [docs/contract.md](docs/contract.md) | Rules other bricks rely on: methods, scopes and repeat groups, image types, the statistics file, intensity checks, parameter columns, validation | -| [docs/followup_preprocessing.md](docs/followup_preprocessing.md) | Changes to morphobricks-preprocessing | -| [docs/followup_cellpatcher.md](docs/followup_cellpatcher.md) | Changes to CellPatcher | +| [CHANGELOG.md](CHANGELOG.md) | Release notes and migration notes | | [AGENTS.md](AGENTS.md) | Rules for anyone changing this repository | ## Related repositories diff --git a/docs/followup_cellpatcher.md b/docs/archive/followup_cellpatcher.md similarity index 97% rename from docs/followup_cellpatcher.md rename to docs/archive/followup_cellpatcher.md index bf79a5d..6deb2ca 100644 --- a/docs/followup_cellpatcher.md +++ b/docs/archive/followup_cellpatcher.md @@ -1,6 +1,8 @@ +> Historical record, not current. Done; merged in CellPatcher PRs 29 to 38. + # Follow-up Plan — CellPatcher -> See also: [Normalization brick plan](plan.md) · [Preprocessing follow-up](followup_preprocessing.md) +> See also: [Normalization brick plan](../plan.md) · [Preprocessing follow-up](followup_preprocessing.md) **Status:** done (29 October 2026). C1–C5 were merged into CellPatcher's `main` (pull requests [29](https://github.com/CellMigrationLab/CellPatcher/pull/29), @@ -10,7 +12,7 @@ [38](https://github.com/CellMigrationLab/CellPatcher/pull/38)), with the `preprocessing` extra pinned to preprocessing `v0.3.0`, which brings this brick's `v0.3.1` ([34](https://github.com/CellMigrationLab/CellPatcher/pull/34)). Decision numbers N refer to §12 of the -[normalization plan](plan.md); D-numbers refer to CellPatcher's own `docs/plan.md`, and this +[normalization plan](../plan.md); D-numbers refer to CellPatcher's own `docs/plan.md`, and this plan revises some of them (see the last section). **Repository:** [CellMigrationLab/CellPatcher](https://github.com/CellMigrationLab/CellPatcher) diff --git a/docs/followup_preprocessing.md b/docs/archive/followup_preprocessing.md similarity index 97% rename from docs/followup_preprocessing.md rename to docs/archive/followup_preprocessing.md index 901268b..33debee 100644 --- a/docs/followup_preprocessing.md +++ b/docs/archive/followup_preprocessing.md @@ -1,10 +1,12 @@ +> Historical record, not current. Done; released in morphobricks-preprocessing v0.2.0 and v0.3.0; see its CHANGELOG. + # Follow-up Plan — morphobricks-preprocessing -> See also: [Normalization brick plan](plan.md) · [CellPatcher follow-up](followup_cellpatcher.md) +> See also: [Normalization brick plan](../plan.md) · [CellPatcher follow-up](followup_cellpatcher.md) **Status:** done (28 October 2026). P1–P5 were released in preprocessing `v0.2.0` and P6 in `v0.3.0` (pinning this brick's `v0.3.1`). The CellPatcher follow-up comes next. Decision -numbers refer to §12 of the [normalization plan](plan.md). +numbers refer to §12 of the [normalization plan](../plan.md). **Repository:** [CellMigrationLab/morphobricks-preprocessing](https://github.com/CellMigrationLab/morphobricks-preprocessing) ## Goal diff --git a/docs/contract.md b/docs/contract.md index 70a4d47..dca7379 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -1,7 +1,7 @@ # Normalization Brick Contract -> See also: [Plan](plan.md) · [Preprocessing follow-up](followup_preprocessing.md) · -> [CellPatcher follow-up](followup_cellpatcher.md) +> See also: [Plan](plan.md) · [Preprocessing follow-up](archive/followup_preprocessing.md) (done) · +> [CellPatcher follow-up](archive/followup_cellpatcher.md) (done) This document fixes the rules other bricks can rely on: the methods, the scopes and repeat groups, the image types, the statistics file, the intensity checks, the parameter columns and @@ -21,7 +21,7 @@ with modality profiles, purpose rules, colour statistics and references is withd | 3.3 RGB method; 5. Image types | Milestone 3 (done) | | 1.3 and 6. Channels across columns (`plane_histograms.csv` schema version 3); per-channel columns of 7.1 and 8.1 | Milestone 2b (done; the columns of 7.1 in Milestone 4, done) | | 7. Intensity checks | Milestone 4 (done) | -| 1.4 Robust histogram range and range check; `plane_histograms.csv` schema version 4 (6.2) | Milestone 5 | +| 1.4 Robust histogram range and range check; `plane_histograms.csv` schema version 4 (6.2) | Milestone 5 (done) | | 9. Validation | With the milestone of each check | Notation: `x` is a pixel value in source units; `n` a pixel count; channels are numbered diff --git a/docs/plan.md b/docs/plan.md index 49597ce..b2c6af1 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1,8 +1,8 @@ # MorphoBricks Normalization Brick — Implementation Plan -> See also: [Preprocessing follow-up](followup_preprocessing.md) · [CellPatcher follow-up](followup_cellpatcher.md) +> See also: [Preprocessing follow-up](archive/followup_preprocessing.md) (done) · [CellPatcher follow-up](archive/followup_cellpatcher.md) (done) -**Status:** design brief, simplified on 26 October 2026 (decisions N29–N40, §12), with +**Status:** implemented through v0.3.4 (Milestones 0 to 5 done; later changes in [CHANGELOG.md](../CHANGELOG.md)); design brief simplified on 26 October 2026 (decisions N29–N40, §12), with channels moved into columns on 27 October 2026 (N41–N45). The earlier design with six modalities, two purposes, colour and stain methods and reference targets (N4–N28) is withdrawn where §12 says so. A robust histogram range was added on 28 October