From ebc5cd171dad50eb07f3c574d0df0e2f026e923c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 11 Nov 2026 10:18:52 +0000 Subject: [PATCH] Release v0.5.0: version, release notes and migration notes pyproject 0.5.0; CHANGELOG 'Unreleased' becomes v0.5.0 (2026-11-11) with a 'Migrating to v0.5.0' section for the consuming bricks (preprocessing, CellPatcher, MorphoFeature): schema version 5, channels by ID, centred tail moments, A_SCOPE_INCOMPLETE and the partial_channels opt-in, parameter columns version 2, HV3 removal, v4 hard cut. README release line and install command pinned to @v0.5.0; plan status, S3.2 entry and a v0.5.0 release entry. No code change; dump_outputs (plain and --corpus) identical to main. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01G3qp9AxjpGPw9xfWAUJNqc --- CHANGELOG.md | 80 +++++++++++++++++++++++++++++++++++++++++++++++--- README.md | 6 ++-- docs/plan.md | 15 +++++++--- pyproject.toml | 2 +- 4 files changed, 91 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e95b467..22047e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,12 @@ Release notes and migration notes of `morphobricks-normalization`. Numbers are p issues of [CellMigrationLab/morphobricks-normalization](https://github.com/CellMigrationLab/morphobricks-normalization). The decisions are in [docs/plan.md](docs/plan.md) §12. -## Unreleased +## v0.5.0 (2026-11-11) -Stage 3 S3.2 (MorphoBricks-Spec roadmap; ADR 0003, plan N61–N64). Breaking: `plane_histograms.csv` -schema version 5 and channels named by canonical channel ID everywhere. Version not bumped yet. +Breaking release. Stage 3 S3.2 (MorphoBricks-Spec roadmap; ADR 0003, plan N61–N66, PR #90): +`plane_histograms.csv` schema version 5 and channels named by canonical channel ID everywhere, +plus the numerical-correctness and validation fixes of #91 and #92 merged since `v0.4.0`. See +[Migrating to v0.5.0](#migrating-to-v050). - **`plane_histograms.csv` schema version 5** (HV1, HV2): one row per plane; the presence column `channels` (the plane's IDs in source order, `;`-separated) after the 15 plane columns; the @@ -110,7 +112,9 @@ bit-identical (`tools/dump_outputs.py` and `--corpus` lines, and the 400 golden of squares is infinite (schema version 4 keeps raw sums of squares) is refused by name instead of giving `inf`: its spread is not known. No dump output changes. Not changed: the raw sums of squares of the file lose precision by cancellation for tails of large values - with a small spread; a centred representation needs a new schema version. + with a small spread; a centred representation needs a new schema version. (Done in this + release by schema version 5's centred tail moments, N65, above; the named refusals of + infinite tail sums are superseded by it.) - **Missing conditions per time point (#41).** With `scope="biological_repeat"` and `per_time_point=True`, `repeat_missing_conditions` checked each repeat group over all its time points, not the groups that are fitted, so a group such as `r1::t=0` with control @@ -148,6 +152,74 @@ bit-identical (`tools/dump_outputs.py` and `--corpus` lines, and the 400 golden images of 32- or 64-bit integers that were fitted before are refused; save them as 8- or 16-bit, or choose image type `fluorescence`. +### Migrating to v0.5.0 + +For the consuming bricks (preprocessing, CellPatcher, MorphoFeature) and any other caller. Every +brick should refuse a schema version 5 table explicitly until it is updated. Tables and parameter +rows written by v0.4.0 are refused, never converted: recompute them. + +**All callers** + +- `PlaneRecord(..., *, position_id, channels)`: `channels` (the plane's canonical channel IDs, in + the source order of its imaging unit's confirmed list) is required. Use + `record.channel_histogram(id)` and `record.has_channel(id)`; an index where an ID is expected is + refused. +- `HistogramTail(count, mean, std)` replaces `HistogramTail(count, sum, sum_of_squares)`; + `.sum` and `.sum_of_squares` are gone. New module `tail_moments`. +- `fit(..., channels=)`: for rows, the IDs to fit (default every ID some row has); for a pixel + array, the ID of each entry along `channel_axis` (required, except RGB, whose IDs are `R`, `G`, + `B`). `PooledHistogram(..., channel=id)` and `pooled_statistics(..., channel=id)`. +- A channel that only some planes of a fitted group have raises the new + `ChannelScopeIncompleteError` (`A_SCOPE_INCOMPLETE`, "present in k of n planes"; `.channels` + maps each such ID to the planes that lack it). Fit only the planes with it when the user + explicitly chooses `NormalizationSettings(partial_channels="planes_with_channel")`; that choice + is stored by `to_dict`, and each fit then reports the `channel_partially_present` warning listing + the planes. Offer it only as an explicit user choice. +- `Parameters.channels` are sorted IDs; `select(ids)`, `apply(..., channels=)`. Parameter columns + are `Normalization_Offset_` / `Normalization_Scale_` plus + `normalization_parameters_version` = `2` (`parameters.COLUMNS` has five entries); `from_columns` refuses a + row without it or of another version (the `..._C` columns of v0.4.0): fit again. +- `Issue.channel` is a channel ID; `summary_columns(channel_ids)`; `range_check_issues` takes + channel IDs; `ImageSummary.channels`. +- New: `ChannelScopeIncompleteError`, `NormalizationSettings.partial_channels`, + `ChannelMeasurementFailure`, `ChannelMeasurementError` (with `.failures` and `.removed`), + `check_measurements`, `records.check_universe`, `read_plane_histogram_table`, + `csv_io.table_header`, `csv_io.header_channels`; `Parameters` lives in the new module `parameters` (re-exported). + +**preprocessing** + +- Build `PlaneRecord(..., channels=)` from the unit's confirmed channel list. +- Call `write_plane_histograms(records, path, channel_ids=, failures=...)`, giving every parallel shard the same IDs so shards share one header. +- On `ChannelMeasurementError`, report `.failures` (plane and channel) and `.removed` (an + earlier table removed from the path); nothing is written (HV3). +- In `RunInputs`, read with `read_plane_histogram_table` (header IDs and records). A schema + version 4 table is refused with a message naming version 5 and saying to run preprocessing + again (HV4). +- Drop the N43 refusals (images may have different channels); pass channel IDs to + `range_check_issues`; show `ChannelScopeIncompleteError` as `A_SCOPE_INCOMPLETE`. +- The tail columns (`Histogram_{Underflow,Overflow}{Count,Mean,Std}_`) are written by the + library; nothing else changes for them. + +**CellPatcher** + +- Use `record.channel_histogram(id)`, `fit(..., channels=)`, and `Parameters.apply` / `select` by + ID. Patch columns are `Normalization_Offset_` / `Normalization_Scale_` plus + `normalization_parameters_version`; old tables are refused. Handle + `ChannelScopeIncompleteError`. + +**MorphoFeature** (`histogram_fit`) + +- Map catalogue names to `record.channels`; expect `ChannelScopeIncompleteError` unless the + settings opt in with `partial_channels`. + +**Numbers.** Planes whose channels are all present give the v0.4.0 histograms and fits bit for +bit (3600 fitted channels on `v13` trees, the 400 golden fits, `tools/dump_outputs.py --corpus` +apart from the stored tail fields). With tails (robust range), the stored tail fields change from +sums to centred moments, and pooled means, standard deviations, percentiles and fits move by at +most about `1.4e-14` relative; the #28 64-bit representative fix moves int64/uint64 numbers as +described above. + ## v0.4.0 (2026-11-10) Breaking release: `PlaneRecord.position_id` is required (NM-04) and the readers of old table layouts diff --git a/README.md b/README.md index 56c9d35..c0bc02c 100644 --- a/README.md +++ b/README.md @@ -29,10 +29,10 @@ out of scope for now. ## Status -Pre-alpha; the latest release is v0.4.0. The library offers: +Pre-alpha; the latest release is v0.5.0. The library offers: - supported pixel types, bin counts and per-plane histograms (`dtypes`, `histogram`), with a robust range and a range check; -- plane records, `plane_histograms.csv` reading and writing and pooling of group statistics (`records`, `csv_io`, `pooling`): schema version 4 in v0.4.0; on `main` after the Stage 3 pull request, schema version 5 with channels named by canonical channel ID (ADR 0003, see CHANGELOG "Unreleased"); +- plane records, `plane_histograms.csv` reading and writing and pooling of group statistics (`records`, `csv_io`, `pooling`): schema version 5 since v0.5.0, with channels named by canonical channel ID (ADR 0003; version 4 tables are refused, see the CHANGELOG migration notes); - settings, scopes, repeat groups and `fit`, with `Parameters` (`apply`, `to_columns`, `from_columns`), for fluorescence and RGB images; - per-image summaries and intensity checks, returned as `Issue` warnings. @@ -56,7 +56,7 @@ for key, rows in groups.items(): ## Installation ```bash -pip install "git+https://github.com/CellMigrationLab/morphobricks-normalization.git@v0.4.0" +pip install "git+https://github.com/CellMigrationLab/morphobricks-normalization.git@v0.5.0" ``` For development, from a clone: diff --git a/docs/plan.md b/docs/plan.md index a80c5b5..1618a6c 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -2,7 +2,7 @@ > See also: [Preprocessing follow-up](archive/followup_preprocessing.md) (done) · [CellPatcher follow-up](archive/followup_cellpatcher.md) (done) -**Status:** implemented through v0.4.0 (Milestones 0 to 5 done; later changes in [CHANGELOG.md](../CHANGELOG.md)); design brief simplified on 26 October 2026 (decisions N29–N40, §12), with +**Status:** implemented through v0.5.0 (Milestones 0 to 5 done; later changes in [CHANGELOG.md](../CHANGELOG.md)); design brief simplified on 26 October 2026 (decisions N29–N40, §12), with channels moved into columns on 27 October 2026 (N41–N45). The earlier design with six modalities, two purposes, colour and stain methods and reference targets (N4–N28) is withdrawn where §12 says so. A robust histogram range was added on 28 October @@ -264,9 +264,9 @@ their formula is implemented here. The preprocessing brick adds the setting and the warning (follow-up P6).* - **Stage 3 S3.2 — channels by canonical channel ID (ADR 0003):** `plane_histograms.csv` schema version 5, `PlaneRecord.channels`, fits, parameters, summaries and checks by channel - ID (N61–N64). *Pull request #90 (10 November 2026; rebased on #91 and #92 and finished on - 11 November 2026, the owner having decided to start Stage 3 now), not merged; see CHANGELOG - "Unreleased".* + ID (N61–N66). *Pull request #90 (10 November 2026; rebased on #91 and #92 and finished on + 11 November 2026, the owner having decided to start Stage 3 now), merged on 11 November 2026; + released in `v0.5.0` (CHANGELOG).* - **Release `v0.3.1`**, pinned by the other bricks. *Released (28 October 2026): Milestone 5. The tag `v0.3.0` points at the same code but its package reports version 0.2.0, because `pyproject.toml` was not updated before tagging; `v0.3.1` corrects the version and nothing @@ -290,6 +290,13 @@ their formula is implemented here. with the same messages (NM-01); the version-2 parity tests became fit goldens (NM-02); plus the numerical-range and refusal fixes merged since `v0.3.4` (CHANGELOG). Valid inputs give bit-identical outputs. Migration notes in the CHANGELOG. +- **Release `v0.5.0`** (11 November 2026): Stage 3 S3.2 (ADR 0003, N61–N66). `plane_histograms.csv` + schema version 5 (one row per plane, the `channels` presence column, per-channel columns by + canonical channel ID, tails as centred moments); version 4 tables are refused (hard cut); fits, + parameters (columns version 2), summaries and checks by channel ID; partial presence in a fitted + group refused by default (`A_SCOPE_INCOMPLETE`) with the recorded opt-in `partial_channels`; plus + the #91 and #92 fixes. Planes whose channels are all present give the v0.4.0 histograms and fits + bit for bit. Migration notes for the consuming bricks in the CHANGELOG. - **Then:** the preprocessing follow-up, then the CellPatcher follow-up (N6). Preprocessing steps P2 and P3 do not depend on this brick and may start earlier; its step P3b (writing schema version 3) follows Milestone 2b. From `v0.2.0` on, the other bricks pin the tag diff --git a/pyproject.toml b/pyproject.toml index a2af597..0b1e568 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "morphobricks-normalization" -version = "0.4.0" +version = "0.5.0" description = "MorphoBricks normalization brick: per-plane statistics, their file formats, and normalization methods for microscopy images." readme = "README.md" license = "MIT"