Repository navigation
P3b: plane_histograms.csv schema version 3, channels in columns - #23
Merged
Merged
Conversation
Follow the normalization brick's Milestone 2b (N41-N44): one row per XYC plane (image, Z, T) with each channel's extrema, counts and bins in its own CellProfiler-style columns, written with the brick's writer and PlaneRecord. - pyproject: pin morphobricks-normalization to commit 0e53e67 (Milestone 2b merge); no tags until its v0.2.0. - planes: iter_xyc_planes / XycPlane read a plane's channels one at a time (the owner's choice), keeping only the finished histograms; the memory estimate adds their bin counts, and one whole/blocks decision covers every channel. Block reading still gives the histograms of reading whole. - metadata, axes: images with different numbers of channels are the error different_channel_counts (N43), from the headers in step 1 and from the confirmed axis roles; different_plane_counts no longer compares channels. - run: rows sorted by path, z, t; schema 1/2 tables are recomputed; pixel warnings check every channel. - report, notebook step 4: pool every channel of a group as before, so the statistics table and plot do not change. - tests: a committed version-2 fixture written by the pre-P3b code; every channel of the new rows equals the version-2 row, whole and in blocks. - docs: contract, plan (question 27), user guide, README, AGENTS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up P3b of the normalization plan:
plane_histograms.csvis now written in schema version 3, one row per XYC plane (image, Z, T) with every channel in its own CellProfiler-style columns (Intensity_MinIntensity_C0,Histogram_Count_C1_00042, …), using the normalization brick's Milestone 2b writer andPlaneRecord(N41–N44). Plan §12 question 27.Summary
pyproject.toml:morphobricks-normalizationpinned to commit0e53e6762617e4297df0593c6977fdfb7b405522(merge of its Milestone 2b, PR Fix singleton-axis readiness and preserve outputs on failed updates #8) instead of@v0.1.0, with a comment on why (no tags until its v0.2.0).planes:iter_xyc_planes/XycPlaneyield one table row per XYC plane; its channels are read and binned one at a time (XycPlane.histograms), keeping only the finished histograms.PlaneLayout.channel_count; the memory estimate (whole and per block) adds the finished bin counts of every channel (8 bytes × bins × channels), so one whole/blocks decision covers every channel of every plane. The last decoded page is kept, so an RGB page is decoded once for its three channels.iter_planesis unchanged (conversion and previews still use it).metadata,axes,dataset: new errordifferent_channel_counts(N43), before any image is read: in the folder check from the headers (C axis, or RGB samples without one) and in the axis review from the confirmed roles, which supersede the header count (a confirmed missing C axis counts as one channel). Shared buildermetadata.channel_count_issues.different_plane_countsnow compares Z, T and other planes only. The new code is inCONSISTENCY_CODES, so the notebook's "Dataset consistency" panel shows it; the CLIcheckandrunprint it and exit 1.run: rows fromiter_xyc_planes, sorted by path, z, t; tables of schema version 1 or 2 are recomputed completely (manifestschema_versiondiffers; a v2 table under a v3 manifest is unreadable and also recomputed). Manifest stays version 1 and records schema3; a file'splane_countcounts XYC planes. Pixel warnings check every channel;no_finite_pixelscounts planes with an empty channel.report,notebook_ui(step 4): read the new layout and pool every channel of a group together exactly as before (report.channel_histograms), so the statistics table and plot are unchanged. A v2 table chosen in step 4 shows the brick's "run the preprocessing brick again" message.Decisions
_moved.py) keep forwarding the moved names to the brick's current objects (now with Milestone 2b signatures, e.g.bin_column(channel, index)). Names the brick added later (channel_column,WeightedHistogram,different_channel_counts, …) are not forwarded;test_movednow checks the moved names are a subset of the brick's.singlefolders, physical sizes per axis (TIFF and OME-Zarr), embeddable notebook steps #20's decisions (questions 23–26) are kept.Notes for review
different_channel_counts, so they use a missing T axis instead, and new tests cover the missing-C case.planes.different_channel_countstogether withmissing_axes/inconsistent_dimensions(e.g. ZCYX next to ZYX); after the axis roles are confirmed, the review's own count replaces it.Tests
tests/test_schema_v2_equivalence.pywith a committed fixture intests/datawritten by the pre-P3b code (55f1074+ normalization v0.1.0, viatests/data/make_v2_fixture.py): uint16 (TIFF plain, tiled-compressed, ImageJ, and an OME-Zarr image in another axis order; 500 bins, exact and binned channels) and float32 (NaN/±inf, an all-NaN channel, a constant channel; 64 bins). Every channel of every v3 row has the same text as the v2 row for that channel, read whole and in blocks; step 4 statistics equal those pooled from the v2 rows; a v2 table is recomputed on the next run..[dev,notebook,zarr]:ruff check .andruff format --check .clean; 935 passed, 2 skipped.🤖 Generated with Claude Code