diff --git a/CHANGELOG.md b/CHANGELOG.md index 2491af36..05df069f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the ## Unreleased +- **Re-pin the vendored Spec to c8f6852 (MorphoBricks-Spec PR 15, owner decisions RC1 and SI1).** + Text only: RC1 adds the optional boolean `rgb_support` key of a brick's provider manifest + (absent or `false` is unsupported; a non-boolean is a manifest error), and SI1 makes a channel or + mask that no image has the same `A_SCOPE_INCOMPLETE` refusal as one that some images lack. + Both are behaviour of the bricks, not of the description reader, so no code here changes + except `SPEC_COMMIT`. Fixtures and the reference validator are unchanged (the fixtures README + alone differs). A draft saved under the previous pin is refused at resume with + `spec_commit_mismatch` (see the draft file contract) and must be started again from the folders. + Every fixture is exact and the cross-check against the reference validator has 0 differences. - **The run side of the Z projection (part of #176).** A run on projected copies records a `projection` summary in its manifest (method, slices requested, and for every image the slices used, types, dtype rule, rounding and the source's content SHA-256, with the diff --git a/docs/conformance.md b/docs/conformance.md index 07817735..3bcb3e7a 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -8,7 +8,7 @@ real run; a skip is never counted as a pass. | | | |---|---| | Base commit | `68b2a4b` (`origin/main`, merge of PR 211); the run included the changes of PR 17 | -| Spec pin | the full SHA in `tests/spec_fixtures/PIN.toml` (vendored `experiment-v1.md` at `2975e0e`) | +| Spec pin | the full SHA in `tests/spec_fixtures/PIN.toml` (vendored `experiment-v1.md` at `c8f6852`) | | Command | `python tools/check.py` (Ruff, Ruff format, then `pytest -q` over the whole `tests/` folder), then `python tools/check.py --rules` | | Whole suite | 4324 passed, 4 skipped, 0 failed (301 s) | | Spec modules (`tests/test_spec_*.py`) | 1535 passed, 0 skipped, 0 failed (the Spec gate fails a skip or xfail there) | diff --git a/src/morphobricks_preprocessing/draft_format.py b/src/morphobricks_preprocessing/draft_format.py index 844969c5..fd71b441 100644 --- a/src/morphobricks_preprocessing/draft_format.py +++ b/src/morphobricks_preprocessing/draft_format.py @@ -33,7 +33,7 @@ DRAFT_FORMAT = "morphobricks-description-draft" DRAFT_VERSION = 1 -SPEC_COMMIT = "2975e0ee09d408d4a87da49d953f7988d4f9cd30" +SPEC_COMMIT = "c8f68526c08633526395ca0180fd6bd1a182fc84" """The MorphoBricks-Spec commit the rules of this brick are pinned to (``tests/spec_fixtures``).""" BASE_MISSING = "R_DRAFT_BASE_MISSING" SHA256 = re.compile(r"[0-9a-f]{64}") diff --git a/tests/spec_fixtures/PIN.toml b/tests/spec_fixtures/PIN.toml index f4edc117..8ed39686 100644 --- a/tests/spec_fixtures/PIN.toml +++ b/tests/spec_fixtures/PIN.toml @@ -1,13 +1,13 @@ # The MorphoBricks-Spec commit these files are copied from (conformance plan section 1). # Updating the pin is a reviewed change: see README.md in this folder. repository = "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/CellMigrationLab/MorphoBricks-Spec" -commit = "2975e0ee09d408d4a87da49d953f7988d4f9cd30" +commit = "c8f68526c08633526395ca0180fd6bd1a182fc84" # `git rev-parse :fixtures`, the git tree-object ID of the fixtures/ folder. The # vendored folder must hash to it, so one changed, added or missing byte fails a test. -fixtures_tree = "6fafcda2a88ee57fba8683a07a215664bce1b1c3" +fixtures_tree = "87d097408e5b794f31f902ec30f2a17dd941a1bd" # `git rev-parse :`, the git blob ID of each other vendored file. [files] -"docs/experiment-v1.md" = "3612d11857c655f459867c6426ffccb8d5b8c32f" +"docs/experiment-v1.md" = "e3d311b63e7ec7d2128a307822339806a10785dd" "tools/audit/refval.py" = "de5a2cdb53ecb5bda7493c5b8cab2908e6e32fe6" diff --git a/tests/spec_fixtures/docs/experiment-v1.md b/tests/spec_fixtures/docs/experiment-v1.md index 3612d118..e3d311b6 100644 --- a/tests/spec_fixtures/docs/experiment-v1.md +++ b/tests/spec_fixtures/docs/experiment-v1.md @@ -284,10 +284,22 @@ accepted, **declared** image type, per imaging unit: contradict the declaration, which then stands on user confirmation and is recorded in the manifest as not header-verified. - The spec gives RGB **no scientific meaning**: no grey conversion, luminance, stain separation - or joint-channel rule. Each brick owns the meaning and MUST declare, in its own documented - capabilities, whether it supports RGB; undeclared means unsupported. An analysis request that - includes an RGB unit's channels for a brick without RGB support is refused at analysis - (`A_RGB_NOT_SUPPORTED`, §8), never converted or split into grey channels silently. + or joint-channel rule. Each brick owns the meaning and MUST declare whether it supports RGB; + undeclared means unsupported. An analysis request that includes an RGB unit's channels for a + brick without RGB support is refused at analysis (`A_RGB_NOT_SUPPORTED`, §8), never converted or + split into grey channels silently. +- **Machine-readable declaration (`rgb_support`).** A brick that publishes a provider manifest + declares RGB support there, as the optional top-level key `rgb_support`, whose value is a + boolean. `true` declares support (the brick owns what RGB means for it); `false` and an absent key + both mean unsupported, so a manifest written before this key existed is read as unsupported and + needs no edit. A value that is not a boolean is a manifest error: it is never read as `true`. A + brick without a provider manifest states the same declaration in its documented capabilities. + The key declares support only; it gives RGB no meaning (the bullet above), and it does not + change a description: `image_type = "rgb"` is still user-declared and never inferred. +- The refusal applies to a **described** run (one opened from a published description). A run + without a description has no declared image type to refuse on; what a brick does with colour + images there is the brick's own behaviour (owner decision RC1: such runs keep working as today, + with a warning), not a Spec rule. - A file MAY hold RGB and non-RGB units; their lists differ, so that layout is heterogeneous (`R_HETEROGENEOUS_LAYOUT_NOT_ENABLED` in Stage 1-2). A unit's `image_type` is part of the `repeats` section digest (§9.3). @@ -718,9 +730,12 @@ Undetectable by any layer: identical-shape, unnamed images with physically swapp Representative previews and explicit confirmation are the guard; image-based checks are advisory. Required analysis-layer refusals (`A_*`): `A_SCOPE_INCOMPLETE` (a requested canonical channel or -mask is absent from part of the requested scope; refuse unless the user selects and records an -eligible subset; never drop FOVs, impute, re-weight technical repeats or change biological N); -`A_MASK_NOT_INSTANCES` (a `binary` mask requested as an object set without the explicit recorded per-request conversion of §4.5c); `A_MASK_UNCHECKED` (a run uses a mask file whose representation check is `not yet checked`, §4.5a); `A_RGB_NOT_SUPPORTED` (a request includes the channels of an RGB unit, §4.4b, for a brick that does not declare RGB support); `A_DRAFT_REFUSED` (§10: `RunInputs` given a draft or an unconfirmed file). Each biological repeat is +mask is absent from part of the requested scope, **or from all of it**: one rule, one code, no +separate code for "no image has it" (owner decision SI1); the refusal names the channel or mask and +the images that lack it, as "present in k of n acquisitions" (`k` may be 0); refuse unless the user +selects and records an eligible subset; never drop FOVs, impute, re-weight technical repeats or +change biological N); +`A_MASK_NOT_INSTANCES` (a `binary` mask requested as an object set without the explicit recorded per-request conversion of §4.5c); `A_MASK_UNCHECKED` (a run uses a mask file whose representation check is `not yet checked`, §4.5a); `A_RGB_NOT_SUPPORTED` (a request on a described run includes the channels of an RGB unit, §4.4b, for a brick whose manifest does not declare `rgb_support = true`, or that does not declare it at all); `A_DRAFT_REFUSED` (§10: `RunInputs` given a draft or an unconfirmed file). Each biological repeat is classified **fully represented**, **partially represented** or **unavailable** per request and recorded in the run manifest. Consumers resolve channels by canonical ID per unit, never by C index alone, and group by canonical repeat ID (`repeat_group`, §3.4) while keeping the source label diff --git a/tests/spec_fixtures/fixtures/README.md b/tests/spec_fixtures/fixtures/README.md index 27356a3f..f5bd34a0 100644 --- a/tests/spec_fixtures/fixtures/README.md +++ b/tests/spec_fixtures/fixtures/README.md @@ -128,7 +128,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | 6 | A binary region mask never becomes an instance object set | `v08` | Stage 2: `A_MASK_NOT_INSTANCES` | | 7 | N35 confirmation survives migration and reopens on discovery changes | `v13`, `i37`; P07 | Stage 1 migration and reconciliation | | 8 | Distinct source repeat names paired by the user give one canonical ID in MorphoCoverage | `v03` | Stage 2 (S2.5) | -| 9 | Missing channels in requested scope: refusal or recorded user subset, never silent drop | `v05`, `v17` | Stage 2: `A_SCOPE_INCOMPLETE` | +| 9 | Missing channels in requested scope: refusal or recorded user subset, never silent drop; a channel no image has is the same refusal (SI1) | `v05`, `v17` | Stage 2: `A_SCOPE_INCOMPLETE` | | 10 | Metadata-only changes keep valid pixel-derived caches | `v01` vs `v21` | Stage 1: section digests | **Further required cases:** @@ -239,7 +239,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | [`v14_no_repeat_folders`](valid/v14_no_repeat_folders.toml) | Extra: single convention | pass | yes | No biological-repeat or technical-repeat folders: the source label is `single` (N31, one group for the whole experiment). | | [`v15_toml_keys_are_text`](valid/v15_toml_keys_are_text.toml) | Extra: implicit-conversion traps (valid side) | pass | yes | Folder labels that YAML would retype (2026-01-01, 012, NO, 1.0, off) stay exact text in TOML keys and strings. The quoted key "1.0" is needed: a bare 1.0 would be a dotted key. | | [`v16_display_name_change`](valid/v16_display_name_change.toml) | Owner refinement: immutable IDs | pass | yes | v01 with every `display_name` changed or added (conditions, channels, repeats). Identity is identical: all four section digests equal v01's, nothing may be invalidated. | -| [`v17_partial_channel_presence`](valid/v17_partial_channel_presence.toml) | Adversarial 9; addendum §3 | pass | **no** | PHASE exists only in CTRL/R002/acquisition_1. For a PHASE request R002 is partially represented and R001 unavailable: the analysis layer must refuse (A_SCOPE_INCOMPLETE) or record a user-selected subset, never drop or re-weight silently. | +| [`v17_partial_channel_presence`](valid/v17_partial_channel_presence.toml) | Adversarial 9; addendum §3 | pass | **no** | PHASE exists only in CTRL/R002/acquisition_1. For a PHASE request R002 is partially represented and R001 unavailable: the analysis layer must refuse (A_SCOPE_INCOMPLETE) or record a user-selected subset, never drop or re-weight silently. A request for a channel ID that no unit of any fixture lists (the "absent from all" case, e.g. `Tubulin` against `v01`) is the same refusal with k = 0 (owner decision SI1); it needs no fixture of its own because the absent ID is a property of the request. | | [`v18_inline_and_dotted_forms`](valid/v18_inline_and_dotted_forms.toml) | Extra: round trip | pass | yes | Same model as v01 written with inline tables, dotted keys and quoted keys; parse(v18) must equal parse(v01). | | [`v19_warnings_unpaired`](valid/v19_warnings_unpaired.toml) | Extra: reference DAG semantics | pass + `W_REFERENCE_UNPAIRED`, `W_REPEAT_MISSING_CONDITIONS` | yes | Valid with warnings: CTRL only in rep1, DrugA only in rep2, so the repeats miss conditions (W_REPEAT_MISSING_CONDITIONS) and DrugA cannot be paired with its reference (W_REFERENCE_UNPAIRED). | | [`v20_unicode_and_spaces`](valid/v20_unicode_and_spaces.toml) | Extra: identities | pass | yes | Condition folder labels with spaces and non-ASCII text are valid labels (quoted keys); channel and mask IDs stay tokens. | @@ -250,7 +250,7 @@ This is a check of the fixtures against the text, not a conformance claim for an | [`v25_removed_condition_id_retained`](valid/v25_removed_condition_id_retained.toml) | Owner decision Q1 | pass | yes | v01 after DrugA was removed: DrugA stays listed in `used_ids.conditions`. Predecessor of t03 and t04 (and, with the recorded removal confirmation of experiment-v1 4.6, the removal of v01). | | [`v26_ids_kept_as_written`](valid/v26_ids_kept_as_written.toml) | Audit B1, B2, B11 | pass | yes | IDs `cl001`, `CL1000`, `R999`, `R1000`: no width, prefix or case rule; kept exactly as written. | | [`v27_descriptive_keys_collide`](valid/v27_descriptive_keys_collide.toml) | Owner decision Q4 | pass | yes | Identical and case-differing `target` / `treatment_key` values on different conditions: valid, no warning, no grouping. | -| [`v28_rgb_units`](valid/v28_rgb_units.toml) | Owner decision AQ6 (supersedes D8) | pass | yes | An RGB experiment (e.g. H&E): every unit declares `image_type = "rgb"` with the fixed channels R, G, B in that order and no aliases. Valid and homogeneous. The scientific meaning is left to each brick: a brick that does not declare RGB support refuses a request on these units (A_RGB_NOT_SUPPORTED, P24); preprocessing compares the declaration with the image header and never with pixels (P13, P23). | +| [`v28_rgb_units`](valid/v28_rgb_units.toml) | Owner decision AQ6 (supersedes D8) | pass | yes | An RGB experiment (e.g. H&E): every unit declares `image_type = "rgb"` with the fixed channels R, G, B in that order and no aliases. Valid and homogeneous. The scientific meaning is left to each brick: a brick that does not declare RGB support refuses a request on these units (A_RGB_NOT_SUPPORTED, P24); preprocessing compares the declaration with the image header and never with pixels (P13, P23). The `rgb_support` manifest key of experiment-v1 §4.4b (owner decision RC1) is a property of the receiving brick, not of the file, so no TOML fixture can express it; P24 states the four manifest cases to run against this one. | | [`v29_rgb_and_channels_mixed`](valid/v29_rgb_and_channels_mixed.toml) | Owner decision AQ6 | pass | **no** | One condition with an RGB brightfield unit and a two-channel fluorescence unit of the same biological repeat. Valid schema; heterogeneous (different lists), so the Stage 1-2 runtime refuses it (R_HETEROGENEOUS_LAYOUT_NOT_ENABLED). Both units stay ONE biological repeat. | | [`v30_same_source_label_different_repeats`](valid/v30_same_source_label_different_repeats.toml) | Owner decision AQ5; S0.1 audit section 5 | pass | yes | Source folder label rep1 is in canonical repeat Day1 for CTRL and in Day2 for DrugA (and rep2 the other way round). Valid: no repeat ID equals another repeat's source label. The FOV key keeps the source label in `biological_repeat` and stays unique through the condition; the added `repeat_group` is Day1 or Day2, and grouping and pairing use it (experiment-v1 3.4, P25). | | [`v31_header_alias_whitespace_and_decomposed`](valid/v31_header_alias_whitespace_and_decomposed.toml) | Owner decision AQ7 | pass | yes | Header evidence is compared exactly after NFC of both sides, with no trimming. DNA lists the aliases "DAPI", " DAPI" (leading space) and "DAPI " (trailing space): outer whitespace needs its own confirmed alias, each is a distinct entry. ACTIN lists a decomposed alias (written with a TOML escape and stored as typed), which matches the header name "Caf\u00e9 488" after NFC of both sides (P26). header_names are not identities and are not NFC-checked. |