Re-pin the vendored Spec to 7dbae0c (byte rules, goldens, t13, records encoding)

CHANGELOG.md

@@ -8,6 +8,17 @@ are in [docs/plan.md](docs/plan.md) §12. Upgrading from v0.3: see the
 
 ## Unreleased
 
+- **Re-pin the vendored Spec to 7dbae0c (MorphoBricks-Spec PR 9).** New in the pin: the byte
+  rules and golden files of the derived `experiment_design.csv` (section 11; `fixtures/expected/`
+  for `v13`, `v33`, `v34`, not yet derived by any code: Stage 1 PR 10), the byte form of one
+  section digest (9.3), `t13` (a changed `image_count` alone does not reopen a confirmation) and
+  the encoding of `morphobricks.records.jsonl` (4.6). The reader, writer and publication agree
+  with every fixture and the reference validator (0 differences, no exception for `image_count`
+  any more; the oracle's section digests equal ours on every fixture). Fix: `read_records` split
+  the records file with `str.splitlines`, which also cuts at U+2028 and U+0085, so a record
+  holding such a character (written unescaped as the Spec requires) was refused as "not JSON";
+  lines now end at LF only. Test helper: the catalogue no longer reads the golden-file rows as
+  fixtures.
 - **Safe display of planes (Stage 1, PR 13; part of #169).** New modules `preview` and
   `preview_render`, not used by the notebook yet. `preview.read_plane(path, mapping, max_side=...)`
   returns at most `max_side` pixels per side (`THUMBNAIL_SIDE` 256, `DETAIL_SIDE` 1024), every

src/morphobricks_preprocessing/run/description_file.py

@@ -126,7 +126,8 @@ def read_records(output_folder: str | os.PathLike) -> tuple[dict, ...]:
     if not data.endswith(b"\n"):
         raise RecordsFileError(f"{RECORDS_FILENAME} ends in the middle of a line")
     try:
-        rows = tuple(json.loads(line) for line in data.decode("utf-8").splitlines())
+        # Lines end at LF only (Spec 4.6): str.splitlines would also cut at U+2028, U+0085, ...
+        rows = tuple(json.loads(line) for line in data.decode("utf-8").split("\n")[:-1])
     except ValueError as error:
         raise RecordsFileError(f"{RECORDS_FILENAME} holds a line that is not JSON") from error
     if not all(isinstance(row, dict) for row in rows):

tests/spec_fixtures/PIN.toml

@@ -1,13 +1,13 @@
 # The MorphoBricks-Spec commit these files are copied from (conformance plan section 1).
 # Updating the pin is a reviewed change: see README.md in this folder.
 repository = "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/CellMigrationLab/MorphoBricks-Spec"
-commit = "745259b72d4252f3ddaa563302d3343a1fa2ed49"
+commit = "7dbae0c600ac051b492f4d5fe41dd0b6008649db"
 
 # `git rev-parse <commit>:fixtures`, the git tree-object ID of the fixtures/ folder. The
 # vendored folder must hash to it, so one changed, added or missing byte fails a test.
-fixtures_tree = "d26daf758c914bdb0f6bfde1bba278d99cc33bb8"
+fixtures_tree = "bb336c9494c371e0ba188f9be31832fdf19b0a14"
 
 # `git rev-parse <commit>:<path>`, the git blob ID of each other vendored file.
 [files]
-"docs/experiment-v1.md" = "6de05a197caf7b57c6ce0409f9593a9c56d23b7c"
-"tools/audit/refval.py" = "2ce3a512fecafeb954aff32308f925ce23b8cd59"
+"docs/experiment-v1.md" = "73662c0b76a18250958beb885b016e4ced56d019"
+"tools/audit/refval.py" = "0c8274429919108bf013e32e8dd0563e081c47b3"

tests/spec_fixtures/docs/experiment-v1.md

@@ -386,6 +386,14 @@ stays in the brick that owns it.
   `morphobricks.toml`, and copies the records unchanged into the manifest of the next run. The
   records file is never rewritten or truncated, and nothing about a re-confirmation is stored in
   `morphobricks.toml`. Products keyed on that ID are invalidated.
+- **Encoding of the records file.** UTF-8, one JSON object per line, each line ending in LF, keys
+  in code point order, non-ASCII characters unescaped. Keys: `type` (`"reconfirmation"`),
+  `id_kind` (`conditions`, `channels`, `repeats` or `technical_repeats`), `old_id`, `new_id`
+  (`null` when the same ID lost or was reassigned something), `confirmed_on` (`YYYY-MM-DD`, the
+  date of the user's action), `old_sha256` (of the published file replaced, or `absent`) and
+  `new_sha256` (of the file written). The records are appended after the file is replaced. A records
+  file whose last line lacks its LF, or that holds a line that is not a JSON object, is refused
+  before anything is published.
 - **A case-only correction needs a new ID (AQ2).** `ACTIN` cannot be corrected to `Actin`: the
   retired spelling stays in `used_ids` and the new one collides with it
   (`E_ID_CASE_COLLISION`, §4.9). The correction is a new, non-colliding ID (for example
@@ -702,7 +710,11 @@ standard/inline/dotted tables, never changes meaning.
 
 The whole-file SHA-256 is provenance. Preprocessing SHOULD also record per-section digests (SHA-256
 of the canonical serialization of `conditions`, `channels`, `repeats`, `masks`) so that consumers
-key derived products only on the sections they use. `source_folder` is part of the `conditions` and
+key derived products only on the sections they use. **The bytes of one section digest** are the
+section's table blocks exactly as the canonical writer emits them in the file (§9.1), each block
+followed by one blank line, with `display_name`, `target` and `treatment_key` left out and nothing
+before or after; a section with no tables (no `masks`) is the SHA-256 of the empty byte string.
+`source_folder` is part of the `conditions` and
 `repeats` digests (it fixes which folder an ID names; a reassignment changes them). Digests are computed with every `display_name`,
 `target` and `treatment_key` omitted (descriptive fields are never cache keys, §4.2a), and
 `used_ids` is not a section: a change to any of them MUST NOT change any digest, a change to `reference` MUST NOT
@@ -736,7 +748,9 @@ silently; edits go to a separate draft and are published atomically after valida
 - **N35 reopening.** The confirmation of a repeat is reopened (draft `confirmed = false`) only when
   channels, layout or grouping change: a new or removed unit or member, a unit's ordered channel
   list, or the pairing of folders into repeats. New FOVs (or removed ones) inside an existing unit
-  whose confirmed layout still holds do **not** reopen it; they are reported and recorded.
+  whose confirmed layout still holds do **not** reopen it; they are reported and recorded. A
+  changed `image_count` alone is therefore not a reopening: a new `confirmed_on` beside only a
+  changed `image_count` is `R_CONFIRMED_ON_CHANGED` (§4.4a).
 - **Adding a member is allowed (AQ1).** A new member folder or unit joins an existing repeat or
   condition without a changed-ID refusal (§4.6); it is a grouping change, so it reopens the N35
   confirmation of that repeat, and it never changes an existing ID. Only removing or reassigning
@@ -774,7 +788,13 @@ deterministically from a published file, with the contract's byte rules:
 | `schema_version` | `1` |
 
 One row per member, sorted by `condition` (the source folder label), then `biological_repeat`, in
-code point order. The CSV carries the **source folder labels exactly as today** and no technical
+code point order. **Bytes** (preprocessing contract §12, rule `Rows`, and the `confirmed_on`
+row): UTF-8, LF line endings, one header row naming the nine columns in the order of the table
+above, booleans written `true` and `false`, `confirmed_on` written as the repeat's `confirmed_on`
+(`YYYY-MM-DD`). The contract does not state the delimiter, the quoting, the final line terminator
+or a byte-order mark, so these four are not yet normative; the golden files use a comma, no
+quoting (no golden value needs it), LF after every row including the last, and no byte-order
+mark, as today's writer does. The CSV carries the **source folder labels exactly as today** and no technical
 repeat or condition ID, so the choice of a condition or technical repeat ID (for example `ctrl` for
 the folder `control`) never changes it. For a legacy dataset whose `repeat_group` values migrate
 to the canonical repeat IDs and whose controls migrate deterministically, the derived file MUST be
@@ -786,8 +806,8 @@ latest, on every row. **Byte-identity is guaranteed only for repeat groups with
 `confirmed_on`**; for a group with several dates the earlier per-row dates are the only difference
 and are recorded in the migration note, which is their only record.
 
-Expected `experiment_design.csv` outputs (golden files) are **deferred to Stage 1**; Stage 0 fixes
-only the derivation rules above.
+Expected `experiment_design.csv` outputs (golden files) for `v13`, `v33` and `v34` are in
+`fixtures/expected/`; a derived file MUST equal the golden file byte for byte.
 
 ## 12. Drafts
 

tests/spec_fixtures/fixtures/README.md

@@ -28,7 +28,7 @@ is executed by this repository.
   fixtures that do not exercise the rule it equals the ID (the key), `single` for `single`;
   `v33`, `v34` and `t10`-`t12` separate the two. The eight undecodable
   or unparsable phase-1 fixtures (`i01`-`i08`) are unchanged and fail before it matters.
-- `transitions/` holds twelve files that are valid alone and are checked against a named published
+- `transitions/` holds thirteen files that are valid alone and are checked against a named published
   predecessor: refused with an exact set of publication codes (`R_CONFIRMED_ID_CHANGED`,
   `R_USED_ID_REUSED`, `R_USED_ID_DROPPED`, `R_CONFIRMED_ON_CHANGED`; experiment-v1 §4.4a, §4.6,
   §4.9; a changed or reassigned `source_folder` counts, §4.6a) or, for `t06`, **accepted** (adding members is not a changed entity, §4.6). They need the
@@ -40,7 +40,8 @@ is executed by this repository.
   plan on a named fixture, listed in the coverage tables below.
 - A copy of this tree vendored by a reader MUST hash to the pinned SHA's `fixtures` tree
   (conformance plan §1).
-- Expected `experiment_design.csv` outputs (golden files) are deferred to Stage 1. The fixed-SHA
+- Expected `experiment_design.csv` outputs (golden files, experiment-v1 §11) are in `expected/`,
+  one per file listed under "Expected outputs" below; the other valid files have none yet. The fixed-SHA
   consumer audit (roadmap S0.1) is done (PR #5); its questions are answered by the third round of
   owner decisions (ADR 0001) and the fixtures `v28`-`v32`, `t06`-`t09`, `i100`-`i119` follow from
   them; the fourth round (FR1, FR2) adds `v33`, `v34`, `t10`-`t12` and `i120`-`i127`.
@@ -255,7 +256,18 @@ This is a check of the fixtures against the text, not a conformance claim for an
 | [`v35_flat_experiment`](valid/v35_flat_experiment.toml) | Matrix audit 2026-11-07, case F01 | pass | yes | The experiment folder is itself the leaf: condition, repeat and technical repeat are all `single`. The condition `single` has `source_folder = "single"` ("no condition folder level", 4.2, 3.2) and is the only condition (owner decision SQ6, matrix audit MS-1). |
 | [`v36_repeat_folder_forbidden_name`](valid/v36_repeat_folder_forbidden_name.toml) | Matrix audit 2026-11-07, case F16 | pass | yes | Biological-repeat folders `z` paired into repeat `R1`; `source_repeat` is never restricted. The folder label `z` is not an allowed repeat ID (`i130`), so no ID is proposed and the user names `R1` (4.6a; owner decision SQ6, matrix audit MS-2). |
 
-## Transition fixtures (12)
+## Expected outputs (3)
+
+Golden `experiment_design.csv` files (experiment-v1 §11), byte for byte. Each is the derivation of the
+named valid file by the stated rules; `tools/audit/refval.py` compares them.
+
+| Golden file | Derived from | Notes |
+|---|---|---|
+| [`v13_homogeneous_legacy.experiment_design.csv`](expected/v13_homogeneous_legacy.experiment_design.csv) | `valid/v13` | One control (`CTRL`, referenced by DrugA and DrugB); every `technical_repeat_count` 0 (only `single`). |
+| [`v33_condition_folder_on_forbidden_list.experiment_design.csv`](expected/v33_condition_folder_on_forbidden_list.experiment_design.csv) | `valid/v33` | `condition` holds the folder label `control`, never the ID `ctrl`; `DrugA` sorts before `control` (code point order). |
+| [`v34_technical_repeat_folder_on_forbidden_list.experiment_design.csv`](expected/v34_technical_repeat_folder_on_forbidden_list.experiment_design.csv) | `valid/v34` | Two technical repeats (folders `files`, `overrides`) give `technical_repeat_count` 2; `rep2` has only `single`, so 0. |
+
+## Transition fixtures (13)
 
 Valid alone; checked against the named published predecessor (refused with the listed set, or accepted).
 
@@ -273,6 +285,7 @@ Valid alone; checked against the named published predecessor (refused with the l
 | [`t10_condition_id_changed_same_folder`](transitions/t10_condition_id_changed_same_folder.toml) | `valid/v33` | R_CONFIRMED_ID_CHANGED (against predecessor valid/v33, without a recorded re-confirmation) | The folder `control` was confirmed as condition `ctrl` in v33 and now carries the ID `control_cells`. A confirmed ID never changes in place, also when the folder is the same (experiment-v1 4.6, 4.6a). `ctrl` stays listed in used_ids, so only the changed ID is refused. Valid alone. |
 | [`t11_technical_repeat_id_changed_same_folder`](transitions/t11_technical_repeat_id_changed_same_folder.toml) | `valid/v34` | R_CONFIRMED_ID_CHANGED (against predecessor valid/v34, without a recorded re-confirmation) | The folder `files` was confirmed as technical repeat `acq1` in v34 and now carries the ID `acqA` in both members. A confirmed technical repeat ID never changes in place (experiment-v1 4.6, 4.6a). `acq1` stays listed in used_ids. Valid alone. |
 | [`t12_condition_folder_reassigned`](transitions/t12_condition_folder_reassigned.toml) | `valid/v33` | R_CONFIRMED_ID_CHANGED (against predecessor valid/v33, without a recorded confirmation) | The ID `ctrl` held the folder `control` in v33 and now holds the folder `ctl` (a folder renamed on disk and attached to the same ID without a recorded confirmation). A folder change for an existing ID is a reassignment, so a changed entity (experiment-v1 4.6, 7). Valid alone. |
+| [`t13_confirmed_on_with_image_count_only`](transitions/t13_confirmed_on_with_image_count_only.toml) | `valid/v01` | `R_CONFIRMED_ON_CHANGED` (N35 not reopened) | Against v01 only CTRL/rep1 `image_count` (12 to 17, new FOVs) and rep1 `confirmed_on` (2026-11-07) change: new FOVs do not reopen the confirmation (experiment-v1 10), so the new date is refused. |
 
 ## Invalid fixtures (134)
 

tests/spec_fixtures/fixtures/expected/v13_homogeneous_legacy.experiment_design.csv

@@ -0,0 +1,10 @@
+condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count,confirmed,confirmed_on,schema_version
+CTRL,rep1,rep1,true,10,0,true,2026-10-30,1
+CTRL,rep2,rep2,true,10,0,true,2026-10-30,1
+CTRL,rep3,rep3,true,10,0,true,2026-10-30,1
+DrugA,rep1,rep1,false,10,0,true,2026-10-30,1
+DrugA,rep2,rep2,false,10,0,true,2026-10-30,1
+DrugA,rep3,rep3,false,10,0,true,2026-10-30,1
+DrugB,rep1,rep1,false,10,0,true,2026-10-30,1
+DrugB,rep2,rep2,false,10,0,true,2026-10-30,1
+DrugB,rep3,rep3,false,10,0,true,2026-10-30,1

tests/spec_fixtures/fixtures/expected/v33_condition_folder_on_forbidden_list.experiment_design.csv

@@ -0,0 +1,5 @@
+condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count,confirmed,confirmed_on,schema_version
+DrugA,rep1,rep1,false,11,0,true,2026-11-06,1
+DrugA,rep2,rep2,false,10,0,true,2026-11-06,1
+control,rep1,rep1,true,12,0,true,2026-11-06,1
+control,rep2,rep2,true,10,0,true,2026-11-06,1

tests/spec_fixtures/fixtures/expected/v34_technical_repeat_folder_on_forbidden_list.experiment_design.csv

@@ -0,0 +1,5 @@
+condition,biological_repeat,repeat_group,control,image_count,technical_repeat_count,confirmed,confirmed_on,schema_version
+CTRL,rep1,rep1,true,12,2,true,2026-11-06,1
+CTRL,rep2,rep2,true,10,0,true,2026-11-06,1
+DrugA,rep1,rep1,false,11,2,true,2026-11-06,1
+DrugA,rep2,rep2,false,10,0,true,2026-11-06,1

tests/spec_fixtures/fixtures/transitions/t13_confirmed_on_with_image_count_only.toml

@@ -0,0 +1,71 @@
+# Fixture t13_confirmed_on_with_image_count_only
+# Expected: R_CONFIRMED_ON_CHANGED (against predecessor valid/v01, N35 not reopened)
+# Source: Preprocessing PR 188 (Stage 1 publication); experiment-v1 10
+# Valid on its own. Against v01, rep1 has the same members, units and channel lists and only two things change: CTRL/rep1 `image_count` 12 to 17 (new FOVs in an existing unit) and `confirmed_on` 2026-11-06 to 2026-11-07. New FOVs do not reopen the N35 confirmation (experiment-v1 10), so the new date is refused: a changed `image_count` alone never makes a new `confirmed_on` acceptable.
+schema = "morphobricks-experiment"
+schema_version = 1
+
+[conditions.CTRL]
+source_folder = "CTRL"
+display_name = "Control"
+
+[conditions.DrugA]
+source_folder = "DrugA"
+display_name = "Drug A"
+reference = "CTRL"
+target = "SRC"
+treatment_key = "saracatinib"
+
+[channels.DNA]
+display_name = "DAPI"
+header_names = ["DAPI"]
+
+[channels.ACTIN]
+display_name = "F-actin"
+header_names = ["Phalloidin-488"]
+
+[repeats.rep1]
+confirmed = true
+confirmed_on = "2026-11-07"
+
+[repeats.rep1.members.CTRL]
+source_repeat = "rep1"
+image_count = 17
+
+[repeats.rep1.members.CTRL.technical_repeats.single]
+source_folder = "single"
+channels = ["DNA", "ACTIN"]
+
+[repeats.rep1.members.DrugA]
+source_repeat = "rep1"
+image_count = 11
+
+[repeats.rep1.members.DrugA.technical_repeats.single]
+source_folder = "single"
+channels = ["DNA", "ACTIN"]
+
+[repeats.rep2]
+confirmed = true
+confirmed_on = "2026-11-06"
+
+[repeats.rep2.members.CTRL]
+source_repeat = "rep2"
+image_count = 10
+
+[repeats.rep2.members.CTRL.technical_repeats.single]
+source_folder = "single"
+channels = ["DNA", "ACTIN"]
+
+[repeats.rep2.members.DrugA]
+source_repeat = "rep2"
+image_count = 10
+
+[repeats.rep2.members.DrugA.technical_repeats.single]
+source_folder = "single"
+channels = ["DNA", "ACTIN"]
+
+[used_ids]
+conditions = ["CTRL", "DrugA"]
+channels = ["ACTIN", "DNA"]
+repeats = ["rep1", "rep2"]
+technical_repeats = ["single"]