Fill position_id per image (D02) and require Python 3.11 (D01)

.github/workflows/ci.yml

@@ -23,11 +23,14 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ["3.10", "3.13"]
+        python-version: ["3.11", "3.13"]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - run: python -m pip install -e ".[dev,notebook,zarr]"
+      # zarr 3 needs Python 3.11 too, so pin 2.18 on 3.11 to keep both majors tested.
+      - if: matrix.python-version == '3.11'
+        run: python -m pip install "zarr>=2.18,<3"
       - run: python -m pytest

AGENTS.md

@@ -57,7 +57,8 @@ Pipeline:
 
 ```
 src/morphobricks_preprocessing/
-  discovery.py      folder walk, leaf folders, labels (condition / biological_repeat / technical_repeat)
+  discovery.py      folder walk, leaf folders, labels (condition / biological_repeat / technical_repeat),
+                    position_id per image and the duplicate check
   metadata.py       TIFF header inspection (tifffile, no pixel reads), dtype/dimension/series checks,
                     OME channel names + significant bits
   validation.py     ValidationIssue (severity, code, message, paths, action), error()/warning() builders
@@ -127,7 +128,7 @@ functions the tests use.
    *Summary* (by module), *Decisions*, *Notes for review* (limitations, what was not
    verified), *Tests* (exact counts, e.g. "455 passed, 2 skipped").
 5. **Done means:** `ruff check .`, `ruff format --check .` and `pytest` all pass (CI runs
-   lint on Ubuntu and tests on Ubuntu, Windows and macOS with Python 3.10 and 3.13), docs
+   lint on Ubuntu and tests on Ubuntu, Windows and macOS with Python 3.11 and 3.13), docs
    updated, and local changes ready for review. Open a PR only when the user requests it.
 6. **Be honest in reports.** Say what was verified and what was not (for example: "widgets
    tested headless; not looked at in a real JupyterLab").
@@ -154,6 +155,11 @@ functions the tests use.
   path.
 - **Depth:** a folder below the technical-repeat level is an **error** (so a layer of images
   is never dropped silently).
+- **Positions (cross-brick decision D02):** one image is one position. Its `position_id`
+  (`discovery.position_id`) is the name without `.tif`/`.tiff`/`.zarr` and a preceding
+  `.ome`. The position key is `condition` + `biological_repeat` + `technical_repeat` +
+  `position_id`. Two images in one leaf folder with the same identifier, ignoring case,
+  are the error `duplicate_position_id`: never guess.
 - **System files:** dot-files and dot-folders, `Thumbs.db`, `desktop.ini` and `__MACOSX` are
   ignored **without** a warning.
 - **Warnings:** non-TIFF files, empty folders, TIFFs outside leaf folders.
@@ -229,14 +235,17 @@ functions the tests use.
 - **`plane_id`** = relative path plus `::z=…,c=…,t=…`, always all three
   (`img.tif::z=0,c=0,t=0` for a 2D image). It is the row key.
 - **Rows** are sorted by path, then z, c, t.
+- **`position_id`** is filled in every row (D02, plan §12 question 20). Earlier tables with
+  it blank are filled on the next run without reading pixels.
 - **Reading:** labels are always read back as text (`NA`, `001`, `TRUE` stay as typed).
   Floats are written with `repr`.
 
 ### Run manifest and reruns (contract §9, plan §7)
 
 - **Manifest contents:**
   - settings, the confirmed labels per leaf, and the axis mapping;
-  - per-file size, `mtime_ns`, axes, `singleton_roles` and plane count;
+  - per-file size, `mtime_ns`, axes, `singleton_roles`, plane count and `position_id`
+    (optional key, null when absent; manifest version stays 1);
   - channel names and significant bits (OME only, information only);
   - the CSV's SHA-256 and package versions (including `morphobricks-normalization`).
 - **A file is unchanged** when its size, mtime, labels and axis layout match and the table
@@ -315,6 +324,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
 | CLI | `check <folder>` (summary, label tree, issues, suggested `--axes`) and `run <folder> --axes ROLES` with `--output`, `--exclude`, `--bins`, `--memory-limit 2GiB`, `--missing-as-singleton`, `--force-rebuild`, `--log`, and the conversion options above. Axis roles are always stated explicitly, never taken silently from metadata. Exit 1 on errors |
 | Large planes (Q15) | Planes over the memory limit are read in blocks automatically, announced by the warning `planes_read_in_blocks` and `read_in_blocks` in the manifest. Block size derived from the limit (whole strips/tiles/chunks, uncompressed pieces and chunks also by rows). Compressed single-strip/tile planes keep the `plane_over_memory_limit` error. Two-pass histograms identical to whole planes. OME-Zarr copies written block by block, smaller levels built from the stored level above. Axis preview reduced (every n-th pixel) with a caption saying so |
 | Normalization brick (P1) | The modules `dtypes`, `histogram`, `records`, `csv_io`, `pooling` and `normalization` moved unchanged to `morphobricks_normalization` (normalization plan N1, N2, N6) and are imported from there. Their public names stay importable from `morphobricks_preprocessing` and from the old submodules for one version: module `__getattr__` stand-ins (`_moved.py`), one `DeprecationWarning` per old path naming `morphobricks_normalization.<module>.<name>`, same objects. The package's own code never uses the old paths. Outputs unchanged (CSV schema 2); the manifest's `versions` adds `morphobricks-normalization`; Step 0 checks for it; contract §3–§7 text moved to the normalization contract (§1.2, §8.2) |
+| Position identity (Q20, D02) | `position_id` = image name without extension, in every CSV row and manifest file entry; FOV key = labels + `position_id`; duplicates in one leaf folder are the error `duplicate_position_id`. CSV columns and schema 2 unchanged; manifest version 1 with an optional key |
 | Notebook style | Follow the lab's NucleiSky and LabChronicle notebooks (see §7) |
 | Widget widths | **99%**, never 100% |
 | Paths and filenames in widget HTML and notebook text | **`<b>…</b>` / `**…**`, never `<code>` or inline code** |
@@ -325,7 +335,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
 
 ## 6. Programming style
 
-- **Python 3.10+.** Put `from __future__ import annotations` in every module, and type
+- **Python 3.11+** (MorphoBricks decision D01). Put `from __future__ import annotations` in every module, and type
   hints on all public functions.
 - **Ruff is the authority.** Line length 100. Rules E, W, F, I, B, UP, NPY and D, with
   numpy docstring convention; D105 and D107 ignored; tests are exempt from D.
@@ -387,7 +397,7 @@ Technical details go to `logging` (shown in the notebook's "Technical details" p
 - **Notebook widgets are tested headless:** build the pages, drive them with
   `button.click()` and `.value = …`, and assert on `.value` of HTML widgets. Tests skip
   if ipywidgets/matplotlib are missing; CI installs `.[dev,notebook,zarr]`. OME-Zarr
-  tests skip without zarr; CI's Python 3.10 jobs get zarr 2.18 and 3.13 jobs zarr 3.
+  tests skip without zarr; CI's Python 3.11 jobs pin zarr 2.18 and 3.13 jobs get zarr 3.
 - **The notebook file is tested too:**
   - valid nbformat 4, with no outputs;
   - code cells hidden and titled `# @markdown ### Run to …`;
@@ -584,7 +594,8 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb
   written (PR #14, question 17), OME-NGFF label readers (PR #15, question 18), and
   follow-up P1 of the normalization plan: the dependency on `morphobricks-normalization`
   v0.1.0 with deprecated re-exports, its version in the manifest, Step 0 checking for it,
-  and contract §3–§7 moved to the normalization contract (PRs #16 and #17, question 19).
+  and contract §3–§7 moved to the normalization contract (PRs #16 and #17, question 19),
+  and `position_id` filled per cross-brick decision D02 (question 20).
 - **Next (not started):** follow-ups P2–P5 of the normalization plan (modality,
   references, colour statistics, previews).
 - **Not yet verified:**

docs/contract.md

@@ -28,6 +28,7 @@ whole experiment; any change to them goes through the normalization brick.
 | Depth | At most three folder levels below the root. A folder below the technical-repeat level is an error. |
 | System files | Dot-files and dot-folders (`.DS_Store`, `._*`, `.ipynb_checkpoints`), `Thumbs.db`, `desktop.ini` and `__MACOSX` are ignored without a warning. |
 | Scope | A group is identified by its full label path, e.g. `(control, bio_rep1)`. |
+| Positions | Every image is exactly one position (field of view): files with several positions, scenes, series or tiles are refused (§2). Its `position_id` is the image name without its extension (`.tif`, `.tiff` or `.zarr`, any case, with a preceding `.ome`): `img_01.ome.tif` gives `img_01`. A position is identified by `condition`, `biological_repeat`, `technical_repeat` and `position_id` (cross-brick decision D02, 26 October 2026), so equal names in two technical repeats are two positions. Two images in one leaf folder with the same `position_id`, ignoring case (`img_01.tif` and `img_01.ome.zarr`), are an error (`duplicate_position_id`): the brick does not guess which is which. |
 | Paths | `source_relative_path` is relative to the experiment root, with `/` separators. |
 
 ## 2. Axes
@@ -71,6 +72,12 @@ Defined by the [normalization brick's contract][normalization contract] §8.2 (c
 §8.1 (shared format). This brick writes it next to `run_manifest.json` (§9), with plane
 coordinates and labels as in §1–§2.
 
+This brick fills `position_id` in every row, with the image's position identifier (§1).
+The file-name fallback follows cross-brick decision D02 (one file is exactly one position
+here); the normalization contract's note that `position_id` is never inferred from file
+names predates D02. The columns and `schema_version` are unchanged. A table written before
+this rule, with `position_id` blank, is filled on the next run without reading pixels.
+
 ## 6. Group statistics
 
 Defined by the [normalization brick's contract][normalization contract] §1.2.3: groups
@@ -85,7 +92,7 @@ method is an offset and a scale, `(pixels − offset) / scale`.
 
 | Severity | Situations | Effect |
 |---|---|---|
-| Error | Mixed dtypes; unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single`; sibling folders differing only in case; folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked |
+| Error | Mixed dtypes; unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single`; sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes | Processing is blocked |
 | Warning | Files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; channel names differing between files; non-finite pixels; different image sizes with a valid mapping; planes without finite pixels while others remain; planes over the memory limit that are read in blocks | Processing continues; the item is reported |
 
 Every message states what happened in plain language, lists the affected paths, and
@@ -123,7 +130,7 @@ file's schema or statistical interpretation.
 | `settings` | `bin_count`, `memory_limit` (bytes), `missing_as_singleton`, `exclude` (sorted relative paths), `convert_tiffs` (whether TIFF images are converted to OME-Zarr and processed from the copies; false when absent), `conversion` (`chunk_size`, `compression`, `compression_level`, `pyramid_levels` — null for automatic; defaults when absent). |
 | `axis_mapping` | `axes` (file axis labels) and `roles` (`X`, `Y`, `Z`, `C`, `T` or null per axis). |
 | `labels` | One entry per leaf folder: `folder` (relative path, `.` for the root) and its three labels. |
-| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count`, `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. |
+| `files` | One entry per analysed file, sorted by path: `relative_path`, `size` (bytes), `mtime_ns`, the three labels, `axes`, `singleton_roles`, `plane_count`, `channel_names` (list, null for unnamed channels; empty if none in metadata), `significant_bits` (null if not in metadata), `ome_zarr_path` (for a TIFF converted to OME-Zarr, the copy relative to the output folder; null otherwise or when the key is absent), `read_in_blocks` (true when the image's planes are over the memory limit and are read in blocks; information only, false when absent), `position_id` (the image's position identifier, §1; null when absent, in manifests written before it). For an OME-Zarr image, `size` is the total size of its files and `mtime_ns` the time of its newest file. |
 | `csv_sha256` | SHA-256 of the table written with this manifest. |
 | `versions` | Versions of `morphobricks-preprocessing`, `morphobricks-normalization` (which computes the histograms and writes the CSV; absent in manifests written before it), `numpy`, `tifffile` and Python. Information only: a different version never causes a rebuild. |
 

docs/development.md

@@ -2,7 +2,7 @@
 
 ## Environment
 
-- Python 3.10 or newer.
+- Python 3.11 or newer.
 - Core dependencies: `numpy`, `tifffile` and `morphobricks-normalization` (installed from
   its GitHub release tag; it holds the histogram, CSV, pooling and normalization code).
   Notebook dependencies (`ipywidgets`,

docs/plan.md

@@ -168,6 +168,7 @@ Columns, in order: `position_id, image_file, z, c, t, condition, biological_repe
 technical_repeat, source_relative_path, plane_id, dtype, bit_depth, pixel_min, pixel_max,
 total_pixel_count, valid_pixel_count, binning_min, binning_max, bin_count,
 schema_version`, then `histogram_bin_00000` onwards. Details in contract §5.
+`position_id` is the image name without its extension (question 20).
 
 The wide CSV is for computational use; spreadsheet programs may not open all columns. The
 package provides `read_plane_histograms()`, which keeps labels as text. Alongside the CSV,
@@ -304,6 +305,10 @@ API, CLI). The notebook and CLI only call the API.
   and normalization modules come from `morphobricks-normalization` v0.1.0, a required
   dependency; the old names stay importable here with a `DeprecationWarning` for one
   version. Outputs are unchanged. *Implemented, see question 19.*
+- **Position identity (cross-brick decision D02):** every row and every manifest file
+  entry carries the image's `position_id`, so `condition`, `biological_repeat`,
+  `technical_repeat` and `position_id` identify a position in every brick; images in one
+  leaf folder with the same identifier are refused. *Implemented, see question 20.*
 
 ## 11. Tests and acceptance criteria
 
@@ -495,3 +500,22 @@ Raised during review; to be decided before the milestones that need them:
     the notebook's Step 0 installs the new dependency into environments set up before it;
     and the text of contract §3–§7 moved to the normalization brick's contract (§1.2 and
     §8.2), leaving short pointers here.*
+20. The CSV has a `position_id` column that this brick left blank, and the run manifest
+    had no position per file. How are positions identified across bricks (cross-brick
+    audit finding X02)?
+    *Decided (October 2026, cross-brick decision D02 of 26 October 2026): a position
+    (field of view) is identified in every brick by `condition`, `biological_repeat`,
+    `technical_repeat` and `position_id`; `position_id` falls back to the image file name
+    without its extension only when one file is exactly one position. Here it always is
+    (several series and position-like axes are errors: question 2 and contract §2), so
+    every row's `position_id` is the image name without `.tif`, `.tiff` or `.zarr` (and a
+    preceding `.ome`): `img_01.ome.tif` gives `img_01`. Equal names in two technical
+    repeats are two positions. Two images in one leaf folder with the same identifier,
+    ignoring case (`img_01.tif` and `img_01.ome.zarr`), are the error
+    `duplicate_position_id` rather than a guess; excluding one of them does not resolve
+    it, renaming or moving one does. The manifest gains the optional per-file key
+    `position_id` (null when absent) and stays at version 1, like the optional keys of
+    questions 12 and 15. The CSV columns and schema version 2 are unchanged; an earlier
+    table with blank `position_id` is filled on the next run from the paths, without
+    reading pixels. The normalization contract's note "never inferred from file names"
+    (§8.2) predates D02 and is to be aligned in that repository.*