Experiment design identity and counts: excluded folders, duplicate columns, output folder (#82, #83, #98)

AGENTS.md

@@ -79,6 +79,8 @@ src/morphobricks_preprocessing/
   design.py         experiment design (P3): ExperimentDesign/DesignRow, default pairing, reconcile with
                     the folders on reruns, design_issues, read/write experiment_design.csv, load_design;
                     read_confirmed_design/ConfirmedDesign (the strict reader other bricks use, §16)
+  design_values.py  the design file's header check (no column twice or unnamed, #83), booleans,
+                    counts, dates and DesignFormatError (pure; re-exported by design.py)
   image_type.py     image type (P4): FLUORESCENCE/RGB, review_image_type (suggestion from
                     TiffMetadata.rgb_samples and the confirmed roles, whether RGB is possible via the
                     brick's check_rgb), image_type_issues (rgb_not_possible)
@@ -501,10 +503,12 @@ functions the tests use.
   get one whole-experiment question. Controls are recorded only.
 - **`experiment_design.csv`** (schema 1): one row per condition x biological-repeat folder,
   sorted; counts at confirmation; `confirmed_on` ISO date. Lenient reader (any-case
-  booleans, BOM); `DesignFormatError` otherwise, also for a repeat group that is not a
+  booleans, BOM); `DesignFormatError` otherwise, also for a header naming a column twice
+  or not at all (#83, both readers), a repeat group that is not a
   D-e identity or a `confirmed_on` that is not `YYYY-MM-DD` (naming line and column).
   Other bricks read it with the strict `read_confirmed_design` (contract §16,
-  question 40): nothing filled in, every group confirmed, SHA-256 of the bytes parsed.
+  question 40): nothing filled in, every group confirmed, SHA-256 of the bytes parsed;
+  `RunInputs.design()` uses it too (#83).
 - **Reruns:** `reconcile_design` keeps answers for known folders, reopens only groups that
   gain a folder, drops removed folders.
 - **Notebook** writes the file after every answer once the labels are confirmed; a
@@ -765,8 +769,10 @@ plainer version "not interactive enough". Descriptive, guided and forgiving is t
    3 shows a "Before you run" checklist naming the first thing still to do, and Run stays
    disabled until it is empty.
 2. **Inputs reset what depends on them.** Changing the experiment folder resets the
-   preflight, the confirmations and the axis roles. Changing an axis role resets the
-   preview confirmation. Choosing another CSV clears the old statistics.
+   preflight, the confirmations and the axis roles. Changing the output folder reads the
+   experiment design from the new folder, or asks it again (#98); never carried over.
+   Changing an axis role resets the preview confirmation. Choosing another CSV clears the
+   old statistics.
 3. **Explain every choice live.** Each axis card shows its length, the suggested role and a
    dropdown with descriptive options ("Z - depth (focal planes)"). Under it, a sentence
    updates with the choice ("3 depth slices, numbered z = 0 to 2; each becomes its own

docs/plan.md

@@ -551,6 +551,12 @@ API, CLI). The notebook and CLI only call the API.
   compared at reuse; each header bound to its image's fingerprint
   (`preflight(bind_headers=True)`, `source_changed_during_check`); the copy's pixel type
   from that header; `parse_size` exact.
+- **Before the audit: experiment design identity and counts (issues #82, #83 and #98)**
+  *(this PR; questions 61, 62 and 63)*: technical-repeat folders whose images are all
+  excluded are not counted, and the balance text names them; every design reader refuses a
+  header naming a column twice or a column without a name, and `RunInputs.design()` reads
+  with `read_confirmed_design`; changing the notebook's output folder reloads the design
+  from the new folder and asks again.
 - **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release
   candidates, in the release PR before the tag is pushed:
   - set `version` in `pyproject.toml`;
@@ -1548,4 +1554,38 @@ Raised during review; to be decided before the milestones that need them:
     (review finding: the PR dropped the warning when only X and Y were stated). Only
     these metadata problems are caught: any other exception while reading the scale makes
     the image unreadable instead of being hidden by a debug log.
-
+61. Which technical-repeat folders are counted (issue #82)?
+    *Decided (owner, 2 November 2026):* only those holding at least one image to analyse.
+    A technical-repeat folder whose images are all excluded is not counted in
+    `BiologicalRepeat.technical_repeat_count`, the balance table or the design's
+    `technical_repeat_count` (it was counted before). The balance table's text says what
+    was left out (`BalanceTable.not_counted()`: the excluded images and the folders whose
+    images are all excluded), in the CLI `check`, the notebook's consistency card and its
+    design card. A technical-repeat folder named `single` keeps question 23's meaning
+    (the level's value `single`, counted as no technical-repeat folder, like a missing
+    level): documented, not changed. Contract §11, §12; user guide §4.
+62. Which design file headers are read, and how does `RunInputs.design()` read the file
+    (issue #83)?
+    *Decided (owner, 2 November 2026):* a header that names a column twice, or has a
+    column without a name (empty or only whitespace), is refused by every design reader
+    (`read_experiment_design`, so `load_design`, the notebook, `check`, `run --design` and
+    the `design` command, and `read_confirmed_design`), with `DesignFormatError` naming
+    the column and both positions; the lenient reader kept the last of two columns before,
+    and a test asserted it. Names are compared ignoring case and outer whitespace (and,
+    from the review, invisible format characters such as a byte-order mark or zero-width
+    space inside the header, which made a second `condition` look new). Nothing
+    is rewritten on refusal. `RunInputs.design()` reads with `read_confirmed_design`
+    checked against the manifest's SHA-256 (it used the lenient reader). The header check
+    and the value parsers moved to `design_values` (pure); `DesignFormatError` is
+    re-exported from `design` unchanged. Contract §9.1, §12, §16; user guide §8.
+63. What happens to the experiment design when the notebook's output folder changes (issue
+    #98)?
+    *Decided (owner, 2 November 2026):* the design belongs to its output folder. Since
+    PP-10a a run reported the other folder's design as a design change but used it, so a
+    design (its confirmations and controls) confirmed for one output folder could be
+    written into another. Now changing the output folder reads the design from the new
+    folder (`load_design`, an unreadable file refused and kept), or asks the questions
+    again when it holds none; nothing is carried over and nothing is written until an
+    answer (the folder may still be being typed), and the design card says which. The run
+    is gated as before, so it runs only on the new folder's confirmed design. No copy
+    action: the user answers again, or copies the file. Contract §12; user guide §4.

docs/user_guide.md

@@ -244,7 +244,9 @@ results or comparing another grouping.
    version.
 1. **Check the experiment folder.**
    - Paste the folder path, or click **Browse…** to choose it in a dialog. Below the output
-     folder, a line shows where the results will go.
+     folder, a line shows where the results will go. An output folder that is a file, or
+     inside one (for example `somefile/out`), cannot be created: the line says so in red,
+     nothing is written there, and step 3 waits until you choose another folder.
    - If you have masks (§2), give their folder under **Masks folder (optional)** the same way
      (on Colab, paste its path: **Browse…** explains how). Leave it empty for no masks.
      **Check folder** pairs them with the images exactly as `check --masks` does.
@@ -292,6 +294,11 @@ results or comparing another grouping.
        naming the line and column, and the file is left as it is. Correct the file and
        undo and confirm the labels again to read it, or answer the questions again: the
        first answer replaces the file.
+     - The design belongs to its output folder. If you change the output folder, the design
+       is read from the new folder (its own answers and controls, a blue box says so), or,
+       when it holds none, the questions are asked again: answers given for the previous
+       output folder are never carried over, and nothing is written in the new folder until
+       you answer. Changing back reads the first folder's design again.
    - **Pixel sizes (calibration)** lists the size of one pixel along X, Y, Z and T for every
      image (images with the same sizes share a row), with its source: **header** (the image
      metadata), **stated** (in **calibration.csv**), **pixels** (stated as deliberately
@@ -307,7 +314,11 @@ results or comparing another grouping.
      as the run records no size for it. No histogram depends on the sizes. (When another
      brick embeds this step without the output folder, the card is left out.)
    - **Dataset consistency** shows how many images each condition (columns) and
-     biological repeat (rows; the repeat groups once paired) has, with the number of technical-repeat folders. A cell is
+     biological repeat (rows; the repeat groups once paired) has, with the number of technical-repeat folders. Excluded images
+     are not counted, nor is a technical-repeat folder whose images are all excluded; the
+     line under the table says how many images were excluded and names those folders. A
+     technical-repeat folder named **single** is the level's value single, so it counts as
+     no technical-repeat folder, like images directly in the biological-repeat folder. A cell is
      amber when it has fewer than half or more than twice the median number of images, or
      when a condition does not have that biological repeat. Expand **Show consistency
      warnings** to see the checks that compare the images with each other: pixel type,
@@ -963,8 +974,8 @@ continues.
 | `rgb_not_possible` | Error | The image type RGB was chosen, but the images are not three channels of a whole-number pixel type. Choose fluorescence, or check the axis roles: the RGB samples must have the role C. |
 | `repeat_group_conflict` | Error | A repeat group holds two folders of the same condition. Move one into another group. |
 | `invalid_experiment_design` | Error | A repeat group has no usable name (empty, `NA`, `nan`, `none`, `null`, `n/a`, `<NA>`, or spaces around it), two repeat groups are spelled the same apart from upper and lower case, or a confirmation date is not written `YYYY-MM-DD`. Correct `experiment_design.csv` (the message names the line and column). |
-| `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. |
-| `unreadable_experiment_design` | Error (`load_design`, notebook step 1, `check`) | The saved `experiment_design.csv` in the output folder cannot be read even leniently (for example a date a spreadsheet program rewrote as 26/10/2026, or a repeat group `NA`); the message names the line and column. The file is left as it is. Correct it, or answer the questions again in step 1, which replaces it. |
+| `missing_experiment_design`, `unreadable_experiment_design` | Error (other bricks) | A brick that pools biological repeats as `experiment_design.csv` pairs them found no such file, or one it cannot use as written by the preprocessing brick (another column order, a column named twice or without a name, a value with spaces around it, `TRUE` or `yes` for `true`, an empty repeat group, a byte-order mark from a spreadsheet program, ...). Run the preprocessing brick on the experiment again: its run writes the file as other bricks read it. |
+| `unreadable_experiment_design` | Error (`load_design`, notebook step 1, `check`) | The saved `experiment_design.csv` in the output folder cannot be read even leniently (for example a date a spreadsheet program rewrote as 26/10/2026, a repeat group `NA`, or a header that names a column twice, such as two `repeat_group` columns, or has a column without a name); the message names the line and column. No reader chooses between two columns of one name: delete one. The file is left as it is. Correct it, or answer the questions again in step 1, which replaces it. |
 | `changed_experiment_design` | Error (other bricks) | `experiment_design.csv` is not the file the preprocessing run that wrote the statistics used (its SHA-256 differs from the one in the run manifest). Give that run's design, or run the preprocessing brick again. |
 | `image_intensity_outlier` | Warning (intensity check) | An image's median or high percentile in a channel is far from the other images of its condition and biological repeat. Look at it: it may be out of focus, contain debris or bubbles, or have been acquired differently. Exclude it if it is faulty. |
 | `repeat_intensity_shift` | Warning (intensity check) | A biological repeat is brighter or darker than the other repeats of its condition. Check whether it was prepared or acquired differently; normalising per biological repeat in later bricks removes such differences. |

pyproject.toml

@@ -171,6 +171,7 @@ baseline = "tools/code_rules_baseline.json"
 "morphobricks_preprocessing.discovery_report" = "pure"
 "morphobricks_preprocessing.dataset" = "pure"
 "morphobricks_preprocessing.design" = "io"
+"morphobricks_preprocessing.design_values" = "pure"
 "morphobricks_preprocessing.design_edit" = "pure"
 "morphobricks_preprocessing.discovery" = "io"
 "morphobricks_preprocessing.image_type" = "pure"

src/morphobricks_preprocessing/dataset.py

@@ -14,7 +14,7 @@
 
 import statistics
 from collections.abc import Iterable, Mapping
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
 from types import MappingProxyType
 
 from morphobricks_normalization.records import SINGLE
@@ -60,8 +60,11 @@ class BiologicalRepeat:
     image_count
         Images to analyse in it, over all its technical repeats.
     technical_repeat_count
-        Technical-repeat folders in it; 0 when its images are directly in the
-        biological-repeat folder.
+        Technical-repeat folders in it with images to analyse; 0 when its images are
+        directly in the biological-repeat folder. A folder whose images are all excluded
+        is not counted (issue #82), and neither is a technical-repeat folder named
+        ``single``: it is the level's value ``single``, like a missing level (plan §12
+        question 23).
     """
 
     condition: str
@@ -86,8 +89,8 @@ class BalanceCell:
         Images, over all technical repeats (and all folders, if a repeat group
         holds more than one folder of the condition).
     technical_repeat_count
-        Technical-repeat folders; 0 when the images are directly in the
-        biological-repeat folder.
+        Technical-repeat folders with images to analyse; 0 when the images are directly
+        in the biological-repeat folder (as :class:`BiologicalRepeat`).
     folders
         The biological-repeat folders in the cell.
     unbalanced
@@ -116,6 +119,11 @@ class BalanceTable:
         images has no cell.
     median
         Median image count of the cells with images; None without images.
+    excluded_images
+        Images left out by the user, not counted.
+    excluded_folders
+        Leaf folders whose images are all excluded, sorted: not counted as technical
+        (or biological) repeats.
     """
 
     conditions: tuple[str, ...] = ()
@@ -124,6 +132,8 @@ class BalanceTable:
         default_factory=lambda: MappingProxyType({})
     )
     median: float | None = None
+    excluded_images: int = 0
+    excluded_folders: tuple[str, ...] = ()
 
     def cell(self, repeat_group: str, condition: str) -> BalanceCell | None:
         """Return the cell of a repeat group in a condition, or None if it has no images."""
@@ -133,6 +143,18 @@ def missing_conditions(self, repeat_group: str) -> tuple[str, ...]:
         """Return the conditions in which a repeat group has no folder."""
         return tuple(c for c in self.conditions if (repeat_group, c) not in self.cells)
 
+    def not_counted(self) -> str:
+        """Say what the counts leave out: excluded images and folders; empty without any."""
+        if not self.excluded_images:
+            return ""
+        text = f"Not counted: {plural(self.excluded_images, 'excluded image')}"
+        if self.excluded_folders:
+            text += (
+                ", so these folders, whose images are all excluded, are not counted as "
+                f"repeats: {', '.join(self.excluded_folders)}"
+            )
+        return text + "."
+
 
 @dataclass(frozen=True)
 class DatasetReport:
@@ -177,8 +199,11 @@ def biological_repeats(
         entry = found.setdefault(
             (leaf.condition, leaf.biological_repeat), [_repeat_folder(leaf), 0, 0]
         )
-        entry[1] += sum(image not in excluded for image in leaf.images)
-        entry[2] += leaf.technical_repeat != SINGLE
+        included = sum(image not in excluded for image in leaf.images)
+        entry[1] += included
+        # A technical-repeat folder whose images are all excluded is not analysed, so it
+        # does not count (issue #82); a folder named single is the level's value (Q23).
+        entry[2] += bool(included) and leaf.technical_repeat != SINGLE
     # A folder whose images are all excluded is not analysed, so it is not listed.
     return [
         BiologicalRepeat(condition, repeat, folder, images, technical)
@@ -256,7 +281,8 @@ def check_dataset(
         repeats across conditions; by default repeats with the same folder name
         are one group.
     excluded
-        Images left out by the user; they are not counted.
+        Images left out by the user; they are not counted, nor are the folders whose
+        images are all excluded (:meth:`BalanceTable.not_counted` says which).
 
     Returns
     -------
@@ -266,8 +292,16 @@ def check_dataset(
         and ``repeat_missing_conditions`` (a repeat group not found in every
         condition).
     """
+    excluded = set(excluded)
     repeats = biological_repeats(discovery, excluded)
-    balance = balance_table(repeats, repeat_groups)
+    leaves = [leaf for leaf in discovery.leaves if leaf.images]
+    balance = replace(
+        balance_table(repeats, repeat_groups),
+        excluded_images=sum(image in excluded for leaf in leaves for image in leaf.images),
+        excluded_folders=tuple(
+            sorted(leaf.relative_path for leaf in leaves if excluded.issuperset(leaf.images))
+        ),
+    )
     issues = [*_unbalanced_issues(balance), *_missing_issues(balance)]
     return DatasetReport(tuple(repeats), balance, tuple(issues))