From c1100e1ed4839af6148689fecee23bee9e935ec3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 07:13:55 +0000 Subject: [PATCH 1/2] Refuse images in a folder that also holds subfolders MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner decision (October 2026): a folder holding both image files and subfolders is an error in discovery, not a warning, as in every other brick (CellPatcher already refuses it). The code stays tiff_outside_leaf (OME-Zarr images included); its severity becomes error, so check/run block and nothing is written. It is no longer counted as a skipped item in the CLI report or the notebook tiles. Contract §1/§8, user guide, plan and AGENTS updated. Regression tests test_images_beside_subfolders_are_an_error and test_a_run_with_images_beside_subfolders_writes_nothing (both fail on main). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VbytvBRB9CmKQ5AAkhEFcv --- AGENTS.md | 4 +++- docs/contract.md | 6 ++--- docs/plan.md | 12 +++++----- docs/user_guide.md | 3 ++- src/morphobricks_preprocessing/discovery.py | 13 ++++++---- src/morphobricks_preprocessing/notebook_ui.py | 2 +- src/morphobricks_preprocessing/report.py | 1 - tests/test_discovery.py | 24 +++++++++++++++---- 8 files changed, 44 insertions(+), 21 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 22277ba6..4b6e9d99 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -198,7 +198,9 @@ functions the tests use. are the error `duplicate_position_id`: never guess. - **System files:** dot-files and dot-folders, `Thumbs.db`, `desktop.ini` and `__MACOSX` are ignored **without** a warning. -- **Warnings:** non-TIFF files, empty folders, TIFFs outside leaf folders. +- **Mixed folders:** images in a folder that also holds subfolders are the error + `tiff_outside_leaf` (owner decision, October 2026; every brick refuses such a folder). +- **Warnings:** non-TIFF files, empty folders. - **Paths:** paths in outputs are relative to the experiment root, with `/`. ### Axes (contract §2) diff --git a/docs/contract.md b/docs/contract.md index 273fb17c..4f914d47 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -20,7 +20,7 @@ whole experiment; any change to them goes through the normalization brick. | Rule | Detail | |---|---| | Images | TIFF files (`.tif`, `.tiff`, any case) and OME-Zarr images: folders whose names end in `.zarr` (any case, usually `.ome.zarr`). An OME-Zarr folder is one image, never a subfolder, and is not entered. Both may be mixed in one experiment. | -| Leaf folder | A folder that contains images and no subfolders. Only images directly in leaf folders are analysed. | +| Leaf folder | A folder that contains images and no subfolders. Only images directly in leaf folders are analysed. A folder holding both images and subfolders is the error `tiff_outside_leaf` (OME-Zarr images included): its images have no level, and every MorphoBricks brick refuses such a folder rather than skip them. | | Missing levels | A branch may stop early; the missing labels are `single`. If the root itself is the leaf, all three labels are `single`. | | Skipped levels | Not possible: a level only exists below its parent level. | | Folders named `single` | `single` is the MorphoBricks value of a level with one member, the value missing levels get. A folder named `single` in lower case is accepted as that value when it is the only folder in its place (system folders aside): `/single/.tif` gives the biological repeat and the technical repeat `single`, the labels of `/.tif`. It stays an error (`reserved_folder_name`) when it is spelled in another case (`Single`, a different label) or has other folders beside it (its level would then have several members, one called `single`). | @@ -118,8 +118,8 @@ method is an offset and a scale, `(pixels − offset) / scale`. | Severity | Situations | Effect | |---|---|---| -| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image, is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | -| Warning | Intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported | +| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image, is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked | +| Warning | Intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported | Every message states what happened in plain language, lists the affected paths, and suggests an action. Technical details go to the diagnostic log. diff --git a/docs/plan.md b/docs/plan.md index 36e76a0d..c726fd9a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -69,8 +69,9 @@ During preflight, show a tree/summary of the inferred labels and ask the user to or correct it. A folder may be named `single` (lower case) when it is the only folder in its place: it is then a level with one member, labelled `single` like a missing level (question 23). Spelled in another case, or beside other folders, it is an error. Report -warnings for files that are not images, empty folders and images outside leaf folders, and -explain that they will not be analysed. +warnings for files that are not images and empty folders, and explain that they will not be +analysed. Images in a folder that also holds subfolders are an error (`tiff_outside_leaf`; +owner decision, October 2026: every brick refuses such a folder). ## 3. Image metadata and axis review @@ -223,12 +224,11 @@ do not fit the experiment's roles; unsupported sample formats or axes; missing/duplicate X or Y, duplicate roles or inconsistent mapping; a plane that cannot be read within the memory limit, not even in blocks; a folder named `single` in another case or beside other folders; sibling folders whose names differ only in -case; a folder below the technical-repeat level; an output folder inside the experiment; -no usable planes. System files (dot-files, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are +case; a folder below the technical-repeat level; images in a folder that also holds +subfolders; an output folder inside the experiment; no usable planes. System files (dot-files, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are ignored silently. -Continue with warnings for: files that are not images; empty folders; images outside leaf -folders; +Continue with warnings for: files that are not images; empty folders; non-finite pixels (ignored and counted); different image sizes with a valid axis interpretation; planes with no finite values while others remain usable; planes over the memory limit that are read in blocks. diff --git a/docs/user_guide.md b/docs/user_guide.md index 854a5d7e..24d2089e 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -548,7 +548,8 @@ continues. | `unreadable_planes` | Error | Pixel data cannot be decoded. Export again, or exclude the image. | | `invalid_bin_count` | Error | Bin count outside the allowed range. Use the default or a value in range. | | `no_usable_planes` | Error | No plane has finite pixel values. Check the images. | -| `non_tiff_file`, `empty_folder`, `tiff_outside_leaf` | Warning | Files or folders that are skipped. | +| `tiff_outside_leaf` | Error | Images sit in a folder that also holds subfolders, so they have no condition or repeat of their own. A folder holds either images or subfolders, never both. Move the images into a folder of their own, or out of the experiment folder. | +| `non_tiff_file`, `empty_folder` | Warning | Files or folders that are skipped. | | `planes_read_in_blocks` | Warning | Planes too large for the memory limit are read in blocks. The results are the same; nothing to do. Raise the limit to read them whole, which is faster. | | `different_sizes` | Warning | Images have different widths or heights. Usually expected; otherwise check the camera settings. | | `different_plane_counts` | Warning | Images have different numbers of depth slices, time points or other planes. Images with more planes weigh more in pooled statistics. Check the acquisition if this is not expected. | diff --git a/src/morphobricks_preprocessing/discovery.py b/src/morphobricks_preprocessing/discovery.py index 33973306..64cfa5a5 100644 --- a/src/morphobricks_preprocessing/discovery.py +++ b/src/morphobricks_preprocessing/discovery.py @@ -453,13 +453,18 @@ def finish(self) -> tuple[ValidationIssue, ...]: ) if self.outside_leaf: issues.append( - warning( + # An error, not a warning (owner decision, October 2026): a folder holding + # both images and subfolders has no level for its images, and every brick + # refuses it, so they are never dropped silently. + error( "tiff_outside_leaf", f"{plural(len(self.outside_leaf), 'image')} {_are(self.outside_leaf)} " - "in folders that also contain subfolders and will not be analysed. Only " - "images in folders without subfolders are used.", + "in folders that also contain subfolders. A folder holds either images " + "or subfolders, never both, so these images have no place in the " + "experiment's levels.", self.outside_leaf, - "Move these images into a folder of their own if they should be analysed.", + "Move these images into a folder of their own, or out of the experiment " + "folder.", ) ) if not self.leaves: diff --git a/src/morphobricks_preprocessing/notebook_ui.py b/src/morphobricks_preprocessing/notebook_ui.py index cafbf080..8c59b1c2 100644 --- a/src/morphobricks_preprocessing/notebook_ui.py +++ b/src/morphobricks_preprocessing/notebook_ui.py @@ -149,7 +149,7 @@ # Problems a user can get past by leaving the affected images out (open question 6). EXCLUDABLE_CODES = ("unreadable_tiff", "multiple_series", "unsupported_dtype") -_SKIPPED_CODES = ("non_tiff_file", "tiff_outside_leaf", "empty_folder") +_SKIPPED_CODES = ("non_tiff_file", "empty_folder") # "What will happen" is a short summary, so its lists of images fold early. _PLAN_PATHS = 5 diff --git a/src/morphobricks_preprocessing/report.py b/src/morphobricks_preprocessing/report.py index 9ef0cf31..61e75be0 100644 --- a/src/morphobricks_preprocessing/report.py +++ b/src/morphobricks_preprocessing/report.py @@ -33,7 +33,6 @@ # Warnings whose paths are files or folders that will not be analysed. _SKIPPED_CODES = { "non_tiff_file": ("file that is not an image", "files that are not images"), - "tiff_outside_leaf": ("image outside leaf folders", "images outside leaf folders"), "empty_folder": ("empty folder", "empty folders"), } diff --git a/tests/test_discovery.py b/tests/test_discovery.py index 821b7f3b..ca8e175e 100644 --- a/tests/test_discovery.py +++ b/tests/test_discovery.py @@ -1,17 +1,20 @@ import os +import numpy as np import pytest from morphobricks_preprocessing import ( SINGLE, + AxisMapping, Severity, check_output_folder, default_output_folder, discover_experiment, position_id, + run_experiment, ) from morphobricks_preprocessing.discovery import is_system_name, is_tiff_name -from tiff_tree import make_tree +from tiff_tree import make_tree, write_data def codes(result): @@ -158,16 +161,29 @@ def test_empty_folders_are_warned(tmp_path): assert result.images == ("cond/a.tif",) -def test_tiffs_outside_leaf_folders_are_warned_and_skipped(tmp_path): +def test_images_beside_subfolders_are_an_error(tmp_path): + # Owner decision (October 2026): a folder holding both images and subfolders is refused, + # as by every other brick; its images are never skipped with only a warning. make_tree(tmp_path, ["stray.tif", "cond/bio/b.tif", "cond/bio/tech/a.tif"]) result = discover_experiment(tmp_path) - assert result.ok + assert not result.ok found = issue(result, "tiff_outside_leaf") - assert found.severity is Severity.WARNING + assert found.severity is Severity.ERROR assert found.paths == ("cond/bio/b.tif", "stray.tif") + assert "never both" in found.message and "folder of their own" in found.action assert result.images == ("cond/bio/tech/a.tif",) +def test_a_run_with_images_beside_subfolders_writes_nothing(tmp_path): + root, output = tmp_path / "exp", tmp_path / "out" + write_data(root / "cond" / "stray.tif", np.zeros((8, 9), dtype=np.uint8)) + write_data(root / "cond" / "bio" / "a.tif", np.zeros((8, 9), dtype=np.uint8)) + result = run_experiment(root, AxisMapping("YX", "YX"), output_folder=output) + assert not result.written + assert "tiff_outside_leaf" in [i.code for i in result.issues if i.is_error] + assert not output.exists() + + def test_system_files_are_ignored_silently(tmp_path): make_tree( tmp_path, From 4f8305a4dd0520d977675afc9fd7e39c98936f43 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 30 Oct 2026 07:26:34 +0000 Subject: [PATCH 2/2] Say in the user guide that images next to subfolders are an error Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VbytvBRB9CmKQ5AAkhEFcv --- docs/user_guide.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/user_guide.md b/docs/user_guide.md index 24d2089e..f5d4e1f7 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -73,7 +73,8 @@ experiment/ in `.zarr` (usually `.ome.zarr`); it counts as one image, not as a folder level, so put it where you would put a TIFF file. TIFF and OME-Zarr images can be mixed. - Only images in folders **without subfolders** are analysed. Images next to subfolders - are reported and skipped. + are an error (`tiff_outside_leaf`): a folder holds either images or subfolders, never + both, so move them into a folder of their own. - A folder may be named `single` (in lower case) when it is the only folder in its place: like a missing level, it means the level has one member, so `treated/single/img_01.tif` has the biological repeat `single`. A folder named `Single` or `SINGLE`, or a `single`