Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,9 @@ functions the tests use.
are the error `duplicate_position_id`: never guess.
- **System files:** dot-files and dot-folders, `Thumbs.db`, `desktop.ini` and `__MACOSX` are
ignored **without** a warning.
- **Warnings:** non-TIFF files, empty folders, TIFFs outside leaf folders.
- **Mixed folders:** images in a folder that also holds subfolders are the error
`tiff_outside_leaf` (owner decision, October 2026; every brick refuses such a folder).
- **Warnings:** non-TIFF files, empty folders.
- **Paths:** paths in outputs are relative to the experiment root, with `/`.

### Axes (contract §2)
Expand Down
6 changes: 3 additions & 3 deletions docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ whole experiment; any change to them goes through the normalization brick.
| Rule | Detail |
|---|---|
| Images | TIFF files (`.tif`, `.tiff`, any case) and OME-Zarr images: folders whose names end in `.zarr` (any case, usually `.ome.zarr`). An OME-Zarr folder is one image, never a subfolder, and is not entered. Both may be mixed in one experiment. |
| Leaf folder | A folder that contains images and no subfolders. Only images directly in leaf folders are analysed. |
| Leaf folder | A folder that contains images and no subfolders. Only images directly in leaf folders are analysed. A folder holding both images and subfolders is the error `tiff_outside_leaf` (OME-Zarr images included): its images have no level, and every MorphoBricks brick refuses such a folder rather than skip them. |
| Missing levels | A branch may stop early; the missing labels are `single`. If the root itself is the leaf, all three labels are `single`. |
| Skipped levels | Not possible: a level only exists below its parent level. |
| Folders named `single` | `single` is the MorphoBricks value of a level with one member, the value missing levels get. A folder named `single` in lower case is accepted as that value when it is the only folder in its place (system folders aside): `<condition>/single/<image>.tif` gives the biological repeat and the technical repeat `single`, the labels of `<condition>/<image>.tif`. It stays an error (`reserved_folder_name`) when it is spelled in another case (`Single`, a different label) or has other folders beside it (its level would then have several members, one called `single`). |
Expand Down Expand Up @@ -118,8 +118,8 @@ method is an offset and a scale, `(pixels − offset) / scale`.

| Severity | Situations | Effect |
|---|---|---|
| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image, is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked |
| Warning | Intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images outside leaf folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported |
| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images, or converting TIFF images to OME-Zarr, without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles; every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); sibling folders differing only in case; images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment; no usable planes; when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named Z/T (`mask_not_2d`), is of another size than its image, is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked |
| Warning | Intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no analysed image, files outside the mask-type folders (§15) | Processing continues; the item is reported |

Every message states what happened in plain language, lists the affected paths, and
suggests an action. Technical details go to the diagnostic log.
Expand Down
12 changes: 6 additions & 6 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,9 @@ During preflight, show a tree/summary of the inferred labels and ask the user to
or correct it. A folder may be named `single` (lower case) when it is the only folder in
its place: it is then a level with one member, labelled `single` like a missing level
(question 23). Spelled in another case, or beside other folders, it is an error. Report
warnings for files that are not images, empty folders and images outside leaf folders, and
explain that they will not be analysed.
warnings for files that are not images and empty folders, and explain that they will not be
analysed. Images in a folder that also holds subfolders are an error (`tiff_outside_leaf`;
owner decision, October 2026: every brick refuses such a folder).

## 3. Image metadata and axis review

Expand Down Expand Up @@ -223,12 +224,11 @@ do not fit the experiment's roles; unsupported sample
formats or axes; missing/duplicate X or Y, duplicate roles or inconsistent mapping; a plane
that cannot be read within the memory limit, not even in blocks; a folder named `single`
in another case or beside other folders; sibling folders whose names differ only in
case; a folder below the technical-repeat level; an output folder inside the experiment;
no usable planes. System files (dot-files, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are
case; a folder below the technical-repeat level; images in a folder that also holds
subfolders; an output folder inside the experiment; no usable planes. System files (dot-files, `Thumbs.db`, `desktop.ini`, `__MACOSX`) are
ignored silently.

Continue with warnings for: files that are not images; empty folders; images outside leaf
folders;
Continue with warnings for: files that are not images; empty folders;
non-finite pixels (ignored and counted); different image sizes with a valid axis
interpretation; planes with no finite values while others remain usable; planes over the
memory limit that are read in blocks.
Expand Down
6 changes: 4 additions & 2 deletions docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ experiment/
in `.zarr` (usually `.ome.zarr`); it counts as one image, not as a folder level, so put
it where you would put a TIFF file. TIFF and OME-Zarr images can be mixed.
- Only images in folders **without subfolders** are analysed. Images next to subfolders
are reported and skipped.
are an error (`tiff_outside_leaf`): a folder holds either images or subfolders, never
both, so move them into a folder of their own.
- A folder may be named `single` (in lower case) when it is the only folder in its place:
like a missing level, it means the level has one member, so `treated/single/img_01.tif`
has the biological repeat `single`. A folder named `Single` or `SINGLE`, or a `single`
Expand Down Expand Up @@ -548,7 +549,8 @@ continues.
| `unreadable_planes` | Error | Pixel data cannot be decoded. Export again, or exclude the image. |
| `invalid_bin_count` | Error | Bin count outside the allowed range. Use the default or a value in range. |
| `no_usable_planes` | Error | No plane has finite pixel values. Check the images. |
| `non_tiff_file`, `empty_folder`, `tiff_outside_leaf` | Warning | Files or folders that are skipped. |
| `tiff_outside_leaf` | Error | Images sit in a folder that also holds subfolders, so they have no condition or repeat of their own. A folder holds either images or subfolders, never both. Move the images into a folder of their own, or out of the experiment folder. |
| `non_tiff_file`, `empty_folder` | Warning | Files or folders that are skipped. |
| `planes_read_in_blocks` | Warning | Planes too large for the memory limit are read in blocks. The results are the same; nothing to do. Raise the limit to read them whole, which is faster. |
| `different_sizes` | Warning | Images have different widths or heights. Usually expected; otherwise check the camera settings. |
| `different_plane_counts` | Warning | Images have different numbers of depth slices, time points or other planes. Images with more planes weigh more in pooled statistics. Check the acquisition if this is not expected. |
Expand Down
13 changes: 9 additions & 4 deletions src/morphobricks_preprocessing/discovery.py
Original file line number Diff line number Diff line change
Expand Up @@ -453,13 +453,18 @@ def finish(self) -> tuple[ValidationIssue, ...]:
)
if self.outside_leaf:
issues.append(
warning(
# An error, not a warning (owner decision, October 2026): a folder holding
# both images and subfolders has no level for its images, and every brick
# refuses it, so they are never dropped silently.
error(
"tiff_outside_leaf",
f"{plural(len(self.outside_leaf), 'image')} {_are(self.outside_leaf)} "
"in folders that also contain subfolders and will not be analysed. Only "
"images in folders without subfolders are used.",
"in folders that also contain subfolders. A folder holds either images "
"or subfolders, never both, so these images have no place in the "
"experiment's levels.",
self.outside_leaf,
"Move these images into a folder of their own if they should be analysed.",
"Move these images into a folder of their own, or out of the experiment "
"folder.",
)
)
if not self.leaves:
Expand Down
2 changes: 1 addition & 1 deletion src/morphobricks_preprocessing/notebook_ui.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
# Problems a user can get past by leaving the affected images out (open question 6).
EXCLUDABLE_CODES = ("unreadable_tiff", "multiple_series", "unsupported_dtype")

_SKIPPED_CODES = ("non_tiff_file", "tiff_outside_leaf", "empty_folder")
_SKIPPED_CODES = ("non_tiff_file", "empty_folder")

# "What will happen" is a short summary, so its lists of images fold early.
_PLAN_PATHS = 5
Expand Down
1 change: 0 additions & 1 deletion src/morphobricks_preprocessing/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
# Warnings whose paths are files or folders that will not be analysed.
_SKIPPED_CODES = {
"non_tiff_file": ("file that is not an image", "files that are not images"),
"tiff_outside_leaf": ("image outside leaf folders", "images outside leaf folders"),
"empty_folder": ("empty folder", "empty folders"),
}

Expand Down
24 changes: 20 additions & 4 deletions tests/test_discovery.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
import os

import numpy as np
import pytest

from morphobricks_preprocessing import (
SINGLE,
AxisMapping,
Severity,
check_output_folder,
default_output_folder,
discover_experiment,
position_id,
run_experiment,
)
from morphobricks_preprocessing.discovery import is_system_name, is_tiff_name
from tiff_tree import make_tree
from tiff_tree import make_tree, write_data


def codes(result):
Expand Down Expand Up @@ -158,16 +161,29 @@ def test_empty_folders_are_warned(tmp_path):
assert result.images == ("cond/a.tif",)


def test_tiffs_outside_leaf_folders_are_warned_and_skipped(tmp_path):
def test_images_beside_subfolders_are_an_error(tmp_path):
# Owner decision (October 2026): a folder holding both images and subfolders is refused,
# as by every other brick; its images are never skipped with only a warning.
make_tree(tmp_path, ["stray.tif", "cond/bio/b.tif", "cond/bio/tech/a.tif"])
result = discover_experiment(tmp_path)
assert result.ok
assert not result.ok
found = issue(result, "tiff_outside_leaf")
assert found.severity is Severity.WARNING
assert found.severity is Severity.ERROR
assert found.paths == ("cond/bio/b.tif", "stray.tif")
assert "never both" in found.message and "folder of their own" in found.action
assert result.images == ("cond/bio/tech/a.tif",)


def test_a_run_with_images_beside_subfolders_writes_nothing(tmp_path):
root, output = tmp_path / "exp", tmp_path / "out"
write_data(root / "cond" / "stray.tif", np.zeros((8, 9), dtype=np.uint8))
write_data(root / "cond" / "bio" / "a.tif", np.zeros((8, 9), dtype=np.uint8))
result = run_experiment(root, AxisMapping("YX", "YX"), output_folder=output)
assert not result.written
assert "tiff_outside_leaf" in [i.code for i in result.issues if i.is_error]
assert not output.exists()


def test_system_files_are_ignored_silently(tmp_path):
make_tree(
tmp_path,
Expand Down
Loading