Links stay inside the experiment and masks folders; --log is checked before it opens (#81, #99)

AGENTS.md

@@ -64,6 +64,8 @@ src/morphobricks_preprocessing/
   discovery.py      folder walk, leaf folders, labels (condition / biological_repeat / technical_repeat),
                     position_id per image and the duplicate check; FOV_KEY, check_identity_text,
                     check_plane_index, check_identity_case (the shared identity rules, contract §16)
+  links.py          LinkGuard: links followed only inside the folder walked (experiment, OME-Zarr
+                    stores, masks folder); link_outside_folder, link_loop, linked_twice (#81)
   metadata.py       TIFF header inspection (tifffile, no pixel reads), dtype/dimension/series checks,
                     OME channel names + significant bits, physical sizes (TiffMetadata.physical_sizes);
                     header consistency warnings (width/height, plane counts, pixel sizes);
@@ -138,7 +140,7 @@ src/morphobricks_preprocessing/
                     manifest SHA-256, what was found, reused, recomputed and replaced
   cli.py, __main__  `morphobricks-preprocessing check|run|masks|calibrate|design|convert` (also `python -m morphobricks_preprocessing`);
                     cli_run.py holds `run`'s options and output; cli_calibrate.py `calibrate`;
-                    cli_convert.py `convert`
+                    cli_convert.py `convert`; log_target.py where `--log` may write (#99)
   masks/           masks (Q32; a package since PP-5: model, files, header, layout, read,
                     checks, labels): find_masks pairs every image with its masks (masks folder with one
                     folder per type mirroring the experiment, or <image>_<type>.tif in the

docs/contract.md

@@ -27,6 +27,7 @@ whole experiment; any change to them goes through the normalization brick.
 | Uniqueness | Folder names need only be unique among siblings, ignoring case. Two folder names of one level (condition, biological repeat or technical repeat) that differ only in case are the error `case_conflict`, whether siblings or anywhere in the experiment (`ctrl/Rep1` and `treated/rep1`; decision D-e, §16; siblings only before D86). |
 | Names | Every label and `position_id` passes `check_identity_text` (§16): no leading or trailing whitespace, and not `""`, `NA`, `nan`, `none`, `null`, `n/a` or `<NA>` in any case, which table readers turn into a missing value. A folder or image name that fails is the error `invalid_name` (decision D-e; accepted before D86). |
 | Depth | At most three folder levels below the root. A folder below the technical-repeat level is an error. |
+| Links | Contain (issue #81, owner 2 November 2026): a symbolic link to a file or folder (on Windows also a directory junction, which Python 3.11 does not report as a link) is followed only when its real path (`os.path.realpath`) is inside the experiment folder, itself resolved. A link that leaves it is not followed and is the error `link_outside_folder`, naming each link and its target (a broken link too); a link to a folder on its own path (the folder holding it, or one above) is `link_loop`; one file reached by two paths (a link and its target, two links, a linked folder and its target, or two hard links: the same device and inode, from the one `stat` taken per image or mask; owner, 2 November 2026) is `linked_twice`, naming both. A broken link that stays inside is kept and reported by its reader (`unreadable_tiff`, `unreadable_mask`), never skipped. Hidden and system names (§1) are skipped before any link is looked at, so a hidden link is never followed; a contained link *to* a hidden folder is followed, because that folder is itself never walked, so its files are found once, under the link's name, which the user chose. The links inside an OME-Zarr store are held to the same rule, though the store is not entered as a folder level: its folders are listed and only its links resolved, about 84 ms per 20,000 store files on each discovery (accepted by the owner, 2 November 2026). The masks folder follows the same rule (§15), with `RunInputs` (§9.1) and `convert` (§17) refusing the same links; one helper, `links.LinkGuard`, walks both folders. |
 | System files | Dot-files and dot-folders (`.DS_Store`, `._*`, `.ipynb_checkpoints`), `Thumbs.db`, `desktop.ini` and `__MACOSX` are ignored without a warning. |
 | Scope | A group is identified by its full label path, e.g. `(control, bio_rep1)`. |
 | Positions | Every image is exactly one position (field of view): files with several positions, scenes, series or tiles are refused (§2). Its `position_id` is the image name without its extension (`.tif`, `.tiff` or `.zarr`, any case, with a preceding `.ome`): `img_01.ome.tif` gives `img_01`; a name that is only an extension (`.tif`, `.ome.zarr`) is refused. `discovery.position_id` is the one function every MorphoBricks brick names positions with (cross-brick decision D85, 29 October 2026), so it also removes the other image extensions bricks read (`.png`, `.jpg`, `.jpeg`, `.bmp`, `.gif`, `.npy`; naming only: this brick still analyses TIFF and Zarr images only). A position is identified by `condition`, `biological_repeat`, `technical_repeat` and `position_id` (cross-brick decision D02, 26 October 2026), so equal names in two technical repeats are two positions. Two images in one leaf folder with the same `position_id`, ignoring case (`img_01.tif` and `img_01.ome.zarr`), are an error (`duplicate_position_id`): the brick does not guess which is which. |
@@ -121,11 +122,22 @@ method is an offset and a scale, `(pixels − offset) / scale`.
 
 | Severity | Situations | Effect |
 |---|---|---|
-| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles or that the confirmed roles contradict (`ome_zarr_axes_mismatch`, §2); OME-Zarr images that do not name their axes when their OME-NGFF version does not fix them (`unreadable_ome_zarr`, §2); every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X, contradict its header or put X before Y (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment or inside the masks folder (`output_inside_masks`, §15); images named as masks of other images (`image_named_as_mask`, §1); no usable planes; no axis roles given or recorded (`axes_not_confirmed`, §9.2); another run holding the output folder's lock (`output_locked`, §9.2); a copy tree whose conversion record cannot be read (`conversion_record_unreadable`, §17); a recorded `experiment_design.csv` that cannot be read (`design_unreadable`); a `calibration.csv` that cannot be read (`calibration_unreadable`) or that states sizes for a target that is neither a folder nor an image of the experiment (`calibration_target_not_found`, §9.3); no experiment folder given or recorded (`experiment_not_recorded`, §9.2); when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named or stated Z/T (`mask_not_2d`), has stated axes that do not fit it or contradict the axes its metadata names (`invalid_mask_axes`, `mask_axes_contradict_header`), is stored with X before Y (its header's or its stated axes; refused by `check` and `run` too, never recorded with empty axes) or with its image's Y and X sizes swapped (`mask_layout`; never transposed), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked |
+| Error | Experiment design not confirmed (`design_not_confirmed`) or a repeat group holding two folders of one condition (`repeat_group_conflict`), when a design is used (§12); image type RGB for images that are not three channels of an integer type (`rgb_not_possible`, §13); mixed dtypes; images with different numbers of channels (`different_channel_counts`, §2) or different channel names at the same channel number (`channel_names_differ`, §2); unreadable TIFF (header or pixel data) or OME-Zarr folder in a leaf folder that is not excluded; OME-Zarr images without the `zarr` extra; OME-Zarr images whose declared axes do not fit the experiment's roles or that the confirmed roles contradict (`ome_zarr_axes_mismatch`, §2); OME-Zarr images that do not name their axes, and every OME-NGFF 0.1 or 0.2 store (`unreadable_ome_zarr`, §2); every image excluded; more than one image series in a file; unsupported sample format or axis; missing/duplicate X or Y, duplicate roles, unassigned non-singleton axis, inconsistent mapping; an image whose confirmed roles swap Y and X, contradict its header or put X before Y (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`, §2); plane that cannot be read within the memory limit, not even in blocks (for example a plane stored compressed in one piece); folder named `single` in another case, or beside other folders (§1); folders of one level differing only in case, anywhere in the experiment (`case_conflict`); folder or image names with outer whitespace or reading as a missing value (`invalid_name`, §1); images in one leaf folder with the same position identifier (§1); links that leave the experiment or masks folder, loop, or reach one file twice (`link_outside_folder`, `link_loop`, `linked_twice`, §1); folder below the technical-repeat level; images in a folder that also holds subfolders (`tiff_outside_leaf`, §1); output folder inside the experiment or inside the masks folder (`output_inside_masks`, §15); images named as masks of other images (`image_named_as_mask`, §1); no usable planes; no axis roles given or recorded (`axes_not_confirmed`, §9.2); another run holding the output folder's lock (`output_locked`, §9.2); a copy tree whose conversion record cannot be read (`conversion_record_unreadable`, §17); a recorded `experiment_design.csv` that cannot be read (`design_unreadable`); a `calibration.csv` that cannot be read (`calibration_unreadable`) or that states sizes for a target that is neither a folder nor an image of the experiment (`calibration_target_not_found`, §9.3); no experiment folder given or recorded (`experiment_not_recorded`, §9.2); when masks are given, a masks folder missing or inside the experiment, mask names differing only in case, two masks of one type for an image, a mask whose name could belong to two images or types, or a mask that is unreadable, has axes other than Y, X and named or stated Z/T (`mask_not_2d`), has stated axes that do not fit it or contradict the axes its metadata names (`invalid_mask_axes`, `mask_axes_contradict_header`), is stored with X before Y (its header's or its stated axes; refused by `check` and `run` too, never recorded with empty axes) or with its image's Y and X sizes swapped (`mask_layout`; never transposed), is of another size than its image (in `run`, its image's Y and X under the confirmed roles; not reported for an image with an axis error), is not of integers or booleans, has other depth slices or time points than the image's other masks or, in `run`, than the image (§15) | Processing is blocked |
 | Warning | Images whose metadata does not name their axes, read with the confirmed roles (`axes_not_named`, §2); in a tree of named-axis copies, sources changed or gone since the conversion (`conversion_source_changed`) and copies changed, gone or added since (`copy_changed_after_conversion`, §17); folders named like a masks folder holding images named as masks (`mask_folder_in_experiment`, §1); intensity checks of the written table (`image_intensity_outlier`, `repeat_intensity_shift`, `saturated_image`, `no_intensity_spread`, §14), reported apart from the run's issues; files that are not images; empty folders; images excluded by the user, or excluded paths that match no image; non-finite pixels; different image widths or heights; different numbers of depth slices, time points or other planes; physical pixel sizes that differ or are known for only some images; a stated pixel size that differs from the header (`calibration_differs_from_header`, §9.3; the stated size is used); unbalanced numbers of images; biological repeats missing from some conditions (§11); planes without finite pixels while others remain; channels whose central intensity range covers too few histogram bins (`narrow_histogram_range`, §4); planes over the memory limit that are read in blocks; images without a mask of a type other images have, masks matching no image, masks of excluded images, files outside the mask-type folders (§15) | Processing continues; the item is reported |
 
 Every message states what happened in plain language, lists the affected paths, and
-suggests an action. Technical details go to the diagnostic log.
+suggests an action. Technical details go to the diagnostic log. The command line's log
+(`--log FILE`, issue #99) is checked before it is opened, and opened only after the other
+arguments: it is refused inside the experiment or masks folder (given, or recorded by the
+earlier run in the output folder), inside `convert`'s folder of copies, as the design file
+given with `--design` (an input of the run), as a folder or in a
+folder that does not exist, and as or inside an output artefact (`plane_histograms.csv`,
+`run_manifest.json`, the report files, `experiment_design.csv`, `calibration.csv`,
+`mask_relationships.json`, the lock, `.morphobricks-conversion.json`, the `.run-` staging
+folders; `log_target.PROTECTED_OUTPUTS`). Paths are compared resolved and, for the folders that exist, by
+device and inode (another spelling of a folder on a file system that ignores case); output
+names ignore case; an existing file with a second name (a hard link) is refused. Any other
+name in the output folder is allowed.
 
 In notebook step 1, message details start collapsed with orange warning and red error
 counts visible outside the dropdown. Collapsing details does not change severity or

docs/plan.md

@@ -557,6 +557,10 @@ API, CLI). The notebook and CLI only call the API.
   header naming a column twice or a column without a name, and `RunInputs.design()` reads
   with `read_confirmed_design`; changing the notebook's output folder reloads the design
   from the new folder and asks again.
+- **Before the audit: links and the log (issues #81 and #99)** *(done; questions 64 and
+  65)*: `links.LinkGuard` follows a link only inside the folder walked (discovery, OME-Zarr
+  stores, the masks folder; `RunInputs` and `convert` refuse the same links), and the CLI
+  checks `--log` (`log_target`) before opening it, after the other arguments.
 - **Release v0.4.0 (PP-13) checklist**, for every tag of v0.4.0 including release
   candidates, in the release PR before the tag is pushed:
   - set `version` in `pyproject.toml`;
@@ -1589,3 +1593,30 @@ Raised during review; to be decided before the milestones that need them:
     answer (the folder may still be being typed), and the design card says which. The run
     is gated as before, so it runs only on the new folder's confirmed design. No copy
     action: the user answers again, or copies the file. Contract §12; user guide §4.
+64. Are symbolic links in the experiment or masks folder followed (issue #81)? *Decided
+    (owner, 2 November 2026): contain.* A link (file or folder; on Windows also a directory
+    junction) is followed only when its real path stays inside the experiment folder, or the
+    masks folder for masks, both resolved; a link that leaves it is refused
+    (`link_outside_folder`, naming each link and its target, a broken link too). One file
+    reached by two paths is refused, naming both (`linked_twice`); hard links too (the same
+    device and inode, from the one `stat` taken per image or mask; owner, 2 November 2026,
+    in review). A link back to a folder on its own path is `link_loop`. A broken link that
+    stays inside is reported by its reader, never skipped (review: masks skipped it). The
+    links inside an OME-Zarr store are held to the same rule (reading and copying a store
+    follow them); the cost, about 84 ms per 20,000 store files on each discovery, is
+    accepted (owner, 2 November 2026). One helper (`links.LinkGuard`) walks the experiment
+    and the masks folder for discovery, the masks pairing, the unrecorded-files check of
+    `RunInputs` (raised as `RunInputsError`) and, through discovery, `convert`. Contract §1,
+    §8.
+65. Where may the command line's `--log` write (issue #99)?
+    *Decided (owner, 2 November 2026):* the log path is checked first and the file opened
+    only after the other arguments are checked. It is refused inside the experiment or masks
+    folder (given or recorded by the earlier run), inside `convert`'s folder of copies, and
+    as or inside a protected output artefact (`plane_histograms.csv`, `run_manifest.json`,
+    the reports, the design, `calibration.csv`, `mask_relationships.json`, the lock, the
+    conversion record, the staging folders). Any other name in the output folder is fine.
+    In review: folders are also compared by device and inode and output names ignoring
+    case (macOS and Windows), and an existing file with a second name (a hard link) is
+    refused. Also in review: the design file given with `--design` (an input) is refused
+    (it was appended to). Contract §8; user guide §5.
+