Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,9 @@ src/morphobricks_preprocessing/
conversion_issues.py its messages (pure)
conversion_plan.py plan_conversion: headers checked with the confirmed roles, each bound to
its image's fingerprint (source_changed_during_check), planes checked
against the memory limit first, PlannedCopy per image
against the memory limit first, PlannedCopy per image;
copy_folder_issues (the folder rule) and preview_conversion (the plan,
space needed and free, for the notebook card; writes nothing, PP-11b)
conversion_reuse.py compare: unchanged copies reused, anything else there refused
conversion_write.py write_all: .partial files moved into place together (any failure
removes what it wrote)
Expand All @@ -179,6 +181,9 @@ src/morphobricks_preprocessing/
notebook_masks.py step 1's masks folder, Masks card and mask-axis statements (PP-10c)
notebook_mask_step.py step 3's Describe the masks card: the masks step after a run that
recorded masks (PP-10d)
notebook_convert.py step 2's Make named-axis copies card under the axis check (PP-11b):
preview_conversion, then convert_experiment under the copy folder's lock,
then Check the copies in step 1 (same output folder)
notebook_ui.py PreprocessingNotebook: the four notebook steps (needs the `notebook` extra);
folder_step/axis_step take title, purpose, intro and show_next (and
show_output and show_design for step 1) so other bricks can embed them
Expand Down Expand Up @@ -917,7 +922,11 @@ jupyter lab notebooks/MorphoBricks_Preprocessing.ipynb
the CLI calls too; see plan §10.
- **Named-axis copies (PP-11):** `convert` / `convert_experiment` writes OME-TIFF copies of
the images with axis errors or unnamed axes with the confirmed roles into a new folder
(contract §17, plan §10 and question 59); the notebook card is PP-11b.
(contract §17, plan §10 and question 59).
- **Notebook conversion card (PP-11b):** step 2's **Make named-axis copies (optional)** card,
under the axis check when it reports these images: the folder for the copies, the
confirmed roles, the plan (`preview_conversion`), the copies (`convert_experiment`) with a
progress bar, the lock card, and **Check the copies in step 1**; see plan §10.
- **Next (not started):** the notebook step for the masks (name each mask, confirm or
correct the relationships) and a file saving those answers for the other bricks
(question 32); 3D and time-lapse masks later.
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,8 @@ gets the permissions a file you save yourself would (your umask). `convert` save
images whose metadata name their axes wrongly or not at all as named-axis OME-TIFF copies
with your roles in a new folder (each plane bit for bit the plane read with those roles;
every other image copied as it is; a conversion record written last), then you run
preprocessing on that folder with the same output folder. Details in the
preprocessing on that folder with the same output folder. The notebook offers the same
conversion in step 2, under the axis check, when it finds such images. Details in the
[user guide](docs/user_guide.md) §5.

## Example
Expand Down
2 changes: 1 addition & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -522,4 +522,4 @@ exclude=, missing_as_singleton=, memory_limit=, progress=)` (`conversion`), the
| Repeat | A copy already in the tree is reused when its source (fingerprint and axes), roles, copy axes, pixel sizes, unknown sizes (roles and reasons), channel names, significant bits, the copy's fingerprint and the copy's SHA-256 (read whole) are as recorded; a new image is added; when every copy is reused the record is kept as it was (same SHA-256). Anything else (a source changed, other roles, a copy changed, an image no longer in the experiment, another experiment's copy tree, a file at a new image's place that the record does not list) is `copy_differs`: nothing is overwritten. A conversion record that cannot be read, or of version 1, is `conversion_record_unreadable`. |
| Conversion record | `.morphobricks-conversion.json` in the copy tree's root (a dot-file, so discovery skips it), record version 2: `source_root`, `mapping`, `missing_as_singleton`, `written_at`, `versions`, and per image `relative_path`, `action` (`converted` or `copied`), `reason` (the axis code, `excluded`, or null), `source` and `copy` fingerprints (`size`, `mtime_ns`), `copy_sha256` (the copy's SHA-256, computed in one pass after it is written; an OME-Zarr copy's is the SHA-256 of its files' relative paths and SHA-256s in path order, its `labels` group left out as for fingerprints: `content_hash`), `source_axes`, `roles`, `copy_axes`, `pixel_sizes` (by role, as §9), `unknown_sizes` (role: why), `channel_names` (as the copy holds them; `[]` for none) and `significant_bits` (or null). Sources keep the fingerprint rule (design v0.4 §1); the copies' SHA-256 is the owner's decision of 2 November 2026 (plan §12 question 59 (16)). Version 2 (PP-11c) added `copy_sha256`, `channel_names` and `significant_bits`; a version 1 record (v0.4.0rc3 or earlier) is refused everywhere it is read (`convert`, the run's `review_conversion`, `RunInputs`) with a message to convert the originals again into a new folder: its copies cannot be checked. `read_conversion_record(folder)` reads it with its SHA-256. |
| The run on the copies | The user runs preprocessing on the copy tree with the same output folder (`--output`): the recorded experiment root becomes the copy tree and the report says so (`experiment_root_changed`, and `conversion` with the record, its SHA-256 and `source_root`; the text says "named-axis copies of ..."); every row is recomputed (the row key includes the read path, §9); the design, image type and calibration are kept (same relative paths). The manifest records `conversion_record_sha256`. The run and `check` warn when a source changed or went missing after the conversion (`conversion_source_changed`) and when a copy changed (its fingerprint or its SHA-256: each copy is read whole, so `check` and `run` on a copy tree cost one more read of the copies), went missing or was added since (`copy_changed_after_conversion`; the run goes on, and its message says that `RunInputs` will refuse the run for a changed or added copy); an unreadable record stops the run (`conversion_record_unreadable`). `RunInputs` reads the copies like any images, and refuses a run whose conversion record changed or went missing since (`conversion_record_sha256`), is of version 1, or whose recorded images' copies no longer have the recorded fingerprint and SHA-256 (§9.1). |
| Notebook | Not yet: the axis-check card names the `convert` command; the notebook card is the next PR (PP-11b, plan §10). |
| Notebook (PP-11b) | Step 2's **Make named-axis copies (optional)** card, under the axis check, shown only when it reports images with `axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y` or `axes_not_named` for the roles being edited (design v0.4 §5: offered only for files with errors), and not with `folder_step(show_output=False)`. The roles are the user's: those confirmed in step 2, or the roles shown once the preview is checked and the card's own question is answered; any change of a role or of the missing-axes answer takes it back. A required folder for the copies (Browse…, or pasted) checked with the command's rule (`copy_folder_issues`); `preview_conversion` (`conversion_plan`, headers only, writes nothing) gives what will be converted and copied byte for byte (the command's `plan_conversion`), the space needed and the free space; then `convert_experiment` with a progress bar, under the copy tree's lock. Refusals use the existing cards (the lock card with **Remove the lock of a stopped run**, the rule of `--remove-stale-lock`). After the copies: the record's path, and **Check the copies in step 1**, which sets the experiment folder to the copy tree, keeps the output folder and checks again; the card says that the missing-axes answer must be given again and lists the copies whose relabelled sizes are unknown (`ConversionResult.unknown_sizes`), to state under **Pixel sizes**. |
41 changes: 30 additions & 11 deletions docs/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,20 +513,39 @@ API, CLI). The notebook and CLI only call the API.
`RunInputs.open` refuses a record changed or gone since the run (question 59 (14)).
- The axis messages (`axes_yx_swapped`, `axes_contradict_header`, `axes_x_before_y`,
`axes_not_named`) and the notebook's axis marks name the `convert` command.
- Not in this PR: the notebook card for the conversion (PP-11b).
- The notebook card for the conversion followed in PP-11b.
- Review (2 November 2026): the obsolete issues of `convert_tiffs`/`zarr_export` had
acceptance criteria kept here: a memory budget that fails fast (#76), strict settings
(#80), no 64-bit precision loss (refused, #75), the copy's fingerprint and scales
checked before reuse (#87, #88, #92), no mixing of metadata epochs (#96); the run reads
the copies themselves, so the table describes the persisted pixels (#79).
- **PP-11b (notebook card for the conversion)** *(next PR; owner decision of 2 November
2026, plan §12 question 59 (15))*: design v0.4 §5 "the notebook offers conversion only
for files with errors": a card offered after the axis check when images have an axis
error or unnamed axes, calling `convert_experiment` with the confirmed roles and a
folder the user names, showing the result and the next run on the copies.
- **PP-11c (conversion integrity)** *(this PR; issues #80, #87, #88, #92, #96; question 59
(16)-(18))*: `conversion.py` split into `conversion_plan`, `conversion_reuse` and
`conversion_write` (move only); the copies' SHA-256 in the conversion record (version 2,
- **PP-11b (notebook card for the conversion)** *(implemented, this PR; owner decision of
2 November 2026, plan §12 question 59 (15))*: design v0.4 §5 "the notebook offers
conversion only for files with errors" (contract §17 "Notebook"):
- Step 2's **Make named-axis copies (optional)** card (`notebook_convert`, ui), right
under the axis check, shown when it reports `axes_yx_swapped`, `axes_contradict_header`,
`axes_x_before_y` or `axes_not_named` for the roles being edited; hidden with
`folder_step(show_output=False)`; the pinned embedding API is unchanged.
- The roles: those confirmed with **Confirm axis roles**, or, while the axis errors keep
that button disabled, the roles shown once the preview is checked and the card's
**Yes, write the copies with these roles** is pressed; changing a role or the
missing-axes answer takes it back. Nothing is taken from the headers.
- A required folder for the copies (Browse…, the Colab **Copy path** card); the plan from
`preview_conversion` (in `conversion_plan`, beside the command's `plan_conversion`
(PP-11c), with the folder rule `copy_folder_issues` moved there from `conversion`,
plus the space needed and free; writes nothing): images converted (with their code) and
copied byte for byte, the space; errors keep the button disabled.
- **Make named-axis copies** calls `convert_experiment` with a progress bar; refusals on
the existing cards (the lock card, now naming the folder for the copies, with **Remove
the lock of a stopped run**).
- After the copies: the record's path and **Check the copies in step 1** (the experiment
folder becomes the copy tree, the output folder is kept and filled in, and the folder is
checked again); the card says the missing-axes answer must be given again and lists the
copies whose relabelled Y/X sizes are unknown (`ConversionResult.unknown_sizes`, new), to
state under **Pixel sizes**.
- **PP-11c (conversion integrity)** *(implemented, PR #126; issues #80, #87, #88, #92,
#96; question 59 (16)-(18))*: `conversion.py` split into `conversion_plan`,
`conversion_reuse` and `conversion_write` (move only); the copies' SHA-256 in the conversion record (version 2,
`content_hash`) verified at reuse, by `review_conversion` and by `RunInputs.open`
(`conversion_check`); channel names, significant bits and the unknown sizes' reasons
compared at reuse; each header bound to its image's fingerprint
Expand Down Expand Up @@ -1471,8 +1490,8 @@ Raised during review; to be decided before the milestones that need them:
(14) **`RunInputs` checks the conversion record**: a run whose manifest records
`conversion_record_sha256` is refused by `RunInputs.open` when the record changed or
disappeared, as for `calibration_sha256` and the design's SHA-256.
(15) **Notebook card**: the next PR, PP-11b (§10); the axis-check card names the
command meanwhile.
(15) **Notebook card**: PP-11b (§10), implemented: step 2's **Make named-axis copies
(optional)** card under the axis check.
(16) **The copies' SHA-256** *(owner decision of 2 November 2026, after an independent
review of 8db1f4a; implemented in PP-11c, issue #87)*: a fingerprint (size and time)
does not see a change that keeps both, so the record gives each copy's SHA-256,
Expand Down
36 changes: 34 additions & 2 deletions docs/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,8 @@ results or comparing another grouping.
dropdown; the card gives the error (`axes_yx_swapped` or `axes_contradict_header`,
§8) naming the images with their header axes, your roles and the likely cause. Such
images are never read: correct the axes, or save named-axis copies with your roles and
run preprocessing on the copies. An axis the metadata does not name (a plain stack of
run preprocessing on the copies (the **Make named-axis copies (optional)** card under
the axis check, §5 "Named-axis copies"). An axis the metadata does not name (a plain stack of
pages) takes the role you confirm: the card warns (`axes_not_named`) and the run
records the roles as confirmed by you.
- If the preview looks right, press **Yes, the preview looks right**, then click
Expand Down Expand Up @@ -753,7 +754,38 @@ or if a copy no longer has the SHA-256 the record gives.
images or made with other roles is refused (`copy_folder_not_empty`, `copy_differs`);
convert into a new folder then. If it fails (`conversion_failed`), the folder is left as it
was. Images of 64-bit integers cannot be saved as OME-TIFF (`cannot_convert_pixel_type`).
The notebook does not offer the conversion yet: run the command in a terminal.

**In the notebook.** When step 2's axis check reports these images, a **Make named-axis
copies (optional)** card appears under it, calling the same functions as the command:

1. **The roles.** They are yours: the roles confirmed with **Confirm axis roles** (possible
with only `axes_not_named`), or, while the axis errors keep that button disabled, the
roles shown once you pressed **Yes, the preview looks right** and then **Yes, write the
copies with these roles** on the card. Changing a role, or the answer about missing
axes, takes that answer back. Nothing is taken from the header.
2. **The folder for the copies** (required, no default): paste it or use **Browse…** (on
Google Colab, **Copy path** in the **Files** panel). It is checked with the command's rule
(`copy_folder_overlaps_experiment`, `copy_folder_not_a_folder`).
3. **What will happen**, from the command's planning, reading headers only: the images
written as OME-TIFF (with their axis code) and those copied as they are (with `excluded`),
the space needed and the free space. Any error that would stop the conversion is shown
here and keeps **Make named-axis copies** disabled, as does a drive without the space.
4. **Make named-axis copies** writes the copies with a progress bar, holding the folder's
lock. Refusals are shown on the card with what to do; a folder locked by a stopped
conversion shows its holder and **Remove the lock of a stopped run** (the rule of
`--remove-stale-lock`).
5. **After the copies:** the card shows the folder, what was written, copied and reused, the
path of the conversion record and what to do next. **Check the copies in step 1** sets
the experiment folder to the copies, keeps the output folder (an empty box is filled with
the originals' default) and checks again. Confirm the labels, the axis roles (now named in
the copies' metadata) and the image type again; the design and stated pixel sizes are
kept. If you pressed **Yes, they have a single plane there**, press it again (the copies
do not add those axes; `--missing-as-singleton` on the command line). The card lists the
copies whose relabelled Y or X has no pixel size (unless the header gave X and Y the same
size): state them under **Pixel sizes** in step 1.

The card is not shown when another brick embeds step 1 without the output folder
(`folder_step(show_output=False)`).

## 6. Output files

Expand Down
Loading
Loading