Show phase-aware progress and ETA while checking folders and image headers #168

Problem

In the notebook, Check folder currently shows only:

Checking the folders and image headers…

For large experiments this can remain unchanged for a long time, so the user cannot tell whether the scan is progressing, what part of the check is currently running, or roughly how long it will take.

The current code can support a useful progress estimate without changing scientific behavior:

• discover_experiment() must finish before the number of images is known;
• immediately afterwards preflight() has the concrete list of analysed images;
• header inspection is currently a simple serial loop through those paths (inspect_images(), and the equivalent bound-header loop in _inspected(..., bind=True));
• the rest of preflight() already has distinct stages: calibration, dataset checks, masks and conversion-record checks;
• notebook report writing happens after preflight(), so it can expose its own final phase.

Suggested simple design

Do not add tqdm as a runtime dependency. The notebook already uses ipywidgets progress controls, and the core already uses progress callbacks for long-running operations.

1. Show the current phase

Use a compact phase/status line throughout Check folder, even when a percentage is not meaningful.

Suggested phases:

1. Finding images…

   • folder walk / experiment discovery
   • indeterminate: total is not known yet

2. Reading image headers…

   • determinate once discovery has returned
   • show image N / total, elapsed time, ETA and current file

3. Checking calibration…

   • usually short; status only

4. Checking dataset consistency…

   • status only

5. Checking masks…

   • if masks or OME-Zarr label images are present
   • if a reliable mask total is cheaply available, show N / total; otherwise status only

6. Checking conversion information…

   • only when relevant

7. Writing check report…

   • notebook-side final step

8. Done

   • replace the working UI with the normal check results

Do not invent percentages for phases that do not have a meaningful denominator. The main goal is simply that the user can always see what the program is doing.

2. Header progress + ETA

Add an optional progress callback to the header-inspection path.

A simple general callback could carry both phase and optional progress, for example:

progress(phase: str, done: int | None, total: int | None, detail: str = "") -> None

or an equivalent small typed event.

Keep the default None, so API/CLI behavior is unchanged.

For header inspection:

• emit the header phase with done=0, total=len(images);
• update after each image header;
• detail is the current/just-completed relative path.

Both paths must use the same semantics:

• normal inspect_images();
• bind_headers=True in _inspected().

3. Notebook display

Use the existing notebook widget style.

During header reading, show something compact such as:

Reading image headers
137 / 842
Elapsed 18 s · about 1 min 33 s remaining
drug/rep2/FOV_0137.ome.tif

with an IntProgress underneath or alongside it.

Estimate remaining time from time.monotonic() and completed headers. A simple running mean is enough. Wait for a few completed images before showing an ETA so one unusually slow/fast file does not immediately create a misleading estimate.

The ETA is explicitly approximate because TIFF and OME-Zarr header costs may differ.

When the header phase reaches 100%, immediately switch the label to the next real phase rather than leaving a full bar labelled as though the whole check is finished.

4. Avoid fake work

Do not:

• pre-scan the experiment just to obtain a folder/image denominator;
• add tqdm only for notebook presentation;
• add percentage estimates for calibration/dataset/report phases when there is no useful total;
• change the order or scientific meaning of the existing checks.

The phase UI should report the actual existing pipeline, not create a second workflow.

Why this is feasible

At current main, metadata.inspect_images() is:

files = tuple(inspect_image(root / rel, rel) for rel in sorted(relative_paths))

and run.preflight._inspected() has the same per-image loop when bind_headers=True.

preflight() then proceeds through explicit calls for:

• calibration;
• dataset consistency;
• masks;
• conversion review.

So the existing execution already has natural phase boundaries.

Folder discovery itself cannot provide a meaningful percentage without first doing another traversal, which would be wasteful. An indeterminate discovery phase followed by determinate header progress is therefore the simplest honest design.

Acceptance

• The notebook always shows which phase Check folder is currently executing.
• Folder discovery shows Finding images… rather than a fake percentage.
• Header inspection shows current image / total.
• Header inspection shows elapsed time.
• Header inspection shows approximate remaining time after enough images have completed.
• The current file/path is shown during header inspection.
• After headers complete, the UI moves through meaningful remaining phases such as calibration, dataset consistency, masks and report writing.
• Phases that have no useful denominator show status only, not a fabricated percentage.
• The notebook remains responsive enough to see updates in Colab and JupyterLab.
• No extra tqdm dependency is added.
• CLI/API behavior is unchanged when no callback is supplied.
• Normal preflight and bind_headers=True use the same header-progress semantics.
• Existing preflight results are semantically identical with and without progress reporting.
• Progress/status widgets reset cleanly on success and on failure.

Possible follow-up

If mask-header discovery is itself a large measurable bottleneck, add a determinate Checking masks N / total sub-progress once the mask inventory can expose that count cheaply. Otherwise keep masks as a named phase only.