[Performance] Add simple resource-aware FOV-level parallel preprocessing #107

Goal

Speed up preprocessing by processing independent source images/FOVs concurrently, while keeping the scientific output and current integrity guarantees unchanged.

Current main (620c41604925da7e143b860eabd7ae69760a7225) is serial at the FOV/file level: compute_rows() builds to_read, then new_rows() reads each file one after another through _read_one() / _read_file().

Keep the first implementation deliberately simple and general.

Suggested V1

1. Parallelize one FOV / source file per worker

Use a ThreadPoolExecutor.

A worker processes one complete source file, including all of its Z/T planes and channels. Do not parallelize individual planes or channels.

The worker should only:

1. read the source through the existing _read_file();
2. perform the existing post-read fingerprint/content check;
3. return a typed per-file result.

The parent remains responsible for staging, aggregation, QC and publication.

This keeps the image-reading and scientific code unchanged and confines threading to orchestration.

2. Add a small public worker option

Support something like:

run_experiment(..., workers=1)
run_experiment(..., workers=4)
run_experiment(..., workers="auto", max_ram=...)

and equivalent CLI options.

Keep workers=1 as the compatibility path.

3. Simple automatic worker calculation

Do not build a complex dynamic scheduler in V1.

For the FOVs that actually need computing:

1. get the machine CPU count;
2. use the existing plane_layout() memory estimates to estimate the peak working memory of each FOV;
3. take the largest FOV estimate;
4. choose:

cpu_cap = min(number_of_jobs, available_cpu_count)

if max_ram is supplied:
    ram_cap = floor(max_ram / largest_fov_estimate)
    workers = min(cpu_cap, ram_cap)
else:
    workers = cpu_cap

Apply a small documented parent-process reserve before calculating ram_cap if needed.

This is intentionally conservative. It may leave some RAM unused when FOV sizes vary, but it is predictable, safe and easy to maintain. A smarter memory-aware scheduler can be added later only if benchmarks show it is worthwhile.

If one FOV cannot fit within the explicit RAM budget even using the existing block-reading path, refuse with a clear error rather than silently exceeding the limit.

Use working-memory estimates from plane_layout(), not file size on disk.

4. Parent owns all mutable run state

Workers must not modify shared run state.

The parent alone should:

• load/reject/resume existing staged rows;
• submit the remaining FOV jobs;
• receive worker results;
• call stage_file();
• aggregate records, narrow, read, failed and changed;
• run dataset-wide checks;
• perform final source/content verification;
• publish the final generation.

The progress callback should also run only in the parent thread.

5. Deterministic output

Worker completion order must not affect scientific output.

Before aggregation/publication, retain the existing canonical ordering by source path / row identity.

The same stable input should produce the same scientific output with:

workers=1
workers=2
workers=8
workers="auto"

Concurrency settings are operational metadata, not part of scientific identity.

6. Preserve current failure semantics

Keep the existing rules:

• an unreadable FOV is reported as unreadable;
• a FOV changed during reading is never staged;
• any required-file failure prevents final publication;
• unexpected worker exceptions propagate and abort the run;
• no silent serial fallback;
• completed valid staged FOVs remain reusable after interruption;
• final source/content verification remains immediately before publication.

Acceptance tests

At minimum:

• workers=1 preserves current behaviour.
• 1 and N workers produce identical histogram CSV bytes for stable inputs.
• 1 and N workers produce equivalent scientific manifest contents.
• Forced out-of-order worker completion does not change output ordering.
• Existing staged FOVs are not recomputed.
• One unreadable FOV gives the same run failure as serial execution.
• A FOV changed during its read is not staged or published.
• Worker exception / interruption publishes nothing and leaves completed staging reusable.
• TIFF and OME-Zarr paths both have serial/parallel parity.
• Whole-plane and block-reading paths both have serial/parallel parity.
• Automatic worker count never exceeds the CPU cap or explicit RAM budget.

Non-goals for V1

Do not add:

• a dynamic bin-packing scheduler;
• process/thread backend selection;
• nested plane/channel parallelism;
• parallel writes to the final CSV;
• a shared cross-brick worker framework;
• automatic benchmarking/tuning of worker count.

Those can be considered later if the simple implementation is measurably insufficient.

The main design rule is: one FOV per worker, conservative resource cap, parent-only state mutation, identical scientific output.