Skip to content

Research: shared memory for big images (issue #66), benchmarks and findings - #123

Closed
guijacquemet wants to merge 1 commit into
mainfrom
claude/perf-shared-memory
Closed

guijacquemet wants to merge 1 commit into
mainfrom
claude/perf-shared-memory

Conversation

@guijacquemet

Copy link
Copy Markdown
Contributor

Research only, do not merge as a feature. Adds docs/PERF_SHARED_MEMORY.md and benchmarks/ (scripts and raw results). No change to labconstrictor_tools; pyproject.toml only gets a ruff per-file-ignore for benchmarks/.

Headline:

  • Transport overhead is 2 to 5 ms; pixels (fresh memory, TIFF write/read) are the cost.
  • Against a warm page cache, shared memory is not faster than the TIFF path on a Python host (512 MiB: prewarmed medians 0.64 s disk vs 0.75 s shm for input only).
  • The plan's unexplained 2.4 s and 15 s are reproduced and explained: first touch of fresh memory on the micro-VM (1 to 3 s per 512 MiB instead of 0.1 s) across three allocating steps, not the shm conversion (0.2 s).
  • Fiji (ImageJ FileSaver/Opener vs the Appose shm class it already ships) gains about 0.5 s out and 0.4 s back per 512 MiB; real Java to worker proof passes.
  • A 1 GiB shared round trip was OOM-killed in the 16 GB sandbox where the TIFF path finished.

Recommendation: do it for Fiji only, opt-in with a size threshold, after a small Tools change (validate descriptor order, feature shm_input_v1); not for Napari or QuPath; no shared-memory outputs yet. Try tifffile.memmap in the worker first (Tools-only, no protocol work).

Unverified: Windows and macOS behaviour, real Fiji/QuPath/Napari runs, slow disks and antivirus. See section 10 of the doc.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN


Generated by Claude Code

…ndings

docs/PERF_SHARED_MEMORY.md: measured end-to-end table (128/512/1024 MiB, TIFF path versus
shared memory, median and spread), cause of the unexplained 2.4 s and 15 s, per-host feasibility
(Napari, Fiji, QuPath), protocol and cleanup implications, recommendation. benchmarks/: the
scripts and raw results (Python host model, Java proofs with the Appose shm classes, JDK-only
mmap and Java 21 FFM). Nothing in labconstrictor_tools changes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants