You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Sad Catapella B3][Performance] Histogram run materializes every plane's bins and rewrites the entire CSV #77
Baseline:d7a538e56d41ac1093ba79adcad397c21b3dfd32. Classification: confirmed architecture/space complexity; specific speed and RSS impact unmeasured.
Evidence:_read_file() builds all PlaneRecord objects for a file; run_experiment() lines 582–645 retains all new and unchanged records and range checks in process-wide Python lists; lines 644–690 sorts all records, calculates intensity checks and rewrites the full table atomically. _load_previous() lines 914–952 reads the entire existing CSV into PlaneRecord objects even for no-change reruns. memory_limit is documented as per plane, not total job; this is a scalability opportunity, not a violation of the documented limit.
Scale illustration (not a measurement): 10,000 rows × 65,536 bins × 8-byte int64 counts = ~5.24 GB for one channel's count arrays alone (before records, range metadata, CSV text, channels and old/new structures). This can undermine successful blockwise image processing for large datasets.
Fix: design bounded-memory record aggregation and incremental staged CSV output, or an indexed columnar intermediate that preserves the current public CSV and atomic rollback semantics. Stream per-file rows, avoid accumulating both old and new representations, and make intensity summaries aggregate incrementally only where scientifically equivalent. Confirm row checksums/source identity and deterministic row ordering. Benchmark N planes, N channels, bins/row and repeated no-change runs, reporting RSS and wall time; no Numba benefit for orchestration.
Activity