You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Sad Catapella N2][Numerical correctness] Linear interpolation percentile overflows between finite extreme values #32
Normalization Batch 2, commit f784c95e3049483f87466150ffc5764375fa631d. Confirmed formula overflow via isolated NumPy arithmetic probe, not a repository test.
Evidence
_linear_percentile, lines 376–396 forms difference = above - below and then interpolates, without guarding finite endpoint values whose subtraction exceeds float64 range.
Pooling tests do not cover extreme opposite-sign finite endpoints; weighted tests use ordinary uint16 intensities.
Minimal counterexample
An exact conceptual distribution containing only [-1e308, +1e308] has NumPy type-7 50th percentile 0. Inside _linear_percentile, above - below overflows to infinity, so the interpolation gives a nonfinite result. Independent NumPy arithmetic with those endpoints reproduced inf.
Impact / scope
Incorrect linear percentiles for fully finite data, and potentially an invalid normalization scale. Normal scientific RGB uint8/16 input cannot reach this numeric range, but linear_percentile is a public API that accepts float64 pooled data; a robust implementation must not silently return nonfinite results from finite, mathematically representable interpolations.
Fix and acceptance
Compute an overflow-safe convex combination, chosen to preserve bit-identical outputs for the existing normal-range CSBDeep/StarDist parity test (or document a scientific precision policy if strict parity cannot be preserved at extremes). Reject truly nonrepresentable results explicitly. Test ±1e308, mixed signs, subnormal endpoints, adjacent floats, q=0/50/100, arbitrary positive weights, and cases with widely different counts; compare to a high-precision reference. Do not introduce Numba before numerical semantics and parity are proven.
Activity