Skip to content

[Sad Catapella N2][Numerical correctness] Linear interpolation percentile overflows between finite extreme values #32

Closed

Description

@guijacquemet

Baseline and classification

Normalization Batch 2, commit f784c95e3049483f87466150ffc5764375fa631d. Confirmed formula overflow via isolated NumPy arithmetic probe, not a repository test.

Evidence

Minimal counterexample

An exact conceptual distribution containing only [-1e308, +1e308] has NumPy type-7 50th percentile 0. Inside _linear_percentile, above - below overflows to infinity, so the interpolation gives a nonfinite result. Independent NumPy arithmetic with those endpoints reproduced inf.

Impact / scope

Incorrect linear percentiles for fully finite data, and potentially an invalid normalization scale. Normal scientific RGB uint8/16 input cannot reach this numeric range, but linear_percentile is a public API that accepts float64 pooled data; a robust implementation must not silently return nonfinite results from finite, mathematically representable interpolations.

Fix and acceptance

Compute an overflow-safe convex combination, chosen to preserve bit-identical outputs for the existing normal-range CSBDeep/StarDist parity test (or document a scientific precision policy if strict parity cannot be preserved at extremes). Reject truly nonrepresentable results explicitly. Test ±1e308, mixed signs, subnormal endpoints, adjacent floats, q=0/50/100, arbitrary positive weights, and cases with widely different counts; compare to a high-precision reference. Do not introduce Numba before numerical semantics and parity are proven.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions