Skip to content

Narrow float64 histogram ranges mis-bin pixels (halved range rounds to zero, n / half_range overflows) #84

Closed

Description

@guijacquemet

From an external static review (batch 18, N4), reproduced on main at 567c20b.

Where: histogram.py:665-680 (_float_bins), histogram.py:851-864 (_bin_index), bin_centres.py:16.

  • half_range = hi*0.5 - lo*0.5 can round to zero for distinct subnormal endpoints. float_bin_centres then collapses distinct bins.
  • n / half_range can overflow to inf for a very narrow non-zero range, before the integer bin cast.
  • Either way, finite pixels can be mis-binned and pooled percentiles corrupted.

Fix: compute a bounded ratio first and multiply by n after. When hi - lo is finite, use (value - lo) / (hi - lo). Keep the halved form only for the large-range case where hi - lo overflows. Apply the same rule to bin centres and _bin_index. The large-range fix of #76 (±float64 max) must still pass.

Reproduction on main:

  • float64 plane [0, 5e-324, ...], 4 bins: ZeroDivisionError (range check path / _bin_index).
  • plane [0, 1e-320, 0, 1e-320], 4 bins: counts [4, 0, 0, 0] (the 1e-320 pixels land in bin 0) plus RuntimeWarning: invalid value encountered in cast.
  • plane [0, 5e-307, 1e-306], 1024 bins: n / half_range is inf, so the position is nan or inf before the cast.

Regressions wanted: the smallest-subnormal range and a narrow normal range; existing histograms must stay bit-identical.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions