Skip to content

[Sad Catapella N5][Scientific integrity] CSV parser accepts NaN, infinity and impossible nonempty tail moments #46

Closed

Description

@guijacquemet

Baseline: f784c95. Confirmed missing validation, no repo tests executed. Distinct from #33 (overflow during legitimate moment calculation).

Code: csv_io._tail checks only negative count and zero-count nonzero moments; with count > 0 it accepts nonfinite sum, nonfinite or negative sum_of_squares, or moments with physically impossible variance. HistogramTail constructor also has no checks. These moments enter pooled mean and standard deviation. Existing corrupted-tail tests test count consistency and empty-tail sums, not nonempty NaN/inf/negative moments.

Deterministic fixture: Write a valid robust-range row with underflow count >0; edit only Histogram_UnderflowSum_C0 to nan or Histogram_UnderflowSumOfSquares_C0 to -1/ inf. Count/range checks remain unchanged, and invalid statistics reach downstream computations.

Fix: For nonempty tails validate finite sum/second moment, nonnegative second moment, and the feasible relationship n * sum_of_squares >= sum**2 with justified numerical tolerance. Negative sum may be valid for signed intensities. Define how previously serialized overflowed moments are handled in coordination with #33, never silently replace with zero. Validate direct HistogramTail construction too. Add robust tests for signed negatives, normal populated tails, NaN/±inf, negative squares, impossible finite moments and high dynamic range; report row/channel/field. No Numba priority.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions