Skip to content

rate/increase functions do no counter-reset correction #476

Description

@zaoxing

Root cause: IncreaseAccumulator keeps only starting + last_seen and query(Increase) returns last − first (no reset detection), so any window spanning a counter restart is undercounted — often negative.

What causes a reset
A Prometheus/VM counter is in-process and goes back to 0 whenever the target restarts: deploys/rollouts, crashes/OOM-kills, autoscaling, node drains/preemptions, pod reschedules. (Counter overflow is also a "reset" but is rare with float64.) So a "reset within the window" simply means "the exporter restarted during [d]."

Testing
Drop this into the test module of asap-query-engine/src/precompute_operators/increase_accumulator.rs (it isolates the bug to the accumulator, which is exactly what MultipleIncrease stores per key):

#[test]

fn counter_reset_increase_is_wrong() {
    use crate::data_model::Measurement;
    use promql_utilities::query_logics::enums::Statistic;

    // One counter, sampled 100 -> 150 -> (restart) 10 -> 60 within the window.
    let mut acc = IncreaseAccumulator::new(Measurement::new(100.0), 0,
                                           Measurement::new(100.0), 0);
    acc.update(Measurement::new(150.0), 1_000);
    acc.update(Measurement::new(10.0),  2_000); // reset: 150 -> 10
    acc.update(Measurement::new(60.0),  3_000);

    let inc = acc.query(Statistic::Increase, None).unwrap();
    // BUG: last - first = 60 - 100 = -40  (negative!)
    // VictoriaMetrics/Prometheus reset-correct this to +110.
    assert_eq!(inc, -40.0);
}

#Run:
cargo test -p query_engine_rust --lib counter_reset_increase_is_wrong -- --nocapture

Expected Result
It passes asserting -40.0, which is the bug: rate shows the same (negative value_diff/time_diff). Prometheus/VictoriaMetrics' reset-corrected value here is +110 (its removeCounterResetsMaybeNaNs rebuilds the monotonic series [100,150,160,210] → 210−100).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions