Repository navigation
fix(web): usage page says when model rates are unavailable - #5947
wezzcoetzee wants to merge 2 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| // `unavailable` is worth a line: stale rates move cost by rounding, not by | ||
| // orders of magnitude. `unpricedShare` carries the magnitude either way, so | ||
| // the message never has to guess how much cost went missing. | ||
| const pricingNotice = |
There was a problem hiding this comment.
🟡 Medium usage/UsagePage.tsx:441
When multiple environments contribute records, the pricingStatus === "unavailable" branch falsely reports that the rate table failed to load as the cause of unpriced responses. pricingStatus is a merged environment-level worst status while unpricedShare is aggregated independently, so one environment can be unavailable (contributing only priced records) while a fresh environment contributes unpriced records from an unknown model. In that case the message claims the rate table could not be loaded, but the excluded records actually came from the loaded table lacking a model rate. Consider preserving per-record provenance for unpriced responses, or using a non-causal generic message that does not attribute the exclusion to a rate-table failure for mixed environments.
🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/web/src/components/usage/UsagePage.tsx around line 441:
When multiple environments contribute records, the `pricingStatus === "unavailable"` branch falsely reports that the rate table failed to load as the cause of unpriced responses. `pricingStatus` is a merged environment-level worst status while `unpricedShare` is aggregated independently, so one environment can be `unavailable` (contributing only priced records) while a `fresh` environment contributes unpriced records from an unknown model. In that case the message claims the rate table could not be loaded, but the excluded records actually came from the loaded table lacking a model rate. Consider preserving per-record provenance for unpriced responses, or using a non-causal generic message that does not attribute the exclusion to a rate-table failure for mixed environments.
There was a problem hiding this comment.
Good catch — fixed in c989e51.
You are right that the two values are aggregated independently: pricingStatus is the weakest status across contributing environments, while unpricedShare sums unpriced records regardless of which environment produced them. Across several devices, the environment whose rate table failed need not be the one that contributed the unpriced records, so asserting the failure caused the exclusion is wrong.
I kept both facts but dropped the causal link rather than adding per-record provenance:
Model rates could not be loaded. 100.0% of responses are excluded from cost. Token counts are unaffected.
Each sentence now stands on its own and is true in the mixed case. Per-record provenance would be the more precise fix, but it means a contract change to carry the reason per bucket, which is more than this PR should take on.
There was a problem hiding this comment.
Sorry, I'm unable to act on this request because you do not have permissions within this repository.
The usage page renders $0.00 across every model when the LiteLLM rate table fails to load, indistinguishable from having spent nothing. The server already reports why - UsageSummary.pricing.status and per-bucket unpricedRecords - but no client reads either field, so a total pricing failure looked like a real answer. mergeUsage now derives pricingStatus from the environments that contributed buckets, and UsageCoverageNotice states when rates could not be loaded and what share of responses that excluded from cost. The share comes from unpricedShare rather than an assumption, so the message is also correct for the partial case where a single new model is missing from the table. No contract change. cached is intentionally silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
5c7fdce to
c989e51
Compare
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. You can customize Macroscope's approvability policy. Learn more. |
|
Fixed in #7181 |
Closes #5948
What Changed
mergeUsagenow derives apricingStatusfrom the environments that actually contributed buckets, andUsageCoverageNoticerenders one line when the model rate table could not be loaded:No contract change, no new UI surface, no change to any cost calculation. Two tests added in
usageMerge.test.ts.Why
When the LiteLLM rate table fails to fetch,
UsageService.ensureRatesswallows the failure and every bucket falls throughlookupRateto{ costUsd: 0, costSource: "unpriced" }. The usage page then renders a confident$0.00for every model,0.0% of costfor every provider, and$0.00cache savings — beside perfectly correct token counts. It is indistinguishable from having spent nothing.Degrading instead of failing the page is the right call, and this PR does not change it. The bug is that it degrades invisibly.
The server already ships the diagnosis:
UsageSummary.pricing.statuscarriesfresh | cached | unavailable, with a doc comment saying it exists "so the UI can be honest about how good the cost figures are", and every bucket carriesunpricedRecords. Both cross the wire on every request and no client reads either —grep -rn "pricing" apps/web/src apps/mobile/srcreturns nothing. This plumbs the existing signal to the existing notice strip.Two deliberate details:
unpricedShare, not an assumption. Codex buckets can carryproviderReportedcost even with no rate table, so "everything reads $0.00" would be an overclaim. This also makes the message correct for the partial case: a genuinely new model missing from LiteLLM's table silently undercounts cost today, and gets the same message with a real percentage.cachedgets no line. Stale rates move cost by rounding, not orders of magnitude; warning about it would be noise.Reproduced on a machine behind TLS inspection, where the fetch to
raw.githubusercontent.comfails. Diagnosing it required readingserver.trace.ndjsonand noticingusage-model-rates.jsonwas absent from the state dir. That should not be the discovery path.UI Changes
Before and after, same data, same viewport, same commit base — the only difference is the notice line.
Follow-up: #5949 tracks the underlying fragility — the rate table has no bundled fallback, so a fresh offline install has no prices at all. Deliberately not addressed here.
Checklist
Note
Show pricing unavailability notice on the usage page when model rates cannot be loaded
pricingStatusfield toMergedUsagein usageMerge.ts, computed as the weakest rate-table status (unavailable<cached<fresh) across environments that contributed usage buckets; idle environments are excluded.UsageCoverageNoticein UsagePage.tsx to display an explanatory message when some responses are unpriced, distinguishing between rates being unavailable (failed to load) and models having no published rate.Macroscope summarized 38d1f68.
Note
Low Risk
UI-only messaging plus merge metadata derived from existing pricing fields; cost math and contracts are unchanged.
Overview
When model rates fail to load or some responses have no published price, the usage page can show $0 cost while token counts look correct. This PR surfaces that gap in the existing coverage notice strip—no new API fields and no change to how cost is calculated.
mergeUsagenow exposespricingStatuson merged totals: the weakestUsageSummary.pricing.statusamong environments that actually contributed buckets (unavailable<cached<fresh). Idle environments with failed rate tables do not drag down the merged status.UsageCoverageNoticeshows a line when there are records andunpricedShare> 0: either that rates could not be loaded, or that a percentage of responses lack a published rate (with token counts called out as unaffected where relevant).cachedpricing does not get its own warning.Reviewed by Cursor Bugbot for commit 38d1f68. Bugbot is set up for automated code reviews on this repo. Configure here.