Repository navigation
fix(proxy): set per-model pricing so LiteLLM budgets/spend are accurate - #10
Merged
Merged
Conversation
added 2 commits
June 23, 2026 13:11
The blockrun model ids (anthropic/claude-opus-4.8, openai/gpt-5.5, ...) are not in LiteLLM's built-in model_cost map, so the proxy priced these calls at $0 — a team/key max_budget (e.g. $200) never tripped and the spend dashboard under-reported. This is why a customer capped at $200 kept running while the wallet drained. Add each model's official list price (= the gateway's 1:1 price) as input_cost_per_token / output_cost_per_token (+ cache_read/creation costs for Claude) in the example config, converted $X/1M -> $/token. The sidecar forwards the gateway's real upstream token counts (incl. prompt-cache), so spend = list price x actual tokens, and budgets enforce true consumption. Documents the requirement + that the live deployment config must mirror it, and that the unpriced blockrun/* wildcard contributes $0 to budgets.
Independent review found two over-claims in the prior comment: (1) cache_read/creation_input_token_cost in litellm_params are honored only on newer LiteLLM versions — on 1.70.0 they are silently dropped and cache tokens price at the full input rate (a conservative over-count, budget trips earlier not later); (2) streaming spend is exact only when the gateway emits a usage SSE frame, else LiteLLM uses tiktoken estimates. Document both and point to the gateway reconciliation report as the exact cost source of truth. input/output per-token pricing (the part that makes the $200 cap trip on non-streaming) is verified honored.
VickyXAI
pushed a commit
that referenced
this pull request
Jun 24, 2026
#11) LiteLLM computes per-call cost as token_count × list-price (from its model price map). That does NOT match BlockRun's real wallet deduction — the gateway price carries a per-call minimum floor + margin — so on small/short calls it is off by ~10x (customer: LiteLLM showed $0.00x, wallet was debited $0.0x, token counts matched). PR #10 priced the proxy at list price so budgets stop reading $0, but its own note flags that exact wallet-charge reconciliation still needs an x-blockrun-cost-usd path. This is that path for the custom-provider mode. - _adapter: pull the SDK's real per-call charge off the response (response.cost_usd, blockrun-llm with the cost attach; falls back to _last_call_cost on older SDKs) and thread it through as _blockrun meta. - provider: write it into _hidden_params["response_cost"] so LiteLLM's own spend tracking / budgets reflect the true charge, and expose blockrun_cost_usd / blockrun_settlement for callers that want the exact number + on-chain receipt alongside usage. - logger: record the real charge as cost_usd with cost_source="blockrun_x402", keep LiteLLM's estimate as estimated_cost_usd for side-by-side auditing, and log the settlement. Verified live against the gateway: 2 paid deepseek-v4-pro + 2 paid claude-sonnet-4.6 calls — response_cost == blockrun_cost_usd == logged cost_usd == the get_balance() delta, to the cent. Scope: non-streaming custom-provider (sync + async). Streaming and the proxy raw-passthrough path are not covered yet — follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
LiteLLM computes per-call spend (and enforces
max_budget) frominput_cost_per_token/output_cost_per_token. The blockrun model ids (anthropic/claude-opus-4.8,openai/gpt-5.5, …) are not in LiteLLM's built-inmodel_costmap, so the proxy priced these calls at $0 — a team/keymax_budget(e.g. $200) never tripped and the spend dashboard under-reported. This is the proxy-side half of the incident where a customer capped at $200 kept running while the wallet drained.What
examples/litellm_config.yaml: add each model's official list price (= the gateway's 1:1 price) asinput_cost_per_token/output_cost_per_token, pluscache_read/creation_input_token_costfor Claude, converted$X/1M → $/token:spend = list price × actual tokens→ budgets enforce true consumption.blockrun/*wildcard contributes $0 to budgets (add an explicit entry per model you want counted).This is the example/template. Apply the same
*_cost_per_tokenvalues to your live LiteLLM Proxy config — that's the file that governs the real $200 cap. Without them the cap stays meaningless.Notes
$/1M → $/tokenconversions checked.x-blockrun-cost-usdheader + cost callback could make the proxy bill the exact settled amount, but the per-model pricing here already makes budgets accurate from real token counts.