fix(gateway): recognise DeepSeek's cache-hit token spelling for non-streamed responses - #188
Merged
Merged
Conversation
…treamed responses 非流式响应解析器 `parse_usage` 的 openai_chat 分支原先只认 OpenAI 拼写 `prompt_tokens_details.cached_tokens`,不认 DeepSeek 原生顶层拼写 `prompt_cache_hit_tokens`,导致非流式 DeepSeek 响应把缓存命中部分按**未命中全价** 计费,且 `usage_records.cached_tokens` 记为 0。 流式侧早已修过同一问题:`UsageCapture::finish` 注释明写「cached 三拼写兼容」 (rant 2026-08-23T14:05:02),`sse.rs::extract_cache_read_tokens` 同样按 DeepSeek → OpenAI 优先级取;六处 `record_usage` 亦然。CHANGELOG 对 v0.7.10 / v0.7.11 两次修复的范围描述都是 passthrough/流式路径 ⇒ 非流式解析器属**漏修**,非有意豁免。 修复:`cached` 先取 `prompt_cache_hit_tokens`,回退 `prompt_tokens_details.cached_tokens`, 优先级与两条流式实现一致;`(prompt_tokens − cached).max(0.0)` 的 disjoint 语义不变。 changelog 里 2026-08-20T10:17:27 引入的归属说明同步补齐(它此前只列了两种拼写)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Non-streamed responses from an OpenAI-protocol upstream that reports cache hits with DeepSeek's native spelling (
prompt_cache_hit_tokens) are currently billed as if nothing was cached:cached_tokensis recorded as0and the cached input is charged at the full cache-miss rate.The streaming paths already recognise this spelling —
UsageCapture::finish(cached 三拼写兼容, rant 2026-08-23T14:05:02) andsse.rs::extract_cache_read_tokensboth try DeepSeek-native first, then OpenAI, and all sixsse.rs::record_usagecall sites were fixed for it.CHANGELOG.mdscopes both of those fixes to the passthrough/streaming path (v0.7.10 / v0.7.11), so the non-streaming parser is a missed sibling rather than a deliberate exemption. This PR closes that gap.Related Issue
No tracking issue exists for this; none is fabricated. Verified
closingIssuesReferencesis empty on purpose.Changes
src/gateway.rs::parse_usage,openai_chatarm: thecachedlookup now triesusage.prompt_cache_hit_tokens(DeepSeek native) first and falls back tousage.prompt_tokens_details.cached_tokens(OpenAI) — the same priority the two streaming implementations use. The disjoint return(prompt_tokens − cached).max(0.0)is unchanged, as is every other protocol arm.src/gateway.rs: the doc comment aboveparse_usage(from 2026-08-20T10:17:27) listed only two cache spellings foropenai_chatand was stale after this change; it now names both.Measured impact
Fixture:
prompt_tokens=1000,prompt_cache_hit_tokens=900,completion_tokens=50, prices 10 / 20 USD/M for miss / output (with a cache-hit price of 2 USD/M).cached_tokensSo the same request is charged 2.9x more before the fix. With no cache-hit price configured at all (the shape the earlier probe used), the ratio is 5.5x, because then the entire input is charged at the miss price. Under DeepSeek's real peak rates (3.0 / 0.10) a cache hit is effectively billed at the miss price, i.e. its own unit price is wrong by 30x.
This is an overcharge, and it also corrupts
usage_records.cached_tokens— the column the transaction page renders as the cache-hit breakdown. It is reachable on the plain DeepSeek path:deepseek-paygoexposesopenai_chatathttps://api.deepseek.com, and DeepSeek's native non-streaming usage uses the top-level spelling. The code path depends on the upstream request shape, not the client'sstreamflag, so an anthropic- or responses-protocol client cross-translated to anopenai_chatupstream hitsparse_usageas well — and there the user seescache_read_input_tokens: 900in the response while being billed for a miss.Tests
cargo test全部通过 — 175 passed / 0 failed (baseline 172 + 3 new)cargo fmt --check通过Added tests:
parse_usage_openai_deepseek_native_cache_spelling— the DeepSeek-native body must yield(100, 900, 50), with the OpenAI spelling as a positive control and a third case proving the DeepSeek spelling wins when both are present (same priority as streaming).usage_parsers_agree_on_cache_spelling— feeds the same three usages to bothparse_usageandUsageCapture::finishand asserts the two parsers return identical triples, so the stream/non-stream split cannot silently diverge again. The third case also pins that a body with no cache field still means "no cache" (cached = 0) — that is absence-of-a-field, not the "upstream reported nothing" case C2037 addressed.e2e_nonstream_deepseek_cache_spelling_billing— end-to-end through the realforwardpath against a fake upstream emitting the DeepSeek spelling, assertingusage_records.cached_tokens = 900,tokens = 1050, the settled cost, and the consumer/owner balance split.A/B verification (reverting only the
cachedlookup, keeping the tests) turns exactly the new assertions red while all four pre-existingparse_usagetests stay green, and the failure prints the defect:left: (1000.0, 0.0, 50.0)/right: (100.0, 900.0, 50.0).Checklist
Deliberately out of scope: refactoring the three protocol arms into a shared helper (they legitimately differ —
anthropic'sinput_tokensis already disjoint whileopenai/responsesare not), handlingprompt_cache_miss_tokens, and any new gate (a missing recognition in one function is caught by a test, not a scanner).