Skip to content

fix(gateway): recognise DeepSeek's cache-hit token spelling for non-streamed responses - #188

Merged
argszero merged 1 commit into
mainfrom
fix/nonstream-deepseek-cache-spelling
Sep 12, 2026
Merged

argszero merged 1 commit into
mainfrom
fix/nonstream-deepseek-cache-spelling

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Non-streamed responses from an OpenAI-protocol upstream that reports cache hits with DeepSeek's native spelling (prompt_cache_hit_tokens) are currently billed as if nothing was cached: cached_tokens is recorded as 0 and the cached input is charged at the full cache-miss rate.

The streaming paths already recognise this spelling — UsageCapture::finish (cached 三拼写兼容, rant 2026-08-23T14:05:02) and sse.rs::extract_cache_read_tokens both try DeepSeek-native first, then OpenAI, and all six sse.rs::record_usage call sites were fixed for it. CHANGELOG.md scopes both of those fixes to the passthrough/streaming path (v0.7.10 / v0.7.11), so the non-streaming parser is a missed sibling rather than a deliberate exemption. This PR closes that gap.

Related Issue

No tracking issue exists for this; none is fabricated. Verified closingIssuesReferences is empty on purpose.

Changes

  • src/gateway.rs::parse_usage, openai_chat arm: the cached lookup now tries usage.prompt_cache_hit_tokens (DeepSeek native) first and falls back to usage.prompt_tokens_details.cached_tokens (OpenAI) — the same priority the two streaming implementations use. The disjoint return (prompt_tokens − cached).max(0.0) is unchanged, as is every other protocol arm.
  • src/gateway.rs: the doc comment above parse_usage (from 2026-08-20T10:17:27) listed only two cache spellings for openai_chat and was stale after this change; it now names both.
  • No configuration or data-structure change, so no example-file sync is needed.

Measured impact

Fixture: prompt_tokens=1000, prompt_cache_hit_tokens=900, completion_tokens=50, prices 10 / 20 USD/M for miss / output (with a cache-hit price of 2 USD/M).

path cached_tokens cost
non-stream, DeepSeek spelling — before 0 0.0792
non-stream, DeepSeek spelling — after 900 0.02736
non-stream, OpenAI spelling 900 0.02736
stream, DeepSeek spelling 900 0.02736

So the same request is charged 2.9x more before the fix. With no cache-hit price configured at all (the shape the earlier probe used), the ratio is 5.5x, because then the entire input is charged at the miss price. Under DeepSeek's real peak rates (3.0 / 0.10) a cache hit is effectively billed at the miss price, i.e. its own unit price is wrong by 30x.

This is an overcharge, and it also corrupts usage_records.cached_tokens — the column the transaction page renders as the cache-hit breakdown. It is reachable on the plain DeepSeek path: deepseek-paygo exposes openai_chat at https://api.deepseek.com, and DeepSeek's native non-streaming usage uses the top-level spelling. The code path depends on the upstream request shape, not the client's stream flag, so an anthropic- or responses-protocol client cross-translated to an openai_chat upstream hits parse_usage as well — and there the user sees cache_read_input_tokens: 900 in the response while being billed for a miss.

Tests

  • cargo test 全部通过 — 175 passed / 0 failed (baseline 172 + 3 new)
  • cargo fmt --check 通过
  • 新增/更新了单元测试(如适用)

Added tests:

  • parse_usage_openai_deepseek_native_cache_spelling — the DeepSeek-native body must yield (100, 900, 50), with the OpenAI spelling as a positive control and a third case proving the DeepSeek spelling wins when both are present (same priority as streaming).
  • usage_parsers_agree_on_cache_spelling — feeds the same three usages to both parse_usage and UsageCapture::finish and asserts the two parsers return identical triples, so the stream/non-stream split cannot silently diverge again. The third case also pins that a body with no cache field still means "no cache" (cached = 0) — that is absence-of-a-field, not the "upstream reported nothing" case C2037 addressed.
  • e2e_nonstream_deepseek_cache_spelling_billing — end-to-end through the real forward path against a fake upstream emitting the DeepSeek spelling, asserting usage_records.cached_tokens = 900, tokens = 1050, the settled cost, and the consumer/owner balance split.

A/B verification (reverting only the cached lookup, keeping the tests) turns exactly the new assertions red while all four pre-existing parse_usage tests stay green, and the failure prints the defect: left: (1000.0, 0.0, 50.0) / right: (100.0, 900.0, 50.0).

Checklist

  • 分支命名符合约定(feat/ /fix/ /docs/ ...)
  • Commit message 使用 Conventional Commits 格式
  • 单一职责,改动最小化

Deliberately out of scope: refactoring the three protocol arms into a shared helper (they legitimately differ — anthropic's input_tokens is already disjoint while openai/responses are not), handling prompt_cache_miss_tokens, and any new gate (a missing recognition in one function is caught by a test, not a scanner).

…treamed responses

非流式响应解析器 `parse_usage` 的 openai_chat 分支原先只认 OpenAI 拼写
`prompt_tokens_details.cached_tokens`,不认 DeepSeek 原生顶层拼写
`prompt_cache_hit_tokens`,导致非流式 DeepSeek 响应把缓存命中部分按**未命中全价**
计费,且 `usage_records.cached_tokens` 记为 0。

流式侧早已修过同一问题:`UsageCapture::finish` 注释明写「cached 三拼写兼容」
(rant 2026-08-23T14:05:02),`sse.rs::extract_cache_read_tokens` 同样按
DeepSeek → OpenAI 优先级取;六处 `record_usage` 亦然。CHANGELOG 对 v0.7.10 / v0.7.11
两次修复的范围描述都是 passthrough/流式路径 ⇒ 非流式解析器属**漏修**,非有意豁免。

修复:`cached` 先取 `prompt_cache_hit_tokens`,回退 `prompt_tokens_details.cached_tokens`,
优先级与两条流式实现一致;`(prompt_tokens − cached).max(0.0)` 的 disjoint 语义不变。
changelog 里 2026-08-20T10:17:27 引入的归属说明同步补齐(它此前只列了两种拼写)。
@argszero
argszero merged commit 73fae8c into main Sep 12, 2026
1 check passed
@argszero
argszero deleted the fix/nonstream-deepseek-cache-spelling branch September 12, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant