Skip to content

fix(protocol): forward DeepSeek's cache-hit tokens in non-streamed openai→anthropic translation - #189

Merged
argszero merged 1 commit into
mainfrom
fix/nonstream-anthropic-cache-spelling
Sep 12, 2026
Merged

argszero merged 1 commit into
mainfrom
fix/nonstream-anthropic-cache-spelling

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

The non-streaming openai_chatanthropic response translator (build_anthropic_usage, src/protocol.rs) recognised only two cache-hit spellings (cache_read_input_tokens, prompt_tokens_details.cached_tokens). DeepSeek's native top-level prompt_cache_hit_tokens was dropped, so a non-streamed response converted to the Anthropic protocol reported input_tokens inclusive of the cache hits and emitted no cache_read_input_tokens at all.

The streaming twin already handles all three: sse::extract_cache_read_tokens documents the priority (DeepSeek native → OpenAI → Anthropic) and sse::tests::deepseek_cache_hit_tokens_spelling asserts that the forwarded JSON contains "cache_read_input_tokens":90. The non-streaming translator is therefore the last path still holding the shorter list — missed, not exempted, the same judgement as #187 and #188.

Severity: display/data-only, not money. gateway::forward settles from the original upstream bytes before transform_response runs, so billing for this path is already correct (#188 closed that face). The user-visible effect is that an Anthropic-protocol client sees input_tokens inclusive and no cache_read_input_tokens, which disagrees with the gateway's own transaction page for the same call. This is not an overcharge.

Related Issue

No linked issue exists for this change and none may be fabricated — closingIssuesReferences is empty on purpose, as in #184#188. This is an internal-consistency fix found by re-reading the response translators.

Changes

  • build_anthropic_usage now tries the three spellings in the streaming sibling's documented priority (DeepSeek native prompt_cache_hit_tokens → Anthropic cache_read_input_tokens → OpenAI prompt_tokens_details.cached_tokens), and treats a zero-valued spelling as "not reported" so it falls through — matching sse.rs's > 0 semantics. That makes the two transports agree on every input, not just the common one.
  • The disjoint arithmetic (saturating_sub) and the if cached > 0 { … } emission rule are untouched.
  • No config/data-structure change (nothing to mirror into config.example.toml).

Production diff is one lookup chain plus its doc comment.

Tests

  • cargo test178 passed (175 baseline + 3 new)
  • cargo fmt --check clean
  • cargo clippy --all-targets -- -D warnings clean

New tests, using the same fixture numbers as the streaming test (prompt_tokens=100 with 90 cached → input=10 / cached=90 / output=50):

  • openai_chat_resp_to_anthropic_deepseek_cache_spelling — the axis: the DeepSeek-native spelling is forwarded and the hit is disjoint from input_tokens; plus the priority case (DeepSeek wins when two spellings are present) and the zero-valued fall-through.
  • openai_chat_resp_to_anthropic_other_cache_spellings — positive control (OpenAI + Anthropic spellings), green before and after the fix.
  • anthropic_usage_breakdown_agrees_across_transports — a structural cross-path assertion: the same upstream usage object fed to the streaming converter (sse::openai_sse_to_anthropic) and to the non-streaming translator must yield the same input_tokens / cache_read_input_tokens / output_tokens breakdown. This is the constraint that would have caught the defect originally.

A/B — reverting only the lookup, keeping the tests:

test before (73fae8c) after
openai_chat_resp_to_anthropic_deepseek_cache_spelling FAILEDleft: Null, right: 90 (cache_read_input_tokens absent, input_tokens: 100) ok
anthropic_usage_breakdown_agrees_across_transports FAILED — stream input_tokens: 10 vs non-stream 100 ok
openai_chat_resp_to_anthropic_other_cache_spellings (control) ok ok
the other 175 tests ok ok

Checklist

  • Branch name follows the convention (fix/...)
  • Commit message uses Conventional Commits
  • Single responsibility, minimal change

@argszero
argszero merged commit 9bd2dec into main Sep 12, 2026
1 check passed
@argszero
argszero deleted the fix/nonstream-anthropic-cache-spelling branch September 12, 2026 23:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant