fix(protocol): forward DeepSeek's cache-hit tokens in non-streamed openai→anthropic translation - #189
Merged
Conversation
…enai→anthropic translation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The non-streaming
openai_chat→anthropicresponse translator (build_anthropic_usage,src/protocol.rs) recognised only two cache-hit spellings (cache_read_input_tokens,prompt_tokens_details.cached_tokens). DeepSeek's native top-levelprompt_cache_hit_tokenswas dropped, so a non-streamed response converted to the Anthropic protocol reportedinput_tokensinclusive of the cache hits and emitted nocache_read_input_tokensat all.The streaming twin already handles all three:
sse::extract_cache_read_tokensdocuments the priority (DeepSeek native → OpenAI → Anthropic) andsse::tests::deepseek_cache_hit_tokens_spellingasserts that the forwarded JSON contains"cache_read_input_tokens":90. The non-streaming translator is therefore the last path still holding the shorter list — missed, not exempted, the same judgement as #187 and #188.Severity: display/data-only, not money.
gateway::forwardsettles from the original upstream bytes beforetransform_responseruns, so billing for this path is already correct (#188 closed that face). The user-visible effect is that an Anthropic-protocol client seesinput_tokensinclusive and nocache_read_input_tokens, which disagrees with the gateway's own transaction page for the same call. This is not an overcharge.Related Issue
No linked issue exists for this change and none may be fabricated —
closingIssuesReferencesis empty on purpose, as in #184–#188. This is an internal-consistency fix found by re-reading the response translators.Changes
build_anthropic_usagenow tries the three spellings in the streaming sibling's documented priority (DeepSeek nativeprompt_cache_hit_tokens→ Anthropiccache_read_input_tokens→ OpenAIprompt_tokens_details.cached_tokens), and treats a zero-valued spelling as "not reported" so it falls through — matchingsse.rs's> 0semantics. That makes the two transports agree on every input, not just the common one.saturating_sub) and theif cached > 0 { … }emission rule are untouched.config.example.toml).Production diff is one lookup chain plus its doc comment.
Tests
cargo test— 178 passed (175 baseline + 3 new)cargo fmt --checkcleancargo clippy --all-targets -- -D warningscleanNew tests, using the same fixture numbers as the streaming test (
prompt_tokens=100with 90 cached →input=10 / cached=90 / output=50):openai_chat_resp_to_anthropic_deepseek_cache_spelling— the axis: the DeepSeek-native spelling is forwarded and the hit is disjoint frominput_tokens; plus the priority case (DeepSeek wins when two spellings are present) and the zero-valued fall-through.openai_chat_resp_to_anthropic_other_cache_spellings— positive control (OpenAI + Anthropic spellings), green before and after the fix.anthropic_usage_breakdown_agrees_across_transports— a structural cross-path assertion: the same upstreamusageobject fed to the streaming converter (sse::openai_sse_to_anthropic) and to the non-streaming translator must yield the sameinput_tokens/cache_read_input_tokens/output_tokensbreakdown. This is the constraint that would have caught the defect originally.A/B — reverting only the lookup, keeping the tests:
73fae8c)openai_chat_resp_to_anthropic_deepseek_cache_spellingleft: Null, right: 90(cache_read_input_tokensabsent,input_tokens: 100)anthropic_usage_breakdown_agrees_across_transportsinput_tokens: 10vs non-stream100openai_chat_resp_to_anthropic_other_cache_spellings(control)Checklist
fix/...)