fix(gateway): require upstream usage for OpenAI streams instead of billing zero - #187
Merged
Merged
Conversation
…lling zero A streaming request served by an OpenAI-protocol upstream was billed from the usage block the upstream reports. For that protocol usage is opt-in (stream_options.include_usage) and the gateway forwarded the client body verbatim, never setting the flag - so a stream carrying no usage was settled as zero tokens: no balance change, no usage_records row, no transactions row. Since month_calls is COUNT(*) FROM usage_records, the same root cause also hid the call from the operator view. Ask for the usage: inject stream_options.include_usage for openai_chat outbound bodies (same-protocol and cross-protocol paths share one helper); a client that set the field keeps its own value. anthropic (message_start) and responses (response.completed) always report usage, so they are left untouched. Fail closed when a stream still reports nothing: UsageCapture can now say 'no usage seen at all' as distinct from 'usage reported as 0', and that path logs an error with key id and model instead of being indistinguishable from a free call. Tokens are not estimated - that would be a new pricing mechanism.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A streaming request served by an OpenAI-protocol upstream was billed from the
usageblock that the upstream reports. For that protocol, usage is opt-in — it is only emitted when the request setsstream_options.include_usage: true— andforward_streamforwarded the client body to the upstream verbatim, never setting the flag. An OpenAI-protocol stream that carries no usage chunk was therefore settled as zero tokens: no balance change, nousage_recordsrow, notransactionsrow. Becausemonth_callson the ops page isCOUNT(*) FROM usage_records, the same root cause also made the call invisible in the operator view.The gateway agreed with itself throughout: the tail parser even documents the condition (
openai:最后 chunk 的 usage(stream_options.include_usage 时)) while the code never set the flag.Related Issue
None — no issue exists for this and none is fabricated. Verified
closingIssuesReferences(GraphQL) is empty; recorded as empty on purpose, as in #184/#185/#186.Changes
src/gateway.rsonly.with_include_usage(body, outbound)setsstream_options.include_usage = trueon the outbound body whenoutbound == "openai_chat", applied in one place so both the same-protocol pass-through and the cross-protocol transform branch are covered.openai_chatis touched because it is the one protocol whose server-side usage is opt-in: anthropic always emitsmessage_startusage (captured at the stream head byUsageCapture::push) and theresponsespath always emits aresponse.completedusage block. That asymmetry is exactly why the defect bit the OpenAI pair only.stream_optionsitself keeps its own value (includingfalse), and other keys understream_optionssurvive.UsageCapturegains ausage_seenflag, so "no usage seen at all" is distinguishable from "usage reported as 0" — the two collapsed into the same(0.0, 0.0, 0.0)tuple, whichsettle_usagetreats astokens <= 0.0 → return. On that path the stream finalizer logslog::error!with key id, model and protocol, matchingsettle_usage's existing failure convention, so an unbilled call leaves a trace. The cross-protocol path (whoseUsageSlotalready distinguishesNonefrom a recorded value) gets the same log line.Tests
cargo test: 172 passed, 0 failed (169 pre-existing, all unchanged).with_include_usage_sets_flag_for_openai_only— unit test of the helper: flag injected foropenai_chat; a client-set value kept (includingfalse); siblingstream_optionskeys preserved;anthropic/responsesreturned byte-identical; non-JSON returned unchanged.sse_openai_stream_requests_usage_from_upstream— end-to-end through the router with a body-capturing fake upstream that streams no usage; asserts the outbound request carriesstream_options.include_usage == true.sse_stream_without_usage_is_not_billed_but_flagged— asserts a usage-less stream still bills nothing (balance unchanged, 0usage_records, 0transactions) and thatusage_seendistinguishes "no usage object anywhere" (false) from "usage present but zero" (true), which is the basis for the fail-closed trace.cargo fmt --checkandclippy --all-targets -- -D warningsare clean.A/B. With only the behavioural lines reverted (the helper short-circuited to pass-through and the
usage_seenassignments removed), the three new tests fail while all 169 pre-existing tests still pass, and the failure output prints the leak —{"model":"test-model","stream":true,"messages":[...]}with nostream_options. With the fix restored, 172/172 pass. The three existing streaming tests (sse_openai_stream_passthrough_and_settle,sse_anthropic_stream_settle,sse_cross_protocol_openai_to_anthropic_conversion) are unaffected: their fake upstreams supply usage explicitly.Checklist
main(repository default).fix(gateway): …), branchfix/stream-billing-include-usage.docs//CHANGELOG.mdedit.CHANGELOG's documented rule, a client disconnect still terminates the upstream and bills nothing — that is deliberate and unchanged.let … else, let-chains preserved).Note for the maintainer (not decided here)
The fix takes the non-breaking path: request the usage, and log when it is still missing. The open product question is whether a usage-less stream should instead be rejected (a stream error after the opening
200) rather than merely logged. Rejection changes client-visible behaviour for upstreams that legitimately omit usage; this PR deliberately does not decide that. It is a one-line change at the same call site if you want it.