Skip to content

fix(gateway): require upstream usage for OpenAI streams instead of billing zero - #187

Merged
argszero merged 1 commit into
mainfrom
fix/stream-billing-include-usage
Sep 12, 2026
Merged

argszero merged 1 commit into
mainfrom
fix/stream-billing-include-usage

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

A streaming request served by an OpenAI-protocol upstream was billed from the usage block that the upstream reports. For that protocol, usage is opt-in — it is only emitted when the request sets stream_options.include_usage: true — and forward_stream forwarded the client body to the upstream verbatim, never setting the flag. An OpenAI-protocol stream that carries no usage chunk was therefore settled as zero tokens: no balance change, no usage_records row, no transactions row. Because month_calls on the ops page is COUNT(*) FROM usage_records, the same root cause also made the call invisible in the operator view.

The gateway agreed with itself throughout: the tail parser even documents the condition (openai:最后 chunk 的 usage(stream_options.include_usage 时)) while the code never set the flag.

Related Issue

None — no issue exists for this and none is fabricated. Verified closingIssuesReferences (GraphQL) is empty; recorded as empty on purpose, as in #184/#185/#186.

Changes

src/gateway.rs only.

  1. Ask for the usage (the fix). New with_include_usage(body, outbound) sets stream_options.include_usage = true on the outbound body when outbound == "openai_chat", applied in one place so both the same-protocol pass-through and the cross-protocol transform branch are covered.
    • Only openai_chat is touched because it is the one protocol whose server-side usage is opt-in: anthropic always emits message_start usage (captured at the stream head by UsageCapture::push) and the responses path always emits a response.completed usage block. That asymmetry is exactly why the defect bit the OpenAI pair only.
    • A client that set stream_options itself keeps its own value (including false), and other keys under stream_options survive.
    • A non-JSON body (the server-side fallback pass-through path) is returned unchanged rather than blocking the forward.
  2. Fail closed when the stream still reports nothing. UsageCapture gains a usage_seen flag, so "no usage seen at all" is distinguishable from "usage reported as 0" — the two collapsed into the same (0.0, 0.0, 0.0) tuple, which settle_usage treats as tokens <= 0.0 → return. On that path the stream finalizer logs log::error! with key id, model and protocol, matching settle_usage's existing failure convention, so an unbilled call leaves a trace. The cross-protocol path (whose UsageSlot already distinguishes None from a recorded value) gets the same log line.
    • Tokens are deliberately not estimated from streamed content: that is a new pricing mechanism, not a bug fix.

Tests

  • Existing tests pass — cargo test: 172 passed, 0 failed (169 pre-existing, all unchanged).
  • New tests added:
    • with_include_usage_sets_flag_for_openai_only — unit test of the helper: flag injected for openai_chat; a client-set value kept (including false); sibling stream_options keys preserved; anthropic/responses returned byte-identical; non-JSON returned unchanged.
    • sse_openai_stream_requests_usage_from_upstream — end-to-end through the router with a body-capturing fake upstream that streams no usage; asserts the outbound request carries stream_options.include_usage == true.
    • sse_stream_without_usage_is_not_billed_but_flagged — asserts a usage-less stream still bills nothing (balance unchanged, 0 usage_records, 0 transactions) and that usage_seen distinguishes "no usage object anywhere" (false) from "usage present but zero" (true), which is the basis for the fail-closed trace.
  • cargo fmt --check and clippy --all-targets -- -D warnings are clean.

A/B. With only the behavioural lines reverted (the helper short-circuited to pass-through and the usage_seen assignments removed), the three new tests fail while all 169 pre-existing tests still pass, and the failure output prints the leak — {"model":"test-model","stream":true,"messages":[...]} with no stream_options. With the fix restored, 172/172 pass. The three existing streaming tests (sse_openai_stream_passthrough_and_settle, sse_anthropic_stream_settle, sse_cross_protocol_openai_to_anthropic_conversion) are unaffected: their fake upstreams supply usage explicitly.

Checklist

  • Base branch is main (repository default).
  • Conventional Commit (fix(gateway): …), branch fix/stream-billing-include-usage.
  • No schema change, no i18n key, no new dependency, no new gate, no docs//CHANGELOG.md edit.
  • Scope is confined to streaming billing: per the CHANGELOG's documented rule, a client disconnect still terminates the upstream and bills nothing — that is deliberate and unchanged.
  • Follows the modern Rust style already used in this file (let … else, let-chains preserved).

Note for the maintainer (not decided here)

The fix takes the non-breaking path: request the usage, and log when it is still missing. The open product question is whether a usage-less stream should instead be rejected (a stream error after the opening 200) rather than merely logged. Rejection changes client-visible behaviour for upstreams that legitimately omit usage; this PR deliberately does not decide that. It is a one-line change at the same call site if you want it.

…lling zero

A streaming request served by an OpenAI-protocol upstream was billed from the
usage block the upstream reports. For that protocol usage is opt-in
(stream_options.include_usage) and the gateway forwarded the client body
verbatim, never setting the flag - so a stream carrying no usage was settled as
zero tokens: no balance change, no usage_records row, no transactions row. Since
month_calls is COUNT(*) FROM usage_records, the same root cause also hid the
call from the operator view.

Ask for the usage: inject stream_options.include_usage for openai_chat outbound
bodies (same-protocol and cross-protocol paths share one helper); a client that
set the field keeps its own value. anthropic (message_start) and responses
(response.completed) always report usage, so they are left untouched.

Fail closed when a stream still reports nothing: UsageCapture can now say
'no usage seen at all' as distinct from 'usage reported as 0', and that path
logs an error with key id and model instead of being indistinguishable from a
free call. Tokens are not estimated - that would be a new pricing mechanism.
@argszero
argszero merged commit ec158bd into main Sep 12, 2026
1 check passed
@argszero
argszero deleted the fix/stream-billing-include-usage branch September 12, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant