Skip to content

fix(llm): prompted JSON for models refusing forced tools, mid-stream quota classification - #391

Merged
filipeforattini merged 4 commits into
mainfrom
llm-tool-quota
Sep 23, 2026
Merged

filipeforattini merged 4 commits into
mainfrom
llm-tool-quota

Conversation

@filipeforattini

@filipeforattini filipeforattini commented Sep 23, 2026 •

Copy link
Copy Markdown

Plan wave 1, item 3 (llm-forced-tool-and-quota): A4 + A5.

A4: LLM.generateObject without a forced tool choice

  • New pure LLM.supportsForcedToolChoice(modelID, declared?) in packages/llm/src/llm.ts (Claude Opus >= 5.5, Fable, Mythos refuse; declared === false wins). ProviderTransform.supportsForcedToolChoice in redcode now delegates to it (redcode -> llm dependency direction).
  • ModelCompatibility.forcedToolChoice lets a model or router declare the refusal on an llm Model.
  • For refusing models, generateObject sends no tools/tool_choice, appends a system instruction with the JSON Schema, decodes the reply (fences stripped) against the same schema, and repairs once (same shape as fix(provider): never force a tool choice on models that refuse one #377 for Agent.generate). A second failure is an InvalidProviderOutput LLMError.

A5: mid-stream quota / content-policy classification

  • New shared streamProviderError in packages/llm/src/provider-error.ts reuses isQuotaFailure / isContentPolicyFailure (the fix(provider): classify gateway quota exhaustion and 402 as non-retryable #378 classifiers). A stream has no status, so throttle shapes (rate_limit_error, rate_limit_exceeded, Bedrock throttlingException) stand in for 429.
  • Anthropic Messages, OpenAI Responses and Bedrock Converse stream parsers route their provider-error events through it: quota -> classification: "quota", retryable: false, message with the shared QUOTA_HINT; content policy -> classification: "content-policy", retryable: false.
  • ProviderFailureClassification gains quota and content-policy.
  • QUOTA_HINT moved to llm and reused by provider/error.ts.
  • Legacy processor: a refusal-classified provider-error becomes a non-retryable ResponseStreamError (refusal), which MessageV2.fromError records as metadata.classification, and SessionRetry.retryable never retries it, whatever the message pattern says. (Before, it became NamedError.Unknown, and the quota hint's "exhausted" would have read as an overload.) The v2 runner already never retries provider-error events.

Tests

  • packages/llm/test/generate-object.test.ts: capability table, prompted path, one repair, failure after repair, declared refusal.
  • Per protocol: anthropic-messages, openai-responses, bedrock-converse quota/content-policy events.
  • packages/redcode/test/session/retry.test.ts: classified stream refusal is not retried.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@filipeforattini
filipeforattini enabled auto-merge (squash) September 23, 2026 18:07
@filipeforattini
filipeforattini merged commit 04d1fde into main Sep 23, 2026
35 of 37 checks passed
@filipeforattini
filipeforattini deleted the llm-tool-quota branch September 23, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant