[FEATURE]:Provider-Specific Rate Limiting

[Fighteros]

Feature hasn't been suggested before.

• I have verified this feature I'm about to request hasn't been suggested before.

Describe the enhancement you want to request

Feature Request: Provider-Specific Rate Limiting

Summary

Add support for configurable rate limits on a per-provider basis.

Problem

Many providers enforce different RPM (Requests Per Minute) and TPM (Tokens Per Minute) limits. While users can switch between multiple providers, there is currently no built-in way to throttle requests for a specific provider.

This can lead to:

• Frequent 429/rate-limit errors.
• Providers becoming temporarily unavailable during agent-heavy workflows.
• The need for users to maintain custom forks or external proxies solely for request throttling.

Proposed Solution

Allow rate limits to be configured per provider.

Example configuration:

{
  "providers": {
    "anthropic": {
      "rateLimit": {
        "requestsPerMinute": 40
      }
    },
    "openai": {
      "rateLimit": {
        "requestsPerMinute": 100
      }
    }
  }
}

Or alternatively:

{
  "rateLimits": {
    "anthropic": {
      "rpm": 40
    },
    "openai": {
      "rpm": 100
    }
  }
}

Expected Behavior

When requests are generated:

• Requests for a provider should be queued and throttled according to its configured limit.
• Limits should be applied independently per provider.
• Other providers should continue operating normally without being affected.
• Queued requests should be processed automatically once capacity becomes available.

Benefits

• Reduces 429 errors.
• Improves stability for long-running agent workflows.
• Eliminates the need for custom forks and proxy-based throttling.
• Makes it easier to work with providers that have lower rate limits than others.

Additional Considerations

Future enhancements could include:

• TPM (tokens per minute) limits.
• Concurrent request limits.
• Automatic backoff and retry strategies.
• Provider-specific burst settings.
• Visibility into queued requests and current rate-limit utilization.

Thanks for considering this feature. I believe it would significantly improve the experience for users working with multiple providers that have different rate-limit constraints.

[github-actions]

This issue might be a duplicate of existing issues. Please check:

• [FEATURE]: Make rate limit handling configurable with custom request limits #26314: [FEATURE]: Make rate limit handling configurable with custom request limits — same core request for per-provider configurable rate limits to avoid 429 errors

[MMOStars]

• need this to use nvidia build properly

[github-actions]

To stay organized issues are automatically closed after 60 days of no activity. If the issue is still relevant please open a new one.