You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Keep Jev answer confidence for OpenRouter evaluation models in Vercel AI #24860
Vercel AI experimental_evaluate with openrouter.evaluationModel('typesafe/jev-1.13') (@openrouter/ai-sdk-provider 3.1.0) gives an evaluate span, but the answers in gen_ai.output.messages lose confidence and legend.
The OpenRouter provider moves these fields into providerMetadata.openrouter.answers[<question id>] ({ confidence, legend }). Our withProviderConfidence in packages/server-utils/src/integrations/vercel-ai/vercel-ai-dc-subscriber.ts only reads providerMetadata.typesafe.confidence, which is the shape of @ai-sdk/typesafe-ai.
Expected
Evaluate answers from the OpenRouter provider keep confidence (choice, score) and legend (score), the same as answers from @ai-sdk/typesafe-ai.
Notes
OpenRouter also sends providerMetadata.openrouter.usage.cost and provider. We could record these too, but they are optional.
Activity
linear-code commentedon Sep 30, 2026
JS-3854