Repository navigation
Add a Test connection action to model and provider configuration #409
Description
Feature request
Add a Test connection action to the model/provider configuration page so users can verify their configuration before returning to a Bot conversation.
Motivation
During first-time setup, choosing a model does not establish whether its API key, endpoint, and model selection actually work. Users currently discover configuration problems only when a conversation run fails. A test alongside the configuration would provide immediate feedback and make setup easier to troubleshoot.
Suggested behavior
- Offer a Test connection button beside the model/provider configuration.
- Test the configured provider endpoint, credentials, and selected model with a minimal model request, rather than only checking whether the server is reachable.
- Show an in-progress state and a clear success or failure result on the same page.
- On failure, give an actionable explanation where possible, such as missing/invalid API key, incorrect endpoint, unavailable model, timeout, or rate limit. Do not display credentials in results.
- If the test sends a billable model request, make that clear to the user.
The test should use the same configuration and routing as an actual Bot run, so a successful result is meaningful for subsequent use.
Context
Requested after first-time model setup in FlyBot Web. Related setup improvements: #405 (configuration guidance below missing-key errors) and #407 (direct navigation to model configuration).
This is a user feature request; the reporting agent has not independently verified whether an equivalent test exists elsewhere in the underlying Harness UI.
Activity
Birfy commentedon Oct 19, 2026
Triage on main
7b9e90b, 2026-10-19 ~12:10 Asia/Shanghai: needs-human. Questions for the owner: is a billable minimal model request acceptable for a Test connection button (with a note that it costs a request)? And should this wait for DSH to offer a provider test (FlyBot must not patch DSH), or be built in the FlyBot model menu?Birfy commentedon Oct 19, 2026
Triage on main
0852b3f, 2026-10-19 16:59 Asia/Shanghai: still needs-human. FlyBot cannot add a button to DSH's provider page without patching DSH, so a FlyBot version would live in FlyBot's own UI.Owner-delegated details (if approved):
Worksor a short reason: no/invalid key, wrong endpoint, model not found, timeout, rate limit. Provider text is never echoed, in case it contains a key, as in Add a model configuration shortcut or guidance below missing API key errors #405.Confirming: OK to send one billable request per click, and to build it in FlyBot now instead of waiting for DSH?
Recommendation: yes to both.
Birfy commentedon Oct 19, 2026
Owner decision (2026-10-19 17:21 Asia/Shanghai): build "Test connection" in FlyBot now, in Bot settings under Model. Each click sends one tiny real model request on the Bot's own route; the UI says it may be billed. Ready to develop.
Birfy commentedon Oct 19, 2026
Claimed by Birfy at 2026-10-19 17:22 Asia/Shanghai. Working on branch
feat/409-test-connection(Birfy/FlyBot), based on upstream maind65e09d.Birfy commentedon Oct 19, 2026
Minor details decided while building (PR #434):
desiredModel: its chosen model, or the fallback a run would use when that model is not served); the result names the model it reached.maxTokens16, the model's least thinking (off, else its first level), 30 s timeout, no retry, one test per Bot at a time. A reply cut off at the token limit still counts as working. Usage is recorded on the Bot as Other.Count a test stream with no finish as a failure
Add Test connection to Bot settings → Model (ThinkFlowLab#434)