Add OpenAI adapter and runtime support for ReFRAG - #142
Conversation
Introduces scripts/refrag_openai.py, providing an OpenAI-based adapter for decoder-side ReFRAG operations. Updates requirements.txt to require openai>=2.11.0. Modifies context_answer.py and query_expand.py to support OpenAI as a runtime, including auto-detection via OPENAI_API_KEY and proper client instantiation. Decoder availability checks and prompt handling are updated to accommodate the new OpenAI integration.
There was a problem hiding this comment.
Some notes:
force_json mismatch (likely breakage) - query_expand.py + refrag_openai.py
query_expand.py asks for a JSON array, but OpenAIRefragClient sets response_format={"type":"json_object"} when force_json=True. That combination is inconsistent and can cause hard failures or malformed outputs.
Ask: either switch output to an object schema (e.g., {"queries":[...]}) or don’t use json_object and just parse an array defensively.
Auto-selecting OpenAI when OPENAI_API_KEY exists (behavioral surprise) - context_answer.py
Runtime auto-detect now prefers OpenAI if OPENAI_API_KEY is set. In many environments that key may exist incidentally, causing silent routing to OpenAI (cost, latency, privacy, different outputs).
Ask: require explicit REFRAG_RUNTIME=openai (or gate auto-detect behind an opt-in env flag), and log the selected runtime.
Async event-loop bug potential generate_pseudo_tags_batch()
It calls run_until_complete after grabbing/creating an event loop. If invoked from an already-running loop (common in async servers), this can raise at runtime.
Ask: provide an async variant and keep a safe sync wrapper (or detect running loop and avoid run_until_complete).
Autodetect of REFRAG_RUNTIME is now opt-in via REFRAG_RUNTIME_AUTODETECT to prevent unintended API calls. Query expansion prompts for OpenAI and GLM now require a JSON object with a 'queries' key. Added an async variant generate_pseudo_tags_batch_async to refrag_openai.py and improved event loop handling for batch pseudo tag generation.
Updated .env.example and CONFIGURATION.md to add OpenAI as a supported decoder backend, document new environment variables for OpenAI integration, and clarify runtime selection with an opt-in auto-detection mechanism. Expanded documentation for decoder configuration and query expansion to reflect these changes.
Matches 1793a46 pattern: detect running loop and use ThreadPoolExecutor to avoid RuntimeError when generate_pseudo_tags_batch is called from async context (e.g., FastMCP server handlers).
|
Re-checking |
Eliminated the REFRAG_RUNTIME_AUTODETECT environment variable and all related auto-detection logic from configuration, documentation, and code. The decoder runtime must now be set explicitly, defaulting to 'llamacpp' if unset, to prevent unintended API calls.
…ngine into open-ai-support
Add OpenAI adapter and runtime support for ReFRAG
3 participants
Introduces scripts/refrag_openai.py, providing an OpenAI-based adapter for decoder-side ReFRAG operations. Updates requirements.txt to require openai>=2.11.0. Modifies context_answer.py and query_expand.py to support OpenAI as a runtime, including auto-detection via OPENAI_API_KEY and proper client instantiation. Decoder availability checks and prompt handling are updated to accommodate the new OpenAI integration.