Repository navigation
Migrate to vLLM #236
Description
We have configs, tests and code with ollama, but I believe using vLLM is a better option.
It allows more freedom in configuration and deployment.
Ollama, as I understand is "good for starters".
I'd love if someone could link comprehensive comparisons, otherwise happy to get this migration going.
Activity
linear commentedon Dec 30, 2025
CHR-66
01PrathamS commentedon Dec 30, 2025
can i take this up, don't know how to make around and how to make around without gpu (if needed at all - don't know).
thestumonkey commentedon Jan 1, 2026
be aware that this has different setup challenges on different hardware
AnkushMalaker commentedon Jan 1, 2026
Hey @01PrathamS
I'm copying what I said on discord for broader visibility
Let me know any thoughts.
0xrushi commentedon Jan 2, 2026
vllm is hard, it won't work on amd/ even some nvidia gpus.
also vllm for what? we can have it decoupled into config.yml with openai base url
AnkushMalaker commentedon Jan 3, 2026
@0xrushi
I just want to move away from ollama to a more flexible and performant inference engine.
Llama.cpp, vllm, anything works. Yes I agree we should de couple from config and only have the API url there