feature: unbuffered token stream

[mudler]

Now this should be quite easy at least for the llama.cpp backend: go-skynet/go-llama.cpp#28 thanks to @noxer's contribution ( ❤️ ) now it's just a matter of wiring things up in the SSE callback here in the server

• go-llama.cpp
• gpt4all.cpp
• gpt2.cpp
• rwkv.cpp

[mudler]

the go-gpt4all-j backend has support as well for unbuffered token stream, so we should (almost) all good as 2 out of 3 backends supports it till now

[mudler]

implemented for llama.cpp backend!

[apiad]

This is the most exciting feature for my use case! I'm wondering, do you have already planned how the API will support this? Thanks!

[mudler]

This is the most exciting feature for my use case! I'm wondering, do you have already planned how the API will support this? Thanks!

LocalAI is following the OpenAI specs. Therefore tokens are pushed via SSE (server-sent events) streams. It currently works already for llama.cpp models, like: vicuna, alpaca, wizardlm, and alikes.

[EchedelleLR]

I am getting memory issues with LocalAI on 16 GiB of RAM even in the smaller chat model.

This only happens on "stream: true". I guess this is related to buffering being enabled in the request?

If I make a single POST request without "stream: true", the request works properly and I receive a proper answer.

I am using GPT4All models.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.