[New feature] mlc-llm support

[ctlllll]

https://github.com/mlc-ai/mlc-llm
https://github.com/mlc-ai/llm-perf-bench

[kmn1024]

This issue (and repos) feels pretty dead. What's happening? Are the maintainers working on something that obsoletes Medusa (https://github.com/FasterDecoding/REST)? Is the roadmap still active? @ctlllll @leeyeehoo

[leeyeehoo]

Indeed we are working on Medusa... will release a new version soon :)

[kmn1024]

Thanks Yuhong =) Looking forwards!!

[kmn1024]

I want to ask for some advice regarding model performance. My goal is to run a custom model on pretty cheap, OpenCL-compatible, hardware. Using MLC, the current speed is ~ 3 toks/sec, which is insufficient for fluid interaction.

What would you recommend? Getting Medusa to work on MLC would help a lot, and I would love to try working on it (though it looks pretty daunting). However, if Medusa v2 is coming out, perhaps I should just wait?

[leeyeehoo]

You can refer to this branch. No significant change on the model side. We are working on fully finetuning the models (including new models like Zephyr) and will release the finetuning recipe. Supporting more libraries will be the next step.

[kmn1024]

Thanks for the heads up! If you have a chance, please also include a recipe for adding new types of models too.

[leeyeehoo]

Check this Not sure if you talked about a template to add new model support?

[kmn1024]

Yes! Thanks for pointing that out.