[monatis]

closes #60, #49, #32

This is still WIP and not yet for anything yet.
I hope to finish it by Sunday.

• Rewrite the conversion script for GGUF support.
• Output text-only, vision-only and two-tower models according to the CLI argument.
• Allow overriding mean / std values for image processing.
• Load GGUF file and infer with it (WIP).
• Re-implement the quantization function.
• Support f32, q5_0, q5_1 and q8_0 models in addition to f16, q4_0 and q4_1.
• Handle memory buffer allocation logic for new text-only and vision-only models.
• Test and upload different CLIP models in new GGUF format.
• Update Python bindings.

[monatis]

I set this ready for review although it will require some tests with different models and fixes in memory allocation for text-only and vision-only model variants.

I'll do tests and make some code polishing over the coming days. Any test and feedback will be much appreciated btw.

[monatis]

I'm preparing the PR for merging later today. Added a shell script for bulk model conversion. Another script will do quantization in bulk. Then uploading models + updating readme and merging.

Python binding will require an update in its automatic model selection as now the smallest model in a repo is a text-only model, but I'll leave it to another PR.

[monatis]

Also updated the Python bindings to work with the new GGUF format and find pre-converted models uploaded to HF.

I think this is good to merge now.

[Green-Sky]

you sure this is a good idea? i can take down /reupload my repos too.

also, you only changed the text, not the link

[monatis]

I don't have a strong opinion in this, but I just wanted to close this PR to jump to other tasks (implement BERT and large multimodal models). I only uploaded a few models --my internet speed is not so good. If you'd like to re-upload models I'd only be happy :D and new shell scripts might help you with conversions and quantizations.

[Green-Sky]

file names are somewhat lenghty now, with "ggml-model" "ggml-text-model" and "ggml-vision-model". feels like the "ggml" here is redundant (.gguf).

[monatis]

yes, it might be {full|text|vision}-model-{ftype}.gguf