Skip to content

Support using models from HuggingFace directly #140

Open
Open
Support using models from HuggingFace directly#140
Assignees
vipannalla

Description

@samos123
samos123
opened on Oct 12, 2024

I should be able to serve a model by simply providing the HuggingFace model ID. Requiring users to convert checkpoints is too troublesome.

Activity

  1. vipannalla commented on Oct 30, 2024

    @vipannalla
    vipannalla
    on Oct 30, 2024
    Collaborator

    Thanks for the feedback. This feature is currently not supported, but we have added it to our roadmap to simplify. Some models (such as LLama variants) need explicit acknowledgements from Meta's site before you can use them.

  2. vipannalla
    self-assigned this
    on Oct 30, 2024

  3. samos123 commented on Oct 30, 2024

    @samos123
    samos123
    on Oct 30, 2024
    Author

    That can be handled by respecting HF_TOKEN environment variable to automatically download auth gated models. That's how vLLM and other OSS does it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions