Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or requestin-prA pull request is open to resolve this issueA pull request is open to resolve this issue
Milestone
Description
Phi-4 uses Tiktoken tokenizer (100k vocab).
we now use the tiktoken tokenizer (for better multilingual support) with a padded vocabulary size of 100,352 (including unused tokens)
Consider adding it as an option to the encoding map so it's easier to create.
machinelearning/src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs
Lines 1025 to 1035 in 01c4164
| private static readonly (string Prefix, ModelEncoding Encoding)[] _modelPrefixToEncoding = | |
| [ | |
| // chat | |
| ( "o1-", ModelEncoding.O200kBase ), // e.g. o1-mini | |
| ( "gpt-4o-", ModelEncoding.O200kBase), // e.g., gpt-4o-2024-05-13 | |
| ( "gpt-4-", ModelEncoding.Cl100kBase), // e.g., gpt-4-0314, etc., plus gpt-4-32k | |
| ( "gpt-3.5-", ModelEncoding.Cl100kBase), // e.g, gpt-3.5-turbo-0301, -0401, etc. | |
| ( "gpt-35-", ModelEncoding.Cl100kBase ) // Azure deployment name | |
| ]; | |
| private static readonly Dictionary<string, ModelEncoding> _modelToEncoding = |
Reacted by Jason Haley
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestin-prA pull request is open to resolve this issueA pull request is open to resolve this issue
Activity
MaxAkbar commentedon Jan 12, 2025
I got the model to load using the following code, and generate text but I can't seem to workout out the stop sequence:
tarekgh commentedon Jan 12, 2025
@luisquintanilla looking at https://huggingface.co/microsoft/phi-4/tree/main looks it is using
vocab.jsonandmerges.txtsimilar to Phi-2 tokenizer files format (which is based on CodeGen tokenizer). I know the technical article mentionsTiktokenbut I couldn't find vocab files that Tiktoken tokenizer can load. Do you know who can clarify that?MaxAkbar commentedon Jan 12, 2025
Looking at the tokenizer_config.json it says it's using
GPT2Tokenizer. I think the reason the StopSequences isn't working is that the tokenizer isn't quite correct.I also ran the following:
Got the following confirmation:
tarekgh commentedon Jan 12, 2025
For record:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/tokenization_gpt2_fast.py
https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_fast.py#L82
https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1384
luisquintanilla commentedon Jan 14, 2025
Adding to this thread. Looks like there may have been a bug with the original Phi-4 tokenizer published which validated @MaxAkbar observations.
https://unsloth.ai/blog/phi4