You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@fanyang-mono the change will be included in our next preview (hopefully in 10 days or so). But you don't have to be blocked on that as you can still create the tokenizer using other compatible model name. Like
Activity
stephentoub commentedon Apr 30, 2025
@tarekgh, looks like OpenAI stated yesterday that gpt-4.1 uses the same tokenizer as 4o.
https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758/2
tarekgh commentedon Apr 30, 2025
@stephentoub I was waiting for OpenAI to officially add it to their tokenizer library openai/tiktoken#395.
fanyang-mono commentedon May 6, 2025
@tarekgh Thanks for adding the support. I wonder when this will be available to be consumed?
tarekgh commentedon May 6, 2025
@fanyang-mono the change will be included in our next preview (hopefully in 10 days or so). But you don't have to be blocked on that as you can still create the tokenizer using other compatible model name. Like
or you can create it from the encoding name like:
Both ways should create a tokenizer that can be used with
gpt4.1model. Let me know if you have any question.