Skip to content

Add GPT 4.1 to Tiktoken Tokenizer #7450

Closed

Description

@fanyang-mono

I would like to see GPT 4.1 being added to Tiktoken Tokenizer.

Activity

  1. stephentoub commented on Apr 30, 2025

    @stephentoub
    Member

    @tarekgh, looks like OpenAI stated yesterday that gpt-4.1 uses the same tokenizer as 4o.
    https://community.openai.com/t/whats-the-tokenization-algorithm-gpt-4-1-uses/1245758/2

  2. tarekgh commented on Apr 30, 2025

    @tarekgh
    Member

    @stephentoub I was waiting for OpenAI to officially add it to their tokenizer library openai/tiktoken#395.

  3. added this to the ML.NET 5.0 milestone on Apr 30, 2025
  4. self-assigned this
    on Apr 30, 2025
  5. fanyang-mono commented on May 6, 2025

    @fanyang-mono
    MemberAuthor

    @tarekgh Thanks for adding the support. I wonder when this will be available to be consumed?

  6. tarekgh commented on May 6, 2025

    @tarekgh
    Member

    @fanyang-mono the change will be included in our next preview (hopefully in 10 days or so). But you don't have to be blocked on that as you can still create the tokenizer using other compatible model name. Like

    Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4o");

    or you can create it from the encoding name like:

    Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("o200k_base");
    

    Both ways should create a tokenizer that can be used with gpt4.1 model. Let me know if you have any question.

  7. locked and limited conversation to collaborators on Jun 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions