Search results

Feature: Improve performance by migrating to Rust

Status: Open.

#465 In openai/tiktoken;

· reneleonhardt opened on Nov 14, 2025

Add GPT-5.1 with the o200k_base encoding in <code>tiktoken/model.py</code>

Status: Open.

#464 In openai/tiktoken;

· eshaanpathak opened on Nov 13, 2025

Need clarity on GPT-5 tokenization

Status: Open.

#463 In openai/tiktoken;

· nitishpandey04 opened on Nov 13, 2025

Feature Request: Allow tiktoken to accept a custom HTTP client

Status: Open.

#462 In openai/tiktoken;

· GabrielVGS opened on Nov 12, 2025

bug: o200k_harmony duplicates special token id 200018

Status: Open.

#457 In openai/tiktoken;

· jinzhuer opened on Oct 13, 2025

0.12.0 no longer provides manylinux2014 wheels and breaks builds in AWS SAM Python 3.11 images(requires Rust compiler)

Status: Open.

#456 In openai/tiktoken;

· jotasixto opened on Oct 10, 2025

Interest in porting any of the implementation from TokenDagger? (With Possible PR)

Status: Open.

#441 In openai/tiktoken;

· Jeffrharr opened on Aug 26, 2025

Unable to Release Memory Used by Tiktoken Encoding After Setting to None and GC Collect

Status: Open.

#414 In openai/tiktoken;

· kimhyun5u opened on Jul 14, 2025

dead lock

Status: Open.

#413 In openai/tiktoken;

· tiandalu opened on Jul 11, 2025

Bug in buffer protocol implementation

Status: Open.

#405 In openai/tiktoken;

· mariosasko opened on Jun 1, 2025

Fail to Install on win-arm64

Status: Open.

#403 In openai/tiktoken;

· MrHinsh opened on May 31, 2025

RuntimeError(StackOverflow) with tiktoken.encoding_for_model('gpt-4').encode(txt, disallowed_special=())

Status: Open.

#400 In openai/tiktoken;

· sumitKpandey opened on May 6, 2025