Skip to content

Pull requests: openai/tiktoken

New pull request New
34 Open 103 Closed
34 Open 103 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Upgrade GitHub Actions for Node 24 compatibility
#472 opened Dec 16, 2025 by salmanmkc Loading…
feat: add support for custom HTTP client in encoding functions
#470 opened Dec 9, 2025 by GabrielVGS Loading…
1
Add GPT-5.1 support with o200k_base encoding
#468 opened Nov 21, 2025 by magkue Loading…
1
Replacing fancy-regex for faster encode
#460 opened Oct 31, 2025 by MadMax129 Loading…
1
fix: remove conflicting <|reserved_200018|>; add token-id uniqueness …
#458 opened Oct 13, 2025 by jinzhuer Loading…
1
1
Loosened Encoding.encode*_batch's input types
#455 opened Oct 9, 2025 by jamesbraza Loading…
Refactor encoding name resolution to improve model prefix matching logic
#452 opened Sep 30, 2025 by hrishabhn Loading…
Optimize _byte_pair_merge to o(m log n) using heap-based candidate selection
#442 opened Aug 27, 2025 by ArchishmanSengupta Loading…
1
Swift Binding Generation via UniFFI
#431 opened Aug 14, 2025 by narner Loading…
Add support for building TikToken WoA
#412 opened Jul 4, 2025 by Greenie0701 Loading…
1
Update load.py
#399 opened Apr 23, 2025 by Shivam-Singh-Dev Loading…
Make decoder and sorted_token_bytes re-use existing memory - Unit Tests
#384 opened Feb 19, 2025 by CodeBeaverAI Loading…
In-code documentation update
#383 opened Feb 17, 2025 by louisbrulenaudet Loading…
1
Option to use TIKTOKEN_BPE_HOST environment variable for configurable BPE host URL
#357 opened Nov 4, 2024 by AitBaali-Hamza-bcg Loading…
2
make decoder and sorted_token_bytes re-use existing memory
#352 opened Oct 17, 2024 by tmm1 Loading…
use HashSet from rustc-hash too
#351 opened Oct 17, 2024 by tmm1 Loading…
Uses Regex instead of fancy-regex - 6x speedup
#331 opened Aug 5, 2024 by Majdoddin Loading…
3
Cache for Encoding - Runtime Boosted by 12%
#319 opened Jul 10, 2024 by Majdoddin Loading…
Add Terminal-Based Visualization Tool for Tokenized Data Points in Tiktoken Tokenizer
#314 opened Jun 18, 2024 by LVivona Loading…
1
Remove dependency on requests
#271 opened Mar 26, 2024 by tristan-jl Loading…
2
Optimize byte pair merge for really big tokens (40x faster for a 2500 token word)
#239 opened Jan 15, 2024 by l0rinc Loading…
28
Support enterprise network support for self-hosted encodings (UPDATED v0.6.0)
#238 opened Jan 12, 2024 by blaney83 Loading…
1
0) Add the jtokkit test suite examples to validate the cl100k_base, p50k_base & r50k_base encodings
#237 opened Jan 6, 2024 by l0rinc Loading…
1
feat: Add option TIKTOKEN_FORCE_CACHE
#170 opened Jul 28, 2023 by Kontrolix Loading…
4
✨ feat: Split code rust, for run as python lib and rust lib
#167 opened Jul 18, 2023 by Miuler Loading…
1
5
ProTip! no:milestone will show everything without a milestone.