Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

Kotlin multiplatform BPE tokenizer library for OpenAI models

High performance unsupervised text tokenization for Ruby

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

Byte-level byte pair encoding (BPE) in Haskell

R package for Byte Pair Encoding based on YouTokenToMe

Feature extraction from sequential data

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

Byte-Pair Encoding tokenizer for training large language models on huge datasets

Генерация новостных заголовков

Byte Pair Encoding (BPE)

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.

A Visualizer to check how BPE Tokenizer in an LLM Works

Auto summarization from BPE tokenization

Code for the publication of WWW'22

Modern Eager TensorFlow implementation of Attention Is All You Need

Lightweight, header-only Byte Pair Encoding (BPE) trainer in modern C++17. Produces HuggingFace-compatible vocabularies for transformers and integrates with Modern Text Tokenizer.