Tokenizers#

NeMo 1.0 (Previous Release)#

In NeMo 1.0, tokenizers were configured in the tokenizer section of the YAML configuration file.

NeMo 2.0 (New Release)#

In NeMo 2.0, tokenizers can be initialized directly in Python. get_nmt_tokenizer is a utility function used in NeMo to instantiate many of the common tokenizers used for llm and multimodal training. For example, the following code will construct a GPT2BPETokenizer:

from nemo.collections.nlp.modules.common.tokenizer_utils import get_nmt_tokenizer tokenizer = get_nmt_tokenizer( library="megatron", model_name="GPT2BPETokenizer", vocab_file="/path/to/vocab", merges_file="/path/to/merges", )

The following will construct a SentencePiece tokenizer:

from nemo.collections.nlp.modules.common.tokenizer_utils import get_nmt_tokenizer tokenizer = get_nmt_tokenizer( library="sentencepiece", tokenizer_model='/path/to/sentencepiece/model' )

The following will construct a Hugging Face tokenizer:

from nemo.collections.nlp.modules.common.tokenizer_utils import get_nmt_tokenizer tokenizer = get_nmt_tokenizer( library="huggingface", model_name='nvidia/Minitron-4B-Base', use_fast=True, )

Refer to the get_nmt_tokenizer code for a full list of supported arguments.

To set up the tokenizer using nemo_run, use the following code:

import nemo_run as run from nemo.collections.common.tokenizers import SentencePieceTokenizer from nemo.collections.common.tokenizers.huggingface.auto_tokenizer import AutoTokenizer # Set up Sentence Piece tokenizer tokenizer = run.Config(SentencePieceTokenizer, model_path="/path/to/tokenizer.model") # Set up Hugging Face tokenizer tokenizer = run.Config(AutoTokenizer, pretrained_model_name="/path/to/tokenizer/model")

Refer to the SentencePieceTokenizer or AutoTokenizer code for a full list of supported arguments.

To change the tokenizer path for model recipe, use the following code:

from nemo.collections import llm recipe = partial(llm.llama3_8b)() # Change path for Hugging Face tokenizer recipe.data.tokenizer.pretrained_model_name = "/path/to/tokenizer/model" # Change tokenizer path for Sentence Piece tokenizer recipe.data.tokenizer.model_path = "/path/to/tokenizer.model"

Basic NeMo 2.0 recipes can contain predefined tokenizers. Visit this page to see an example of setting up the tokenizer in the recipe.

On this page