How to add the llama3 prompt format to inference.py & minimal_chat.py? Is there a guide or tutorial?

[minienglish1]

I'm new to writing llm scripts, so I don't understand how to correctly setup the prompt format.

I'm trying to use Llama-3-70B-Instruct-exl2 2.4bpw to summarize & extract key words from long captions, which will then be used by my sdxl fine-tuner. So I don't need full chat, just a simple 1 turn script.

I've modified inference.py & minimal_chat.py to use, and with a simple prompt it seems to work fine. But the llama3 instructions state: "This format has to be exactly reproduced for effective use."

Can anyone point me towards a guide or tutorial that can show me how to correctly configure the prompt format when using exllamav2?

from https://huggingface.co/blog/llama3

How to prompt Llama 3

The base models have no prompt format. Like other base models, they can be used to continue an input sequence with a plausible continuation or for zero-shot/few-shot inference. They are also a great foundation for fine-tuning your own use cases. The Instruct versions use the following conversation structure:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>

{{ user_msg_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{{ model_answer_1 }}<|eot_id|>

This format has to be exactly reproduced for effective use.

[turboderp]

There's many ways to do it, and I guess the most "normal" way would be to format the entire chat history each round as a single text string and tokenize it all. This would work well enough since the generator skips inference for any tokens that haven't changed when you call begin_stream_ex.

For the minimal example I chose to concatenate tokenized sequences instead. For Llama3, that could look something like this:

from exllamav2 import * from exllamav2.generator import * import sys, torch print("Loading model...") config = ExLlamaV2Config("/mnt/str/models/llama3-8b-instruct-exl2/4.0bpw/") model = ExLlamaV2(config) cache = ExLlamaV2Cache(model, lazy = True) model.load_autosplit(cache) tokenizer = ExLlamaV2Tokenizer(config) generator = ExLlamaV2StreamingGenerator(model, cache, tokenizer) generator.set_stop_conditions([tokenizer.single_id("<|eot_id|>")]) # <- Set the correct stop condition gen_settings = ExLlamaV2Sampler.Settings() system_prompt = "You are a duck." prompt = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n" # <- First round starts with BOS + system prompt prompt += system_prompt prompt += "<|eot_id|>" while True: print() instruction = input("User: ") print() print("Assistant: ", end = "") prompt += "<|start_header_id|>user<|end_header_id|>\n\n" # <- First prompt is system prompt plus this prompt += instruction prompt += "<|eot_id|>" prompt += "<|start_header_id|>assistant<|end_header_id|>\n\n" instruction_ids = tokenizer.encode(prompt, encode_special_tokens = True) # <- Make sure control tokens are treated as such context_ids = instruction_ids if generator.sequence_ids is None \ else torch.cat([generator.sequence_ids, instruction_ids], dim = -1) generator.begin_stream_ex(context_ids, gen_settings) while True: res = generator.stream_ex() if res["eos"]: break print(res["chunk"], end = "") sys.stdout.flush() # Note, when the loop above breaks, generator.sequence_ids contains the tokenized context plus all tokens # sampled so far, including whatever caused a stop condition (in this case, <|eot_id|>) even though it is # not returned by stream_ex() as part of the streamed response. print() prompt = "" # <- Start next round with empty string instead of BOS + system prompt

[minienglish1]

That makes sense. I kinda understood how it worked, but having a simple example helps a lot. I should be able to figure it out.

Appreciate the information.