Is your fine-tuned LLM any good?

Over the past few weeks, I've been diving deep into fine-tuning Large Language Models (LLMs) for various applications, with a particular focus on creating detailed summaries from multiple sources. Throughout this process, we've leveraged tools like weights and biases to track our learning rates and optimize our results. However, a crucial question remains: How can we objectively determine if our fine-tuned model is producing high-quality outputs?
While human evaluation is always an option, it's time-consuming and often impractical for large-scale assessments. This is where ROUGE (Recall-Oriented Understudy for Gisting Evaluation) comes into play, offering a quantitative approach to evaluating our fine-tuned models.

ROUGE calculates the overlap between the generated text and the reference text(s) using various methods. The most common ROUGE metrics include:

ROUGE-N: Measures the overlap of n-grams (contiguous sequences of n words) between the generated and reference texts.
ROUGE-L: Computes the longest common subsequence (LCS) between the generated and reference texts.
ROUGE-W: A weighted version of ROUGE-L that gives more importance to consecutive matches.

What do the different metrics mean?

ROUGE-1: Overlap of unigrams (single words)
ROUGE-2: Overlap of bigrams (two-word sequences)
ROUGE-L: Longest common subsequence
ROUGE-L sum: Variant of ROUGE-L computed over the entire summary

Each metric is typically reported as a score between 0 and 1, where higher values indicate better performance

Here is some code you can use for evaluting the models. Note in this example I am loading both models into a single GPU and comparing the results at the same time

import multiprocessing import time from transformers import AutoModelForCausalLM, AutoTokenizer from evaluate import load original_model_name ="" finetuned_model = "" def load_model(model_name: str, device: str): model = AutoModelForCausalLM.from_pretrained( model_name, return_dict=True, load_in_8bit=True, device_map={"":device}, trust_remote_code=True, ) tokenizer = AutoTokenizer.from_pretrained(model_name) tokenizer.pad_token = tokenizer.eos_token return model, tokenizer def inference(model, tokenizer, prompt: str): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) outputs = model.generate(**inputs, max_new_tokens=200, temperature=1.0) return tokenizer.decode(outputs[0], skip_special_tokens=True) def process_task(task_queue, result_queue): original_model, original_tokenizer = load_model(original_model_name, device="cuda:0") fine_tuned_model, fine_tuned_tokenizer = load_model(finetuned_model, device="cuda:0") rouge = load('rouge') while True: task = task_queue.get() if task is None: break prompt, reference = task start = time.time() original_summary = inference(original_model, original_tokenizer, prompt) fine_tuned_summary = inference(fine_tuned_model, fine_tuned_tokenizer, prompt) print(f"Completed inference in {time.time() - start}") original_scores = rouge.compute(predictions=[original_summary], references=[reference]) fine_tuned_scores = rouge.compute(predictions=[fine_tuned_summary], references=[reference]) result_queue.put((original_scores, fine_tuned_scores)) def main(): task_queue = multiprocessing.Queue() result_queue = multiprocessing.Queue() prompt = "Your prompt here" reference = "Your reference summary here" process = multiprocessing.Process(target=process_task, args=(task_queue, result_queue)) process.start() start = time.time() # Run 3 times for _ in range(3): task_queue.put((prompt, reference)) results = [] for _ in range(3): result = result_queue.get() results.append(result) # Signal the process to terminate task_queue.put(None) process.join() end = time.time() print(f"Total time: {end - start}") # Print ROUGE scores for i, (original_scores, fine_tuned_scores) in enumerate(results): print(f"Run {i+1}:") print("Original model scores:") print(original_scores) print("Fine-tuned model scores:") print(fine_tuned_scores) print() if __name__ == "__main__": multiprocessing.set_start_method("spawn") main() 

To illustrate the effectiveness of ROUGE in evaluating fine-tuned models, here are the results from a recent run:
Fine-tuned model:

rouge1: 0.1775
rouge2: 0.0271
rougeL: 0.1148
rougeLsum: 0.1148

Original model:

rouge1: 0.0780
rouge2: 0.0228
rougeL: 0.0543
rougeLsum: 0.0598

As we can see, the fine-tuned model shows significant improvements across all ROUGE metrics compared to the original model. This quantitative assessment provides concrete evidence that our fine-tuning process has indeed enhanced the model's summarization capabilities.
While ROUGE scores shouldn't be the sole criterion for evaluating LLMs, they offer a valuable, objective measure of improvement. By incorporating ROUGE into our evaluation pipeline, we can more efficiently iterate on our fine-tuning process and confidently assess the quality of our models.