Merge pull request microsoft#301 from nitya/feat/add-fine-tuning-chapter

Ch 18 - Fine Tuning [DRAFT - Work in Progress]

Author: koreyspace

.env.copy

@@ -4,6 +4,7 @@ AZURE_OPENAI_DEPLOYMENT='<add your deployment name here>'
 AZURE_OPENAI_KEY='<add your key here>'
 AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT='<add your deployment name here>'
 
+
 # OpenAI Configuration
 
 OPENAI_API_KEY='<add your OpenAI key here>'

18-fine-tuning/README.md

@@ -0,0 +1,84 @@
+# Fine-Tuning Your LLM
+
+Using large language models to build generative AI applications comes with new challenges. A key issue is ensuring response quality (accuracy and relevance) in content generated by the model for a given user request. In previous lessons, we discussed techniques like prompt engineering and retrieval-augmented generation that try to solve the problem by _modufying the prompt input_ to the existing model. 
+
+In today's lesson, we discuss a third technique, **fine-tuning**, which tries to address the challenge by _retraining the model itself_ with additional data. Let's dive into the details.
+
+## Learning Objectives
+
+This lesson introduces the concept of fine-tuning for pre-trained language models, explores the benefits and challenges of this approach, and provides guidance on when and how to use fine tuning to improve the performance of your generative AI models.
+
+By the end of this lesson, you should be able to answer the following questions:
+ - What is fine tuning for language models?
+ - When, and why, is fine tuning useful?
+ - How can I fine-tune a pre-trained model?
+ - What are the limitations of fine-tuning?
+
+Ready? Let's get started.
+
+## What is fine-tuning for language models?
+
+By definition, large language models are _pre-trained_ on large quantities of text sourced from diverse sources including the internet. As we've learned in previous lessons, we need techniques like _prompt engineering_ and _retrieval-augmented generation_ to improve the quality of the model's responses to the user's questions ("prompts").
+
+A popular prompt-engineering technique involves giving the model more guidance on what is expected in the response either by providing _instructions_ (exlicit guidance) or _giving it a few examples_ (implicit guidance). This is referred to as _few-shot learning_ but it has two limitations:
+ - Model token limits can restrict the number of examples you can give, and limit the effectiveness.
+ - Model token costs can make it expensive to add examples to every prompt, and limit flexibility.
+
+Fine-tuning is a common practice in machine learning systems where we take a pre-trained model and retrain it with new data to improve its performance on a specific task. In the context of language models, we can fine-tune the pre-trained model _with a curated set of examples for a given task or application domain_ to create a **custom model** that may be more accurate and relevant for that specific task or domain. A side-benefit of fine-tuning is that it can also reduce the number of examples needed for few-shot learning - reducing token usage and related costs.
+
+## When and why should we fine-tune models?
+
+In _this_ context, when we talk about fine-tuning, we are referring to **supervised** fine-tuning where the retraining is done by **adding new data** that was not part of the original training dataset. This is different from an unsupervised fine-tuning approach whwre the model is retrained on the original data, but with different hyperparameters.
+
+The key thing to remember is that fine-tuning is an advanced technique that requires a certain level of expertise to get the desired results. If done incorrecctly, it may not provide the expected improvements, and may even degrade the performance of the model for your targeted domain.
+
+So, before you learn "how" to fine-tune language models, you need to know "why" you should take this route, and "when" to start the process of fine-tuning. Start by asking yourself these questions:
+- **Use Case**: What is your _use case_ for fine-tuning? What aspect of the current pre-trained model do you want to improve upon?
+- **Alternatives**: Have you tried _other techniques_ to achieve the desired outcomes? Use them to create a baseline for comparison.
+ - Prompt engineering: Try techniques like few-shot prompting with examples of relevant prompt responses. Evaluate the quality of responses.
+ - Retrieval Augmented Generation: Try augmenting prompts with query results retrieved by searching your data. Evaluate the quality of responses.
+- **Costs**: Have you identified the costs for fine-tuning?
+ - Tunability - is the pre-trained model availalbe for fine-tuning?
+ - Effort - for preparing training data, evaluating & refining model.
+ - Compute - for running fine-tuning jobs, and deploying fine-tuned model
+ - Data - access to sufficient quality examples for fine-tuning impact
+- **Benefits**: Have you confirmed the benefits for fine-tuning?
+ - Quality - did fine-tuned model outperform baseline?
+ - Cost - does it reduce token usage by simplifying prompts?
+ - Extensibility - can you repurpose base model for new domains?
+
+By answering these questions, you should be able to decide if fine-tuning is the right approach for your use case. Ideally, the approach is valid only if the benefits outweigh the costs. Once you decide to proceed, it's time to think about _how_ you can fine tune the pre-trained model.
+
+Want to get more insights on the decision-making process? Watch [To fine-tune or not to fine-tune](https://www.youtube.com/watch?v=0Jo-z-MFxJs)
+
+## How can we fine-tune a pre-trained model?
+
+To fine-tune a pre-trained model, you need to have:
+ - a pre-trained model to fine-tune
+ - a dataset to use for fine-tuning
+ - a training environment to run the fine-tuning job
+ - a hosting environment to deploy fine-tuned model
+
+
+
+## Fine-Tuning In Action
+
+The following resources provide step-by-step tutorials to walk you through a real example using a selected model with a curated dataset. To work through these tutorials, you need an account on the specific provider, along with access to the relevant model and datasets.
+
+| Provider | Tutorial | Description |
+|----------|----------|-------------|
+| OpenAI | [How to fine-tune chat models](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_finetune_chat_models.ipynb) | Learn to fine-tune a `gpt-35-turbo` for a specific domain ("recipe assistant") by preparing training data, running the fine-tuning job, and using the fine-tuned model for inference. |
+| Azure Open AI | [GPT 3.5 Turbo fine-tuning tutorial](https://learn.microsoft.com/en-us/azure/ai-services/openai/tutorials/fine-tune?tabs=python-new%2Ccommand-line) | Learn to fine-tune a `gpt-35-turbo-0613` model **on Azure** by taking steps to create & upload training data, run the fine-tuning job. Deploy & use the new model. |
+| Hugging Face | [Fine-tuning LLMs with Hugging Face](https://www.philschmid.de/fine-tune-llms-in-2024-with-trl) | This blog post walks you fine-tuning an _open LLM_ (ex: `CodeLlama 7B`) using the [transformers](https://huggingface.co/docs/transformers/index) library & [Transformer Reinforcement Learning (TRL)](https://huggingface.co/docs/trl/index]) with open [datasets](https://huggingface.co/docs/datasets/index) on Hugging Face. |
+| | | |
+
+## Assignment
+Select one of the tutorials above and walk through them. _We may replicate a version of these tutorials in Jupyter Notebooks in this repo for reference only. Please use the original sources directly to get the latest versions_.
+
+## Great Work! Continue Your Learning.
+
+After completing this lesson, check out our [Generative AI Learning collection](https://aka.ms/genai-collection?WT.mc_id=academic-105485-koreyst) to continue leveling up your Generative AI knowledge!
+
+Congratulations!! You have completed the final lesson from the v2 series for this course! Don't stop learning and building. **Check out the [RESOURCES](README.md) page for a list of additional suggestions for just this topic.
+
+Our v1 series of lessons have also been updated with more assignments and concepts. So take a minute to refresh your knowledge - and please [share your questions and feedback](https://github.com/microsoft/generative-ai-for-beginners/issues) to help us improve these lessons for the community.

18-fine-tuning/RESOURCES.md

@@ -0,0 +1,38 @@
+# Resources For Self-Guided Learning
+
+The lesson was built using a number of core resources from OpenAI and Azure OpenAI as references for the terminology and tutorials. Here is a non-comprehenisve list, for your own self-guided learning journeys.
+
+## 1. Primary Resources
+
+| Title/Link | Description |
+|:---|:---|
+| [Fine-tuning with OpenAI Models](https://platform.openai.com/docs/guides/fine-tuning) | Fine-tuning improves on few-shot learning by training on many more examples than can fit in the prompt, saving you costs, improving response quality, and enabling lower-latency requests. **Get an overview of fine-tuning from OpenAI.**|
+| [What is Fine-Tuning with Azure Open AI?](https://learn.microsoft.com/azure/ai-services/openai/concepts/fine-tuning-considerations#what-is-fine-tuning-with-azure-openai) | Understand **what fine-tuning is (concept)**, why you should look at it (motivating problem), what data to use (training) and measuring the quality |
+| [Customize a model with fine-tuning](https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython&pivots=programming-language-studio#continuous-fine-tuning)| Azure OpenAI Service lets you tailor our models to your personal datasets using fine-tuning. Learn **how to fine-tune (process)** select models using Azure AI Studio, Python SDK or REST API.|
+| [Recommendations for LLM fine-tuning](https://learn.microsoft.com/ai/playbook/technology-guidance/generative-ai/working-with-llms/fine-tuning-recommend) |LLMs may not perform well on specific domains, tasks, or datasets, or may produce inaccurate or misleading outputs. **When should you consider fine-tuning** as a possible solution to this? |
+| [Continuous Fine Tuning](https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning?tabs=turbo%2Cpython&pivots=programming-language-studio#continuous-fine-tuning) | Continuous fine-tuning is the iterative process of selecting an already fine-tuned model as a base model and **fine-tuning it further** on new sets of training examples.|
+| [Fine-tuning and function calling](https://learn.microsoft.com/azure/ai-services/openai/how-to/fine-tuning-functions) | Fine-tuning your model **with function calling examples** can improve model output by getting more accurate and consistent outputs - with similarly-formatted responses & cost-savings|
+| [Fine-tuning Models: Azure OpenAI Guidance](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#fine-tuning-models) | Look up this table to understand **what models can be fine-tuned** in Azure OpenAI, and which regions these are available in. Look up their token limits and training data expiry dates if needed. |
+| [To Fine Tune or Not To Fine Tune? That is the Question](https://learn.microsoft.com/shows/ai-show/to-fine-tune-or-not-fine-tune-that-is-the-question) | This 30-min **Oct 2023** episode of the AI Show discusses benefits, drawbacks and practical insights that help you make this decision. |
+| [Getting Started With LLM Fine-Tuning](https://learn.microsoft.com/ai/playbook/technology-guidance/generative-ai/working-with-llms/fine-tuning) | This **AI Playbook** resource walks you through data requirements, formatting, hyperparameter fine-tuning and challenges/limitations you should know.|
+| **Tutorial**: [Azure OpenAI GPT3.5 Turbo Fine-Tuning](https://learn.microsoft.com/azure/ai-services/openai/tutorials/fine-tune?tabs=python%2Ccommand-line) | Learn to create a sample fine-tuning dataset, prepare for fine-tuning, create a fine-tuning job, and deploy the fine-tuned model on Azure.|
+| **Tutorial**: [Fine-tune a Llama 2 model in Azure AI Studio](https://learn.microsoft.com/azure/ai-studio/how-to/fine-tune-model-llama) | Azure AI Studio lets you tailor large language models to your personal datasets _using a UI-based workflow suitable for low-code developers_. See this example.|
+| **Tutorial**:[Fine-tune Hugging Face models for a single GPU on Azure](https://learn.microsoft.com/azure/databricks/machine-learning/train-model/huggingface/fine-tune-model) |This article describes how to fine-tune a Hugging Face model with the Hugging Face transformers library on a single GPU with Azure DataBricks + Hugging Face Trainer libraries |
+| **Training:** [Fine-tune a foundation model with Azure Machine Learning](https://learn.microsoft.com/en-us/training/modules/finetune-foundation-model-with-azure-machine-learning/) | The model catalog in Azure Machine Learning offers many open source models you can fine-tune for your specific task. Try this module is [from the AzureML Generative AI Learning Path](https://learn.microsoft.com/en-us/training/paths/work-with-generative-models-azure-machine-learning/)|
+**Tutorial:** [Azure OpenAI Fine-Tuning](https://docs.wandb.ai/guides/integrations/azure-openai-fine-tuning) |Fine-tuning GPT-3.5 or GPT-4 models on Microsoft Azure using W&B allows for detailed tracking and analysis of model performance. This guide extends the concepts from the OpenAI Fine-Tuning guide with specific steps and features for Azure OpenAI.|
+| | |
+
+
+## 2. Secondary Resources
+
+This section captures additional resources that are worth exploring, but that we did not have time to cover in this lesson. They may be covered in a future lesson, or as a secondary assignment option, at a later date. For now, use them to build your own expertise and knowledge around this topic.
+
+| Title/Link | Description |
+|:---|:---|
+| **OpenAI Cookbook**: [Data preparation and analysis for chat model fine-tuning](https://cookbook.openai.com/examples/chat_finetuning_data_prep)|This notebook serves as a tool to preprocess and analyze the chat dataset used for fine-tuning a chat model. It checks for format errors, provides basic statistics, and estimates token counts for fine-tuning costs. See: [Fine-tuning method for gpt-3.5-turbo](https://platform.openai.com/docs/guides/fine-tuning). |
+| **OpenAI Cookbook**: [Fine-Tuning for Retrieval Augmented Generation (RAG) with Qdrant](https://cookbook.openai.com/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant)|The aim of this notebook is to walk through a comprehensive example of how to fine-tune OpenAI models for Retrieval Augmented Generation (RAG). We will also be integrating Qdrant and Few-Shot Learning to boost model performance and reduce fabrications. |
+| **OpenAI Cookbook**: [Fine-tuning GPT with Weights & Biases](https://cookbook.openai.com/examples/third_party/gpt_finetuning_with_wandb) |Weights & Biases (W&B) is the AI developer platform, with tools for training models, fine-tuning models, and leveraging foundation models. Read their [OpenAI Fine-Tuning](https://docs.wandb.ai/guides/integrations/openai) guide first, then try the Cookbook exercise. |
+| **Community Tutorial** [Phinetuning 2.0](https://huggingface.co/blog/g-ronimo/phinetuning) - fine-tuning for Small Language Models| Meet [Phi-2](https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/), Microsoft’s new small model, remarkably powerful yet compact. This tutorial will guide you through fine-tuning Phi-2, demonstrating how to build a unique dataset and fine-tune model using QLoRA.|
+| **Hugging Face Tutorial** [How to Fine-Tune LLMs in 2024 with Hugging Face](https://www.philschmid.de/fine-tune-llms-in-2024-with-trl)|This blog post walks you thorugh how to fine-tune open LLMs using Hugging Face TRL, Transformers & datasets in 2024. You define a use case, setup a dev environment, prepare a dataset, fine tune the model, test-evaluate it, then deploy it to production.|
+| **Hugging Face: [AutoTrain Advanced](https://github.com/huggingface/autotrain-advanced)** | Brings faster and easier training and deployments of [state-of-the-art machine learning models](https://twitter.com/abhi1thakur/status/1755167674894557291). Repo has Colab-friendly tutorials with YouTube video guidance, for fine-tuning. **Reflects recent [local-first](https://twitter.com/abhi1thakur/status/1750828141805777057) update** . Read the [AutoTrain documentation](https://huggingface.co/autotrain)|
+| | |