|
85 | 85 | "\n", |
86 | 86 | "LLMs are extremely large in size (parameters in the order of billions). Full fine-tuning (which updates all the parameters in the model) is not required for most applications because typical fine-tuning datasets are relatively much smaller than the pre-training datasets.\n", |
87 | 87 | "\n", |
88 | | - "[Low Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685){:.external} is a fine-tuning technique which greatly reduces the number of trainable parameters for downstream tasks by freezing the weights of the model and inserting a smaller number of new weights into the model. This makes training with LoRA much faster and more memory-efficient, and produces smaller model weights (a few hundred MBs), all while maintaining the quality of the model outputs.\n", |
| 88 | + "[Low Rank Adaptation (LoRA)](https://arxiv.org/abs/2106.09685) is a fine-tuning technique which greatly reduces the number of trainable parameters for downstream tasks by freezing the weights of the model and inserting a smaller number of new weights into the model. This makes training with LoRA much faster and more memory-efficient, and produces smaller model weights (a few hundred MBs), all while maintaining the quality of the model outputs.\n", |
89 | 89 | "\n", |
90 | | - "This tutorial walks you through using KerasNLP to perform LoRA fine-tuning on a Gemma 2B model using the [Databricks Dolly 15k dataset](https://huggingface.co/datasets/databricks/databricks-dolly-15k){:.external}. This dataset contains 15,000 high-quality human-generated prompt / response pairs specifically designed for fine-tuning LLMs." |
| 90 | + "This tutorial walks you through using KerasNLP to perform LoRA fine-tuning on a Gemma 2B model using the [Databricks Dolly 15k dataset](https://huggingface.co/datasets/databricks/databricks-dolly-15k). This dataset contains 15,000 high-quality human-generated prompt / response pairs specifically designed for fine-tuning LLMs." |
91 | 91 | ] |
92 | 92 | }, |
93 | 93 | { |
|
109 | 109 | "\n", |
110 | 110 | "To complete this tutorial, you will first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", |
111 | 111 | "\n", |
112 | | - "* Get access to Gemma on [kaggle.com](https://kaggle.com){:.external}.\n", |
| 112 | + "* Get access to Gemma on [kaggle.com](https://kaggle.com).\n", |
113 | 113 | "* Select a Colab runtime with sufficient resources to run\n", |
114 | 114 | " the Gemma 2B model.\n", |
115 | 115 | "* Generate and configure a Kaggle username and API key.\n", |
|
333 | 333 | "source": [ |
334 | 334 | "## Load Model\n", |
335 | 335 | "\n", |
336 | | - "KerasNLP provides implementations of many popular [model architectures](https://keras.io/api/keras_nlp/models/){:.external}. In this tutorial, you'll create a model using `GemmaCausalLM`, an end-to-end Gemma model for causal language modeling. A causal language model predicts the next token based on previous tokens.\n", |
| 336 | + "KerasNLP provides implementations of many popular [model architectures](https://keras.io/api/keras_nlp/models/). In this tutorial, you'll create a model using `GemmaCausalLM`, an end-to-end Gemma model for causal language modeling. A causal language model predicts the next token based on previous tokens.\n", |
337 | 337 | "\n", |
338 | 338 | "Create the model using the `from_preset` method:" |
339 | 339 | ] |
|
1001 | 1001 | "\n", |
1002 | 1002 | "* Learn how to [generate text with a Gemma model](https://ai.google.dev/gemma/docs/get_started).\n", |
1003 | 1003 | "* Learn how to perform [distributed fine-tuning and inference on a Gemma model](https://ai.google.dev/gemma/docs/distributed_tuning).\n", |
1004 | | - "* Learn how to [use Gemma open models with Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/open-models/use-gemma){:.external}.\n", |
1005 | | - "* Learn how to [fine-tune Gemma using KerasNLP and deploy to Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_gemma_kerasnlp_to_vertexai.ipynb){:.external}." |
| 1004 | + "* Learn how to [use Gemma open models with Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/open-models/use-gemma).\n", |
| 1005 | + "* Learn how to [fine-tune Gemma using KerasNLP and deploy to Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_gemma_kerasnlp_to_vertexai.ipynb)." |
1006 | 1006 | ] |
1007 | 1007 | } |
1008 | 1008 | ], |
|
0 commit comments