𝕏 Follow me on X • 🤗 Hugging Face • 💻 Blog • 📙 LLM Engineer's Handbook

Curated list of datasets and tools for post-training.

Data is the most valuable asset in LLM development. When building a high-quality dataset, we target the three following characteristics:

• Accuracy: Samples should be factually correct and relevant to their corresponding instructions. This can involve using solvers for math and unit tests for code.
• Diversity: You want to cover as many use cases as possible to make sure you're never out of distribution. High diversity is essential as it leads to better generalization.
• Complexity: Samples should be multi-turn, multilingual, well-written, and include step-by-step reasoning when relevant.

To ensure the quality of a dataset, it is essential to combine various techniques, such as manuals reviews, heuristics like rule-based filtering, and scoring via judge LLMs or reward models.

Once a model has been pre-trained on a next-token prediction task, Supervised Fine-Tuning (SFT) is used to turn it into an assistant capable of answering questions and following instructions. During SFT, models learn a chat template and are specialized in one or more domains.

General-purpose datasets offer balanced mixtures of different types of data, including chat, code, and math. These datasets can be used to create general-purpose models that can handle various types of queries.

LLMs often struggle with mathematical reasoning and formal logic, which has led to the creation of specialized datasets. These datasets can include systematic thinking and step-by-step reasoning.

Code is another challenging domain for LLMs. Code datasets, containing diverse programming language examples, are used to fine-tune LLMs and enhance their ability to understand, generate, and analyze code.

Instruction following corresponds to the ability to properly follow constraints in the user prompt, such as "write only two paragraphs", "write your answer in French", etc. Strong instruction-following capabilities is a must-have for modern LLMs.

Learning new languages "from scratch" is a pre-training task, but providing multilingual instruction samples is useful to boost performance in the languages of interest.

Function calling allows large language models (LLMs) to execute predefined functions with parameters inferred from user prompts, rather than generating standard text responses. This enables LLMs to seamlessly integrate with external systems, perform complex operations, and provide more accurate and contextually relevant responses.

Real-world conversations provide valuable insights into how people naturally interact with LLMs, helping us identify the most important use cases and understand typical usage patterns.

Unlike instruction data, preference datasets consist of chosen and rejected answers. Preference alignment is used to align LLM's answers with human preferences to adopt the desired style and values.

Tools listed in this section can help you evaluate, generate, and explore datasets. Start by aggregating available data from various sources (open-source or not) and applying filters like data deduplication and data quality. If the initial dataset is small or insufficient, consider synthetically generating additional data to fill the gap. Iteratively explore and refine the dataset by assessing model performance, identifying gaps, and collecting or generating data to address those shortcomings.

• Trafilatura: Python and command-line tool to gather text and metadata on the web. Used for the creation of RefinedWeb.
• Marker: Quickly convert PDFs to markdown text.

• Rule-based filtering: Remove samples based on a list of unwanted words, like refusals and "As an AI assistant" (example).
• SemHash: Fuzzy deduplication based on fast embedding generation with a distilled model.
• Argilla: Platform that allows you to manually filter and annotate datasets in a collaborative way.
• judges: Small library of LLM judges with various classifiers and graders (early development).

• Curator: Synthetic data generation tool that makes it easy to build pipelines around LLMs, use batching, and view data in progress.
• Distilabel: General-purpose framework that can generate and augment data (SFT, DPO) with techniques like UltraFeedback and DEITA.
• Augmentoolkit: Framework to convert raw text into datasets using open-source and closed-source models.
• Data Prep Kit: Framework for data preparation for both code and language, with modules in Python, Ray, and Spark, and a wide range of scale from laptops to data centers.

• Lilac: Tool for exploration, curation, and quality control of datasets.
• Nomic Atlas: Interact with instructed data to find insights and store embeddings.
• text-clustering: A framework from Huggingface for clustering textual data.
• Autolabel: Automatically label data using popular language models.

Special thanks to geronimi73, Bytes-Explorer, euclaise, RishabhMaheshwary, and ParagEkbote for their PRs.

Please let me know if a dataset is not properly credited.

• Wei-Lin Chiang et al, "Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality," 2023.
• Yihan Cao et al, "Instruction Mining: When Data Mining Meets Large Language Model Finetuning," 2023.
• Subhabrata Mukherjee et al, "Orca: Progressive Learning from Complex Explanation Traces of GPT-4," 2023.
• Chunting Zhou et al, "LIMA: Less Is More for Alignment," 2023.
• Suriya Gunasekar et al, "Textbooks Are All You Need," 2023.
• Lichang Chen et al, "AlpaGasus: Training A Better Alpaca with Fewer Data," 2024.
• Zheng Cai et al, "InternLM2 Technical Report," 2024.
• Lifan Yuan et al, "Advancing LLM Reasoning Generalists with Preference Trees," 2024.
• Wei Liu et al, "What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning," 2024.
• Xingyao Wang et al, "MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback," 2024.