Loaders for various machine learning datasets for testing and example scripts. Previously in thinc.extra.datasets.

The package can be installed via pip:

pip install ml-datasets

Loaders can be imported directly or used via their string name (which is useful if they're set via command line arguments). Some loaders may take arguments – see the source for details.

# Import directly from ml_datasets import imdb train_data, dev_data = imdb()

# Load via registry from ml_datasets import loaders imdb_loader = loaders.get("imdb") train_data, dev_data = imdb_loader()

Each instance contains the text of a movie review, and a sentiment expressed as 0 or 1.

train_data, dev_data = ml_datasets.imdb() for text, annot in train_data[0:5]: print(f"Review: {text}") print(f"Sentiment: {annot}")

• Download URL: http://ai.stanford.edu/~amaas/data/sentiment/
• Citation: Andrew L. Maas et al., 2011

Each instance contains an ontological description, and a classification into one of the 14 distinct labels.

train_data, dev_data = ml_datasets.dbpedia() for text, annot in train_data[0:5]: print(f"Text: {text}") print(f"Category: {annot}")

• Download URL: Via fast.ai
• Original citation: Xiang Zhang et al., 2015

Each instance contains a movie description, and a classification into a list of appropriate genres.

train_data, dev_data = ml_datasets.cmu() for text, annot in train_data[0:5]: print(f"Text: {text}") print(f"Genres: {annot}")

• Download URL: http://www.cs.cmu.edu/~ark/personas/
• Original citation: David Bamman et al., 2013

train_data, dev_data = ml_datasets.quora_questions() for questions, annot in train_data[0:50]: q1, q2 = questions print(f"Question 1: {q1}") print(f"Question 2: {q2}") print(f"Similarity: {annot}")

Each instance contains two quora questions, and a label indicating whether or not they are duplicates (0: no, 1: yes). The ground-truth labels contain some amount of noise: they are not guaranteed to be perfect.

• Download URL: http://qim.fs.quoracdn.net/quora_duplicate_questions.tsv
• Original citation: Kornél Csernai et al., 2017

Loaders can be registered externally using the loaders registry as a decorator. For example:

@ml_datasets.loaders("my_custom_loader") def my_custom_loader(): return load_some_data() assert "my_custom_loader" in ml_datasets.loaders