ir_datasets: C4A version of Google's C4 dataset, which consists of articles crawled form the web.
The "en-noclean" train subset of the corpus, consisting of ~1B documents written in English. Document IDs are assigned as proposed by the TREC Health Misinformation 2021 track.
Language: en
Examples:
import ir_datasets dataset = ir_datasets.load("c4/en-noclean-tr") for doc in dataset.docs_iter(): doc # namedtuple<doc_id, text, url, timestamp> You can find more details about the Python API here.
ir_datasets export c4/en-noclean-tr docs [doc_id] [text] [url] [timestamp] ... You can find more details about the CLI here.
import pyterrier as pt pt.init() dataset = pt.get_dataset('irds:c4/en-noclean-tr') # Index c4/en-noclean-tr indexer = pt.IterDictIndexer('./indices/c4_en-noclean-tr', meta={"docno": 41}) index_ref = indexer.index(dataset.get_corpus_iter(), fields=['text', 'url', 'timestamp']) You can find more details about PyTerrier indexing here.
from datamaestro import prepare_dataset dataset = prepare_dataset('irds.c4.en-noclean-tr') for doc in dataset.iter_documents(): print(doc) # an AdhocDocumentStore break This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore
{ "docs": { "count": 1063805381, "fields": { "doc_id": { "max_len": 41, "common_prefix": "en.noclean.c4-train.0" } } } } The TREC Health Misinformation 2021 track.
Language: en
Examples:
import ir_datasets dataset = ir_datasets.load("c4/en-noclean-tr/trec-misinfo-2021") for query in dataset.queries_iter(): query # namedtuple<query_id, text, description, narrative, disclaimer, stance, evidence> You can find more details about the Python API here.
ir_datasets export c4/en-noclean-tr/trec-misinfo-2021 queries [query_id] [text] [description] [narrative] [disclaimer] [stance] [evidence] ... You can find more details about the CLI here.
import pyterrier as pt pt.init() dataset = pt.get_dataset('irds:c4/en-noclean-tr/trec-misinfo-2021') index_ref = pt.IndexRef.of('./indices/c4_en-noclean-tr') # assumes you have already built an index pipeline = pt.BatchRetrieve(index_ref, wmodel='BM25') # (optionally other pipeline components) pipeline(dataset.get_topics('text')) You can find more details about PyTerrier retrieval here.
from datamaestro import prepare_dataset topics = prepare_dataset('irds.c4.en-noclean-tr.trec-misinfo-2021.queries') # AdhocTopics for topic in topics.iter(): print(topic) # An AdhocTopic This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocTopics.
Inherits docs from c4/en-noclean-tr
Language: en
Examples:
import ir_datasets dataset = ir_datasets.load("c4/en-noclean-tr/trec-misinfo-2021") for doc in dataset.docs_iter(): doc # namedtuple<doc_id, text, url, timestamp> You can find more details about the Python API here.
ir_datasets export c4/en-noclean-tr/trec-misinfo-2021 docs [doc_id] [text] [url] [timestamp] ... You can find more details about the CLI here.
import pyterrier as pt pt.init() dataset = pt.get_dataset('irds:c4/en-noclean-tr/trec-misinfo-2021') # Index c4/en-noclean-tr indexer = pt.IterDictIndexer('./indices/c4_en-noclean-tr', meta={"docno": 41}) index_ref = indexer.index(dataset.get_corpus_iter(), fields=['text', 'url', 'timestamp']) You can find more details about PyTerrier indexing here.
from datamaestro import prepare_dataset dataset = prepare_dataset('irds.c4.en-noclean-tr.trec-misinfo-2021') for doc in dataset.iter_documents(): print(doc) # an AdhocDocumentStore break This examples requires that experimaestro-ir be installed. For more information about the returned object, see the documentation about AdhocDocumentStore
{ "docs": { "count": 1063805381, "fields": { "doc_id": { "max_len": 41, "common_prefix": "en.noclean.c4-train.0" } } }, "queries": { "count": 50 } }