Benchmarking dvc with pytest-benchmark.
Visit bench.dvc.org
Trigger a dispatch workflow with desired dataset and revisions and see results in bench.dvc.org/run_ID_ATTEMPT.html, where ID is github.run_id and ATTEMPT is github.run_attempt. For example, for https://github.com/iterative/dvc-bench/actions/runs/7119039172/attempts/2 it would be http://bench.dvc.org/run_7119039172_2.html
$ uv pip install -r requirements.txt $ dvc pull # optional, otherwise will pull datasets dynamically $ pytest --pyargs dvc.testing.benchmarks$ pytest --pyargs dvc.testing.benchmarks.cli.commands.test_add$ pytest -h ... --dataset=DATASET Dataset name to use in tests (e.g. tiny/small/large/mnist/etc) --dvc-bin=DVC_BIN Path to dvc binary --dvc-revs=DVC_REVS Comma-separated list of DVC revisions to test (overrides `--dvc-bin`) --dvc-repo=DVC_GIT_REPO Path or url to dvc git repo --dvc-bench-repo=DVC_BENCH_GIT_REPO Path or url to dvc-bench git repo (for loading benchmark dataset) --dvc-install-deps=DVC_INSTALL_DEPS Comma-separated list of DVC installation packages --project-rev=PROJECT_REV Project revision to test --project-repo=PROJECT_GIT_REPO Path or url to dvc project ... $ pytest-benchmark compare --histogram histograms/ --group-by name --sort name --csv results.csv and if you want beautiful plots:
$ dvc repro $ dvc plots show Benchmark test definitions are now part of dvc.testing.