DuckDB has an extensive benchmark suite. When making changes that have potential performance implications, it is important to run these benchmarks to detect potential performance regressions.

Getting Started

To build the benchmark suite, run the following command in the DuckDB repository:

BUILD_BENCHMARK=1 CORE_EXTENSIONS='tpch' make 

Listing Benchmarks

To list all available benchmarks, run:

build/release/benchmark/benchmark_runner --list 

Running Benchmarks

Running a Single Benchmark

To run a single benchmark, issue the following command:

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark 

The output will be printed to stdout in CSV format, in the following format:

name	run	timing benchmark/micro/nulls/no_nulls_addition.benchmark	1	0.121234 benchmark/micro/nulls/no_nulls_addition.benchmark	2	0.121702 benchmark/micro/nulls/no_nulls_addition.benchmark	3	0.122948 benchmark/micro/nulls/no_nulls_addition.benchmark	4	0.122534 benchmark/micro/nulls/no_nulls_addition.benchmark	5	0.124102 

You can also specify an output file using the --out flag. This will write only the timings (delimited by newlines) to that file.

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark --out=timings.out 

The output will contain the following:

0.182472 0.185027 0.184163 0.185281 0.182948 

Running Multiple Benchmark Using a Regular Expression

You can also use a regular expression to specify which benchmarks to run. Be careful of shell expansion of certain regex characters (e.g., * will likely be expanded by your shell, hence this requires proper quoting or escaping).

build/release/benchmark/benchmark_runner "benchmark/micro/nulls/.*" 

Running All Benchmarks

Not specifying any argument will run all benchmarks.

build/release/benchmark/benchmark_runner 

Other Options

The --info flag gives you some other information about the benchmark.

build/release/benchmark/benchmark_runner benchmark/micro/nulls/no_nulls_addition.benchmark --info 

display_name:NULL Addition (no nulls) group:micro subgroup:nulls 

The --query flag will print the query that is run by the benchmark.

SELECT min(i + 1) FROM integers; 

The --profile flag will output a query tree.