Adds system benchmarks command

[marc-gr]

This adds the ability to define system (end to end) benchmarks at the package level.

• Refactor benchrunner to decouple reports presentation from pipeline benchmarks internals
• Add benchmark system command
• Collect indices, disk and ingest metrics from all nodes
• Implement docker input services
• Ability to generate data on the fly using the corpus generator
• Report to STDOUT a summary of the benchmark metrics
• Report to a file
• Add documentation
• Add test

Depends on elastic/package-spec#512
Closes #1164

TODO in the future(in no specific order)

• Collect metrics from elastic agents with system package
• Seed generator for consistent data
• Send benchmark runtime metrics to the ES Metricstore
• Add command to compare two benchmark runs
• Support more input service types (like system tests)
• Support benchmark scenarios defined outside of the package scope (for easily running scheduled benchmarks)
• Create benchmarks dashboard (maybe through a benchmark integration)

Example of usage:

With a benchmark scenario defined as

<package root>/_dev/benchmark/system/100mb-logs-benchmark.yml

--- description: Benchmark 100MiB of data ingested input: filestream vars: ~ data_stream.name: test data_stream.vars.paths: - "{{SERVICE_LOGS_DIR}}/corpus-*" warmup_time_period: 10s corpora.generator.size: 100MiB corpora.generator.template.path: ./100mb-logs-benchmark/template.log corpora.generator.config.path: ./100mb-logs-benchmark/config.yml corpora.generator.fields.path: ./100mb-logs-benchmark/fields.yml

We then run:

elastic-package benchmark system --benchmark 100mb-logs-benchmark -v 

To generate a benchmark report:

--- Benchmark results for package: system_benchmarks - START --- ╭─────────────────────────────────────────────────────╮ │ info │ ├──────────────┬──────────────────────────────────────┤ │ benchmark │ 100mb-logs-benchmark │ │ description │ Benchmark 100MiB of data ingested │ │ run ID │ d2960c04-0028-42c9-bafc-35e599563cb1 │ │ package │ system_benchmarks │ │ start ts (s) │ 1682320355 │ │ end ts (s) │ 1682320355 │ │ duration │ 2m3s │ ╰──────────────┴──────────────────────────────────────╯ ╭───────────────────────────────────────────────────────────────────────╮ │ parameters │ ├─────────────────────────────────┬─────────────────────────────────────┤ │ package version │ 999.999.999 │ │ input │ filestream │ │ data_stream.name │ test │ │ data_stream.vars.paths │ [/tmp/service_logs/corpus-*] │ │ warmup time period │ 10s │ │ benchmark time period │ 0s │ │ wait for data timeout │ 0s │ │ corpora.generator.size │ 100MiB │ │ corpora.generator.template.path │ ./100mb-logs-benchmark/template.log │ │ corpora.generator.template.raw │ │ │ corpora.generator.template.type │ │ │ corpora.generator.config.path │ ./100mb-logs-benchmark/config.yml │ │ corpora.generator.config.raw │ map[] │ │ corpora.generator.fields.path │ ./100mb-logs-benchmark/fields.yml │ │ corpora.generator.fields.raw │ map[] │ ╰─────────────────────────────────┴─────────────────────────────────────╯ ╭───────────────────────╮ │ cluster info │ ├───────┬───────────────┤ │ name │ elasticsearch │ │ nodes │ 1 │ ╰───────┴───────────────╯ ╭─────────────────────────────────────────────────────────────╮ │ data stream stats │ ├────────────────────────────┬────────────────────────────────┤ │ data stream │ logs-system_benchmarks.test-ep │ │ approx total docs ingested │ 410127 │ │ backing indices │ 1 │ │ store size bytes │ 136310570 │ │ maximum ts (ms) │ 1682320467448 │ ╰────────────────────────────┴────────────────────────────────╯ ╭───────────────────────────────────────╮ │ disk usage for index .ds-logs-system_ │ │ benchmarks.test-ep-2023.04.22-000001 │ │ (for all fields) │ ├──────────────────────────────┬────────┤ │ total │ 99.8mb │ │ inverted_index.total │ 31.3mb │ │ inverted_index.stored_fields │ 35.5mb │ │ inverted_index.doc_values │ 30mb │ │ inverted_index.points │ 2.8mb │ │ inverted_index.norms │ 0b │ │ inverted_index.term_vectors │ 0b │ │ inverted_index.knn_vectors │ 0b │ ╰──────────────────────────────┴────────╯ ╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ pipeline logs-system_benchmarks.test-999.999.999 stats in node Qa9ujRVfQuWhqEESdt6xnw │ ├───────────────────────────────────────────────┬───────────────────────────────────────────┤ │ grok () │ Count: 407819 | Failed: 0 | Time: 16.615s │ │ user_agent () │ Count: 407819 | Failed: 0 | Time: 768ms │ │ pipeline (logs-system_benchmarks.test@custom) │ Count: 407819 | Failed: 0 | Time: 59ms │ ╰───────────────────────────────────────────────┴───────────────────────────────────────────╯ --- Benchmark results for package: system_benchmarks - END --- Done 

How to run this example locally:

$ make install $ elastic-package stack up -v -d $ eval "$(elastic-package stack shellinit)" $ cd test/packages/benchmarks/system_benchmark/ $ elastic-package build $ elastic-package install $ elastic-package benchmark system --benchmark 20mb-logs-benchmark -v 

[ruflin]

Is it by design that we don't use here names "similar" to ECS like source.ip? I'm aware this is Schema A so not ECS yet at the same time using ECS could make it easy to read (if possible).

This is just an example for a benchmark tests, I'm mostly worried it sets a precedence on how it should be done.

[marc-gr]

In this case since this represents a Schema A (as it comes from the source), I thought it made more sense to let the field names represent what they are in the original logs, since not always there will even be an ECS field to represent the original data (as it comes after transformations, compositions from several fields, etc.). Even though I do not have a strong opinion on this one I think in this case it might not be too troublesome tbh.

[ruflin]

Testing this with the commands provided above and also did some changes to the corpora size for testing. elastic-package install is missing in the command list above to make sure the package is setup, but that is only a testing issue.

Overall, this looks great and I think we should get this rather soonish. I did not review the code on my end only if the command works as expected. The following I noticed: It tells me in the report the corpora was generated under [/tmp/service_logs/corpus-*] but it seems it is generated under /Users/ruflin/.elastic-package/tmp/service_logs (relative path). It seems it also does not clean up the old corpora, so if run it again, it will also ingest the previous ones?

[mrodm]

LGTM
Just a note about it would be needed to re-generate the README, there is still a reference to pipeline-bench

Pending CI to be successful run, there are some lint errors related to the error messages:
https://buildkite.com/elastic/elastic-package/builds/968#0188aec7-c0b5-48ca-a612-9c62fe86407a/188-330

[marc-gr]

All comments in the PR are addressed now, it relies on some pending spec changed added in elastic/package-spec#526

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 223e1ac

History

• 💚 Build #1013 succeeded 2538ae5
• 💔 Build #1005 failed e6cce98
• 💔 Build #1003 failed d226d8d
• 💔 Build #1000 failed 44d40e7
• 💔 Build #968 failed fc37e66
• 💔 Build #965 failed 94f8fd4

cc @marc-gr

[mrodm]

Thanks for addressing all the comments!
LGTM

[jsoriano]

@marc-gr it looks like this field is not available in old versions of Kibana, it also seems that it is not used here. Was it needed?

Integrations CI is failing for something related to this field elastic/integrations#6673.

Trying to fix it by adding omitempty in #1320.

cc @mrodm