Add LogsDB option to route on sort fields

[kkrik-es]

This is a storage optimization, improving codec efficiency. It requires configuring more than one sort fields (excluding @timestamp) to reduce the likelihood for hotspots in shard routing.

Related to #109334

[elasticsearchmachine]

Hi @kkrik-es, I've created a changelog YAML for you.

[kkrik-es]

EDIT: ignore this, more results below.

results.txt
ESBench results look really promising:

| Cumulative indexing time of primary shards | 1263.28 | 591.821 | -671.462 | min | -53.15% | | Dataset size | 35.842 | 14.9673 | -20.8747 | GB | -58.24% | | Store size | 35.842 | 14.9673 | -20.8747 | GB | -58.24% | 

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

ESBench results look really promising:

There was an issue with the run above, there were many indexing errors due to missing sort field values. Updated results (sortin on [host.name, host.os.name, @timestamp):

| Cumulative indexing time of primary shards | 1263.28 | 1158.67 | -104.609 | min | -8.28% | | Dataset size | 35.842 | 29.9704 | -5.87159 | GB | -16.38% | | Store size | 35.842 | 29.9704 | -5.87159 | GB | -16.38% | | Segment count | 1052 | 687 | -365 | | -34.70% | 

Smaller but still sizable wins, esp on indexing time.

[martijnvg]

LGTM 🎉

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116687

[kkrik-es]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation