Configure index sorting through index settings for logsdb

[kkrik-es]

Indexes configured with logsdb mode currently use sorting on [host.name, @timestamp] by default, with keyword host.name getting injected if not present. This can lead to problems, if e.g. host is a leaf field, while sorting on an empty host.name is wasteful.

To fix this, the logsdb index settings provider is extended to detect when field host.name is present and trigger sorting on it (in addition to @timestamp) by injecting an index setting that is later used in IndexSortConfig during IndexSettings initialization. Otherwise, logsdb indexes are only sorted on @timestamp by default.

The downside is that the provider may need to inspect mappings through MapperService more often, a costly operation. The impact will be monitored in nightlies, once this is submitted, and logic will be further optimized as needed.

Fixes #118686

[martijnvg]

I think introducing a setting that is set by index provider when we can index sort on hostname is more robust.

I left a few comments, otherwise looks good!

[martijnvg]

Let's keep this TODO here. After #119072 has been merged, let's try to optimize this in order to address the elastic/elasticsearch-benchmarks#2232 regression.

[kkrik-es]

Can we optimize it, though? With this change, we still need to check dynamic templates etc for the presence of host.name.

[martijnvg]

I think we should add Property.ServerlessPublic also here?

[kkrik-es]

Iiuc Property.ServerlessPublic is used to allow users to set this. This setting should never be set by users, it's only introduced to have the logsdb settings provider control the sort config. Do I understand this correctly?

[martijnvg]

Maybe in that case we should add Property.PrivateIndex? I think this would ensure that only we can use it here.

[kkrik-es]

Good call, done.

[martijnvg]

I didn't realize the fact that more integrations are relying on dynamic mappings for host.name field. I think in that case we do want to inject the host.name field, so that we can configure host.name as primary index sort field. Then I think for this PR, we want to just check mappings for cases when injecting host.name field leads to mapping conflicts? I think just check that host field isn't a leaf is sufficient? (in case subobjects is enabled) If that is the case we then we should just sort by timestamp.

[felixbarny]

Then I think for this PR, we want to just check mappings for cases when injecting host.name field leads to mapping conflicts?

Makes sense to me.

I think just check that host field isn't a leaf is sufficient? (in case subobjects is enabled)

Do we also need to check if host.name is already defined (possibly with a different mapping) and avoid injecting it in that case? Whether or not we can sort on that host.name mapping depends on how it's mapped (only boolean, numeric, date and keyword fields with doc_values can be sorted).

[kkrik-es]

Do we also need to check if host.name is already defined (possibly with a different mapping) and avoid injecting it in that case? Whether or not we can sort on that host.name mapping depends on how it's mapped (only boolean, numeric, date and keyword fields with doc_values can be sorted).

Let me try to incorporate all this in the PR and add tests.

[martijnvg]

I think we can try to merge in the static host.name mapping using mapper service and check if it succeeds. If it does succeed, then we should sort on host.name and timestamp otherwise just on timestamp.

[felixbarny]

Will the merge succeed if the users' mapping also maps host.name as a keyword field but disables doc_values? I guess in this case, we should only sort by timestamp.

[martijnvg]

I guess in this case, we should only sort by timestamp.

Good point. We should check whether it has doc values and otherwise fallback to index sorting only by timestamp.

[martijnvg]

LGTM

[martijnvg]

Maybe in that case we should add Property.PrivateIndex? I think this would ensure that only we can use it here.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 118968

[kkrik-es]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Questions ?

Please refer to the Backport tool documentation