Add selector syntax to index expressions

[jbaiera]

In order to make adoption of failure stores simpler for all users, we are introducing a new syntactical feature to index expression resolution: The selector.

Selectors, denoted with a :: followed by a recognized suffix will allow users to specify which component of an index abstraction they would like to operate on within an API call. In this case, an index abstraction is a concrete index, data stream, or alias; Any abstraction that can be resolved to a set of indices/shards. We define a component of an index abstraction to be some searchable unit of the index abstraction.

To start, we will support two components: data and failures. Concrete indices are their own data components, while the data component for index aliases are all of the indices contained therein. For data streams, the data component corresponds to their backing indices. Data stream aliases mirror this, treating all backing indices of the data streams they correspond to as their data component.

The failure component is only supported by data streams and data stream aliases. The failure component of these abstractions refer to the data streams' failure stores. Indices and index aliases do not have a failure component.

In order to select which components a user wants to operate on within an API call, the components are selected using the new selector syntax in an index expression:

GET my-data-stream::failures/_search 

On supporting APIs, selector syntax is evaluated first in every index expression. The syntax only accepts a predefined list of selector names or the match all * wildcard. It does not support any other selector names and it does not support simple match syntax. Any unsupported usage of the selector syntax will raise an exception.

Examples:

// Accepted my-data-stream::data // Resolves to the data stream's backing indices only my-data-stream::failures // Resolves to the data stream's failure indices only my-data-stream::* // Resolves to both the data stream's backing and failure indices // Rejected my-data-stream::d*ta // Simple match syntax is not supported my-data-stream::custom // Syntax only accepts predefined selector names (currently `data` and `failures`) 

Selector syntax is compatible with other expression syntax like date math and wildcard matching of index names:

GET <logs-daily-{now/d}>::failures/_settings GET logs-*::failures/_mappings GET *::*,-logs-*::data/_search 

Selector syntax will not be supported on every API as it does not always make sense. For example, the create and delete index APIs operate on indices only and will not support selectors. In these cases, any selector usage will raise an exception:

PUT logs-my-index::data // Raises exception because selector was used on unsupporting API 

API's that do support selectors will be configured with the default selectors to use when they are not provided on an index expression. API's like _search and those based on it will use only the ::data selector by default to avoid surfacing unrelated failure information in search results. Management and monitoring API's like the Get Indices API and _stats will default to using both ::data and ::failures to surface information about the entire index/data stream/alias specified.

[jbaiera]

Opening this as WIP for now. The selector syntax is now parsed as part of the expression resolution logic and is surfaced to callers of IndexNameExpressionResolver.resolveExpressions(...) method. Alias filter resolution in search is updated with the new selector logic. Data stream API's need updating, as do all IndicesOptions uses to include the new defaults for selector usage. Test updates and additions are ongoing.

[gmarouli]

In order to pick up this amazing work from @jbaiera , I have to cut it into smaller pieces to ensure I can understand the work already done and take the next steps. To achieve this, I decided to split this into multiple PRs as an exercise.

This is my aspiring plan, due to change, as we go further and understand the code better:

• Add the ResolvedExpression wrapper, for now it is just a wrapper around a string, but at a next step it will also contain the selectors ([Failure store - selector syntax] Add ResolvedExpression wrapper #114592).
• Replace the IndicesOptions boolean constructor with the builder, only on the requests touched by this PR. To reduce the overhead when we actually move to the failure store changes ([Failure store - selector syntax] Refactor IndicesOptions builder #114597).
• Replace the FailureStoreOptions with the default SelectorOptions. This is an internal change, the APIs won't change ([Failure store - selector syntax] Replace failureOptions with selector options internally. #114812).
• Remove the failure_store param and look for the ::. ([Failure store - selector syntax] Introduce selector syntax in the APIs and remove failure_store query param #114992)
• Add support for resolve index API ([Failure store] Use selectors in the Resolve API #115212)
• Add support for rollover (and remove the target_failure_store ([Failure store] Add selector support in the rollover API #115210)
• Add/fix support for snapshots ([Failure store] Use selectors in snapshot creation and restore. #115242)
• Start working the list of all the different endpoints and determining the default behaviour and implement it.
• Fix any bugs found along the way

[gmarouli]

#115210

[gmarouli]

#115210

[gmarouli]

This work is postponed to #115212

[jbaiera]

Closing this in favor of continuing development on #114992