Add retrievers using the parser-only approach

[jdconrad]

This enhancement adds a new abstraction to the _search API called "retriever." A retriever is something that returns top hits.

Three retrievers are going to be added initially:

• standard - This retriever replaces standard query functionality. It contains sub-elements for query, search_after, terminate_after, sort_field, min_score, and collapse.
• knn - This retriever replaces a knn search. It contains sub-elements for field, k, num_candidates, query_vector, query_vector_builder, and similarity.
• rrf - This retriever replaces rank for rrf. It contains sub-elements for retrievers, rank_constant, and window_size.

All retrievers have a filter element as well. For rrf this is propagated to its child retrievers.

Also included is node feature functionality. (Note the plumbing for this was part of #105417). Each retriever has its own node feature to ensure compatibility with serverless. New retrievers can be easily added by adding a new node feature without having to worry about incompatibilities.

Retrievers are always "extracted" to a SearchSourceBuilder. They are simply an abstraction to better show how elements of the _search API interact together in an intuitive way. They are never serialized and are ephemerally used only during parsing.

Some examples:

GET index/_search { "retriever": { "standard": { "query": { "match": { "message": { "query": "This is a test." } } }, "sort": [ "popularity" ] } }, "size": 5, "fields": ["message"] } GET index/_search { "retriever": { "knn": { "field": "vector", "k": 3, "num_candidates": 10, "query_vector": [1, 2, 3] } }, "fields": ["message"] } GET index/_search { "retriever": { "rrf": { "window_size": 100, "retrievers": [ { "knn": { "field": "vector", "k": 3, "num_candidates": 10, "query_vector": [1, 2, 3] } }, { "standard": { "query": { "match": { "message": { "query": "This is a test." } } } } } ] } }, "size": 5, "fields": ["message"] } 

[jdconrad]

@javanna @mayya-sharipova Thank you for the first round of reviews. I have responded to all the PR comments and added documentation changes for this. I would strongly prefer to minimize additional changes unless something is either incorrect or there is a significant design flaw given other work depending on this.

[javanna]

LGTM thanks @jdconrad

[javanna]

I thought we never released sub_searches support to the public. But it seems like we did, yet it was experimental? I wonder if we should even mention it as a migration path.

[jdconrad]

I agree! I actually had removed it, but there are cross references from enterprise-search docs that I wasn't aware of until I got errors here during CI. I figured this allows enterprise-search time to move their documentation to use retrievers instead of sub_searches and we can remove them once they have the opportunity to change their references.

[javanna]

I see, can we clarify whether we need to mention the previous way at all?

[jdconrad]

@javanna I chatted with @kderusso about removing these elements from the documentation. She's going to review the docs in this PR, and we will follow up with getting the cross references changed once this PR is merged.

[kderusso]

Very nice work Jack! I left a few comments, none of which are blocking. Congrats!

[kderusso]

It would be nice if we continued to link to the main RRF page here as well.

Suggested change

	This can also be achieved by using an <<rrf-retriever, `rrf` retriever>>
	with multiple <<standard-retriever, `standard` retrievers>>.
	This can also be achieved using <<rrf>>, through an <<rrf-retriever, `rrf` retriever>>
	with multiple <<standard-retriever, `standard` retrievers>>.

[jdconrad]

Changed.

[kderusso]

I could see this being a little confusing to users, since knn is also a query (e.g. could knn-query be nested under a standard retriever?) I realize this is a weird case, but it might be good to address this somewhere, maybe in a blog post or something? WDYT?

[jdconrad]

You could use a knn query under a standard retriever. I agree this is probably better left to documentation in a different place, though. I worry that adding some notes about how you can use knn-query as part of a standard retriever will just confuse the issue more currently. I'll give this some thought as a possible follow up.

[kderusso]

Is it fewer or <=?

[jdconrad]

Good catch! I added an equal to as well.

[kderusso]

Are we actively trying to change our vocabulary from result to top documents? It may be unintuitive or confusing?

[jdconrad]

I was doing this since the notion of a retriever is an entity that returns top documents. So I wanted to try to be consistent with that. I think I will change this one back to results, though, since it's not in the context of a retriever here.

[kderusso]

Are we planning to add tests once RRF moves out of tech preview?

[jdconrad]

Ideally, yes. Setup here with documentation is tricky with the other example being tested directly.

[kderusso]

Nice example 👍

[kderusso]

This is another example that could be annotated with numbers in the example fragment?

[jdconrad]

Same comment as above. I didn't find a way where I felt like the readability of the example was improved.

[kderusso]

Here's along the lines of what I was thinking:

[source,console] ---- GET example-index/_search { "retriever": { "rrf": { <3> "retrievers": [ { "standard": { <1> "query": { "term": { "text": "blue shoes sale" } } } }, { "standard": { <2> "query": { "text_expansion":{ "ml.tokens":{ "model_id":"my_elser_model", "model_text":"What blue shoes are on sale?" } } } } } ], "window_size": 50, "rank_constant": 20 } } } ---- // TEST[skip:example fragment] In the above example, we execute each of the two `standard` retrievers independently of each other. <1> First we run the `standard` retriever specifying a term query for `blue shoes sales` using the standard BM25 scoring algorithm. <2> Next we run the `standard` retriever specifying a text expansion query for `What blue shoes are on sale?` using our <<semantic-search-elser, ELSER>> scoring algorithm. <3> The `rrf` retriever allows us to combine the two top documents sets generated by completely independent scoring algorithms with equal weighting. Not only does this remove the need to figure out what the appropriate weighting is using linear combination, but RRF is also shown to give improved relevance over either query individually. 

If you don't feel that it provides value though, it's definitely not a necessary change.

[jdconrad]

This is nicer. I will change this. Thank you for the write up here. (It hadn't occurred to me to mark the rrf retriever out of order in this way.)

[kderusso]

One note for search applications - it's possible that people may have defined search applications with templates that use sub_searches (especially since we provided this as an example). Do we want to provide any type of warning to users via search responses that this is deprecated?

[jdconrad]

Good question! Let me follow up on this. Since rank and sub_searches aren't supported on serverless we can do this as a follow up.

[kderusso]

Should we remove the verbiage about syntax changing prior to GA for these features?

[jdconrad]

I think this is pretty standard until a feature does become GA just in case we discover any design flaws, so I'd prefer to error on the side of caution here.

[kderusso]

It would be nice to break this out as a constant.

[jdconrad]

Absolutely! This needs a bit of a re-design to work with re-rankers, so I'm going to leave it for now. I think depth will be less important than specific ordering in the end.

[benwtrent]

@pmpailis @jdconrad we should highlight the new format as its much nicer. I am not sure @giladgal as it is still technical preview.