Adding linear retriever to support weighted sums of sub-retrievers

[pmpailis]

This PR adds a new linear retriever to facilitate hybrid search, that would be able to linearly combine the results of other sub-retrievers and compute the final score of a document based on the weighted sum of each sub-components.

Each sub-component can specify the following elements:

• retriever -> specifies how we will compute the top documents
• normalizer -> specifies how we want to normalize the top documents for this retriever (so that we can ensure that all scores fall within the same range)
• weight -> the weight for the normalized score if the final weighted sum computation

Pagination is similar to that of rrf's retriever, i.e. we compute the global rank_window_size docs and pagination is only available within these bounds.

So, working through an example, let's say that we perform a hybrid search query where:

• we want to run a simple string query through a standard retriever, and normalize the scores to a [0, 1] range
• we want to run knn search through the knn retriever, without normalizing the documents as well
• compute the final score for the retriever as score = 1.5 * standard + 2.5 * knn

Sample syntax:

GET /retrievers_example/_search { "retriever": { "linear": { "retrievers": [ { "retriever": { "standard": { "query": { "simple_query_string": { "query": "artifical intelligence in medicine", "fields": [ "text" ] } } } }, "weight": 1.5, "normalizer": "minmax" }, { "retriever": { "knn": { "field": "vector", "query_vector": [ 0.23, 0.67, 0.89 ], "k": 3, "num_candidates": 5 } }, "weight": 2.5 } ], "rank_window_size": 10 } } } 

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Hi @pmpailis, I've created a changelog YAML for you.

[benwtrent]

Looking much better. I have a concern around testing:

Do we have a test that specifically exercises the path when the different retrievers return different doc IDs? (e.g. they match non-overlapping doc sets).

[pmpailis]

Do we have a test that specifically exercises the path when the different retrievers return different doc IDs? (e.g. they match non-overlapping doc sets).

Added a test to account for this in ea1787f

[benwtrent]

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 120222