ESQL - KNN function uses LIMIT for K, transforms to exact search when not pushed down #132944

carlosdelest · 2025-08-14T17:20:05Z

Changes how KNN works for ESQL:

k is no longer specified as a parameter
LIMIT is used to provide the k parameter
When knn can't be pushed down to Lucene, it is transformed into an exact search
Instead of num_candidates, users can provide min_candidates. That is the minimum number of candidates to use, as in case the query is transformed into an exact search we can't really guarantee that we will be evaluating num_candidates at the most
In case min_candidates is specified, then it is used as k if it's bigger than the LIMIT applied to KNN

Example:

FROM example | WHERE knn(vector_field, [0, 120, 0]) | LIMIT 10

The above example will use k=10, as that is the LIMIT used

The following example specifies the minimum number of candidates as 200:

FROM example | WHERE knn(vector_field, [0, 120, 0], {"min_candidates": 200}) | LIMIT 10

The following example will use exact nearest neighbors search, as it is used as part of a non-pushable disjunction:

FROM example | WHERE knn(vector_field, [0, 120, 0] OR length(title) > 100) | LIMIT 10

The following example will use exact nearest neighbors search, as it has a non-pushable knn prefilter:

FROM example | WHERE knn(vector_field, [0, 120, 0] AND length(title) > 100) | LIMIT 10

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

…act-search-non-pushed # Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/knn-function.csv-spec # x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/KnnFunctionIT.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed

github-actions · 2025-08-28T14:39:42Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/knn.md

carlosdelest · 2025-08-28T14:55:59Z

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/ScoreOperator.java

- assert page.getBlockCount() >= 2 : "Expected at least 2 blocks, got " + page.getBlockCount();
- assert page.getBlock(0).asVector() instanceof DocVector : "Expected a DocVector, got " + page.getBlock(0).asVector();
- assert page.getBlock(1).asVector() instanceof DoubleVector : "Expected a DoubleVector, got " + page.getBlock(1).asVector();
+ assert page.getBlockCount() > scoreBlockPosition : "Expected to get a score block in position " + scoreBlockPosition;


Minor unrelated change - removes unnecessary assertions and uses a non-hardcoded position

…act-search-non-pushed

elasticsearchmachine · 2025-08-29T07:40:56Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-08-29T07:40:56Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-08-29T07:40:57Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

john-wagster · 2025-08-29T15:06:06Z

docs/reference/query-languages/esql/_snippets/functions/functionNamedParams/knn.md

 `boost`
 : (float) Floating point number used to decrease or increase the relevance scores of the query.Defaults to 1.0.

+`min_candidates`


Thinking through this deviation from knn query in the _search dsl may impact my work. I had naively started to add visit_percentage into the set of available options here too but now I'm questioning that a bit given that you are thinking of moving away from num_candidates. Thoughts on my draft here: https://github.com/elastic/elasticsearch/pull/133753/files#diff-0ec49ad4bdf06d1a122ea4657297fa276019d12b7affc9b07ab61f36e7b77c09

visit_percentage would for bbq_disk override num_candidates and provide essentially more fine-grained control over what users can specify for total explored vectors.

visit_percentage would for bbq_disk override num_candidates and provide essentially more fine-grained control over what users can specify for total explored vectors.

IIUC, we could provide a min_visit_percentage here for the same purpose. Would that work?

The main goal is to make sure users are not surprised in case knn needs to be translated to an exact query because it can't be pushed down.

Is this an option we plan to add for knn in general? What will happen in case the underlying index format doesn't support it? Or are we planning to support this for hnsw as well?

IIUC, we could provide a min_visit_percentage here for the same purpose. Would that work?

Let me think about that over the weekend here. I think I understand why you are doing this. So it may just be a matter of whose PR goes first.

Is this an option we plan to add for knn in general? What will happen in case the underlying index format doesn't support it? Or are we planning to support this for hnsw as well?

Like only "disk" related formats which only be "bbq_disk" for now. It will be ignored otherwise. Feedback on that is welcome too.

Thought about this some more; I'm going to take any updates to ESQL out of my PR. For one it complicates what's there. And we also just don't need it yet. We can revisit in the future and that at least makes this conversation moot.

Unrelated my gut reaction was this is a bit of a red flag. I struggle a little bit with differing options between _search and ESQL but after reading a bit I understand why this is needed. +1

Jumping on this thread, I think we'll need some communication around this to users - it's not super intuitive if you're used to working with oldschool KNN search.

Yes, we can do that in the docs and blog post. It's good that num_candidates is no longer a supported option in this knn function to avoid further confusion.

ioanatia · 2025-08-31T18:46:13Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

 assertTrue(secondKnnFilters.contains(firstOr.right()));
 }

+ public void testKnnImplicitLimit() {


can we get a test when knn is used before stats? e.g. from test | where knn(...) | stats x = COUNT(*).
I know it might seem like this does not make a lot of sense to use STATS after KNN, but it would be good to check what k do we set.
since k is set through an optimization rule, it would be great to also have CSV tests with knn used before STATS, RERANK etc.

I added unit tests as part of 9d3c85f, and added CSV tests in af3296c

ioanatia · 2025-08-31T18:53:47Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushLimitToKnn.java

+ // Break if it's not the initial limit
+ breakerReached.set(firstLimit.get());
+ firstLimit.set(true);
+ } else if (plan instanceof TopN || plan instanceof Rerank || plan instanceof Aggregate) {


what happens in the case where we reach TopN/Rerank/Aggregate before we reach a LIMIT plan?
will the value of k be null in this case? does that mean we will later fail to translate the knn function in a knn query?

@ioanatia That fails as of now - I should have tested for this 😞 .

I can think of two ways of addressing this:

Enforcing using a LIMIT for knn. Fail the query if there's no implicit / explicit LIMIT.

The following would fail:

| WHERE knn() | STATS c = count(*)

but also, the following would fail:

| WHERE knn() | STATS c = count(*) where _score > 0.5

Use an exact search when a LIMIT is not enforced. That would help with the examples above, at the cost of doing a search over all matching rows.

I'm favoring 1) as knn makes no sense if not used as part of a TopN, and users should use an explicit exact search for those cases. It's also a restriction that we can eventually lift if we come with other solution.

WDYT?

I've taken a shot at implementing 1) in 9d3c85f

works for now - we can always address this in another way later 👍

…act-search-non-pushed # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed

kderusso

Overall looks good to me, some minor questions

kderusso · 2025-09-02T17:36:08Z

docs/reference/query-languages/esql/_snippets/functions/functionNamedParams/knn.md

 `boost`
 : (float) Floating point number used to decrease or increase the relevance scores of the query.Defaults to 1.0.

+`min_candidates`


Jumping on this thread, I think we'll need some communication around this to users - it's not super intuitive if you're used to working with oldschool KNN search.

kderusso · 2025-09-02T17:43:09Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

- public KnnQuery(Source source, String field, float[] query, Map<String, Object> options, List<QueryBuilder> filterQueries) {
+ public KnnQuery(Source source, String field, float[] query, Integer k, Map<String, Object> options, List<QueryBuilder> filterQueries) {
 super(source);
+ assert k != null && k > 0 : "k must be a positive integer, but was: " + k;


Assert on max limit too?

It was more about a sanity check for having k set before the query translation in ES|QL, as the overall error checking will be done on the QueryBuilder.

kderusso · 2025-09-02T17:45:30Z

...ql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

+ }
+
+ public void testKnnUsesLimitForK() {
+ assumeTrue("dense_vector capability not available", EsqlCapabilities.Cap.DENSE_VECTOR_FIELD_TYPE.isEnabled());


Do we also need to check for the fork capability?

There's no FORK involved in the test - the test name is "test Knn Uses Limit For K", not "FORK" 😁

Haha that's what I get for reading it quickly 🤦

ioanatia · 2025-09-03T09:18:34Z

.../esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushLimitToKnn.java

+ // Break if it's not the initial limit
+ breakerReached.set(firstLimit.get());
+ firstLimit.set(true);
+ } else if (plan instanceof TopN || plan instanceof Rerank || plan instanceof Aggregate) {


works for now - we can always address this in another way later 👍

ioanatia · 2025-09-03T09:22:12Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

+ """), containsString("Knn function must be used with a LIMIT clause"));
+ }
+
+ public void testKnnWithRerankAmdLimit() {


I'd like to get one more test - we have an optimization that combines limits and it would be nice to test the combination of the two, e.g. FROM my-index metadata _score | WHERE knn(...) | LIMIT 200 | LIMIT 10 - I expect k will be 10 here, no?

Yep, it doesn't hurt - added in c4f3da7.

The nice thing about optimization rules is that they can be applied independently and still work together, like in this case 🥳

github-actions · 2025-09-03T10:04:56Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

carlosdelest added 4 commits August 13, 2025 13:28

Make ScoreOperator and LuceneQueryEvaluator more robust

ee3c806

Translate to exact NN when not pushable

82779ec

KNN k is set via optimizer and limit

0fb162a

Fix KnnFunctionIT test

5694ae7

elasticsearchmachine added the v9.2.0 label Aug 14, 2025

carlosdelest added :Analytics/ES|QL AKA ESQL :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch :Search Relevance/ES|QL Search functionality in ES|QL labels Aug 14, 2025

elasticsearchmachine and others added 16 commits August 14, 2025 17:25

[CI] Auto commit changes from spotless

3e68d71

Fix CSV tests

1b2829b

Use min_candidates

e3dd487

Bump capability

38cfe1d

Merge remote-tracking branch 'carlosdelest/non-issue/esql-knn-exact-s…

e4ef60f

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

[CI] Auto commit changes from spotless

d92a5dd

Fix tests

66e3dcb

Fix min_candidates handling

71d3a48

Fix tests

eb47c7b

Add tests

3daf953

Merge remote-tracking branch 'carlosdelest/non-issue/esql-knn-exact-s…

5d1f694

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed

Fix tests

db5c018

Spotless

952a7c9

Fix tests

f499025

Fix generated docs

09e02da

Merge branch 'main' into non-issue/esql-knn-exact-search-non-pushed

69e7731

carlosdelest commented Aug 28, 2025

View reviewed changes

carlosdelest added 2 commits August 28, 2025 17:06

Merge remote-tracking branch 'origin/main' into non-issue/esql-knn-ex…

52f75f1

…act-search-non-pushed

Add docs and fix equals / hashCode

6cbf31a

carlosdelest requested review from afoucret, jimczi, kderusso and svilen-mihaylov-elastic August 29, 2025 07:45

carlosdelest added the >non-issue label Aug 29, 2025

john-wagster reviewed Aug 29, 2025

View reviewed changes

ioanatia reviewed Aug 31, 2025

View reviewed changes

carlosdelest added 9 commits September 1, 2025 13:19

Verify that knn has a limit

9d3c85f

Fix tests

e95a5bc

Merge remote-tracking branch 'origin/main' into non-issue/esql-knn-ex…

a2cdfbc

…act-search-non-pushed # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

Spotless

d11d074

Merge remote-tracking branch 'carlosdelest/non-issue/esql-knn-exact-s…

09d365c

…earch-non-pushed' into non-issue/esql-knn-exact-search-non-pushed

Add CSV tests for stats / rerank

af3296c

Fix CSV test

e9ed8e5

Fix rerank test

9fe7f90

Improve rerank test

8fe374d

carlosdelest mentioned this pull request Sep 1, 2025

ES|QL - dense_vector approximate nearest neighbour search support #126710

Open

8 tasks

kderusso reviewed Sep 2, 2025

View reviewed changes

ioanatia approved these changes Sep 3, 2025

View reviewed changes

carlosdelest and others added 2 commits September 3, 2025 12:01

Add test for multiple limits combination

c4f3da7

Merge branch 'main' into non-issue/esql-knn-exact-search-non-pushed

c19a43a

carlosdelest enabled auto-merge (squash) September 3, 2025 10:03

carlosdelest merged commit e76c2a6 into elastic:main Sep 3, 2025
33 checks passed

This was referenced Sep 11, 2025

ES|QL - so_vector knn function update elastic/rally-tracks#850

Merged

ESQL - KNN functions with non-pushed down filters #131708

Closed

ESQL - KNN function uses LIMIT for setting top k #129353

Closed

ESQL - KNN function uses LIMIT for K, transforms to exact search when not pushed down #132944

ESQL - KNN function uses LIMIT for K, transforms to exact search when not pushed down #132944

Uh oh!

Conversation

carlosdelest commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Choose a reason for hiding this comment

elasticsearchmachine commented Aug 29, 2025

elasticsearchmachine commented Aug 29, 2025

elasticsearchmachine commented Aug 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kderusso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 3, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Labels

5 participants

carlosdelest commented Aug 14, 2025 •

edited

Loading

github-actions bot commented Aug 28, 2025 •

edited

Loading