ESQL - KNN function uses prefilters when pushed down to Lucene #131004

carlosdelest · 2025-07-10T11:45:43Z

KNN function returned less results than expected when using conjunctions:

FROM test METADATA _score WHERE knn(vector, [0, 1, 2], 10) AND primary == true SORT _score DESC

Before this change, knn retrieved the top k results, but then those would be post-filtered by the conjunction. That potentially retrieves in less than k results retrieved (see filters in knn).

This PR pushes down conjunctions as prefilters to KNN queries:

A new PushDownConjunctionsToKnnPrefilters logical optimizer rule examines filters, and pushes conjunctions to knn functions as prefilters. It works by identifying knn functions on one side of a conjunction, and pushing the other side of the conjunction(s) that are parent to the knn function to it as prefilters.
KNN function uses the prefilters when generating the QueryBuilder.
KNN checks that prefilters can be pushed down in order to decide whether it can be pushed down to Lucene.

This PR still doesn't handle the case when KNN cannot be pushed down and has prefilters - that will come as a follow up PR.

knn functions won't have knn queries as prefilters to avoid circular dependencies.

Whit this change, a query with:

WHERE ((knn(A) AND B) OR C) AND (knn(D) OR (E AND F))

will result in:

knn(A) will have B, E AND F as prefilters (other side of the conjunctions it's part of, minus knn(D))
knn(D) will have B OR C as prefilters (other side of the conjunctions it's part of, minus knn(A))

… to be generated with prefilters

…shed down

# Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

github-actions · 2025-07-10T11:47:19Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/knn.md

carlosdelest · 2025-07-10T11:46:46Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/knn-function.csv-spec

-# TODO We need kNN prefiltering here so we get more candidates that pass the filter
 from colors metadata _score 
-| where (knn(rgb_vector, [0,255,255], 140) or knn(rgb_vector, [128, 0, 255], 140)) and primary == true 
+| where (knn(rgb_vector, [0,255,255], 140) or knn(rgb_vector, [128, 0, 255], 10)) and primary == true 


Using prefilter means that we no longer need to retrieve all results, but can use k as inteneded as conjunction is pushed down as a prefilter now

carlosdelest · 2025-07-10T11:47:33Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

 @Override
 public Query asQuery(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler) {
- return queryBuilder != null ? new TranslationAwareExpressionQuery(source(), queryBuilder) : translate(handler);
+ return queryBuilder != null ? new TranslationAwareExpressionQuery(source(), queryBuilder) : translate(pushdownPredicates, handler);


Added pushdownPredicates to translate method, as we need to evaluate the prefilters with them in order to decide

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

elasticsearchmachine · 2025-07-10T14:39:17Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-07-10T14:39:17Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

...gin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/KnnFunctionIT.java

ioanatia · 2025-07-11T10:43:11Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java

 this.field = field;
 this.k = k;
 this.options = options;
+ this.filterExpressions = filterExpressions;


we need to add both filterExpressions and queryBuilder to the nodeInfo method:

https://github.com/elastic/elasticsearch/blob/0b06d9f24dbac94f89bf696daad98041b93e7b1b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java#L292-294

my understanding is that if we don't add them, we risk losing both filterExpressions and queryBuilder during transforms.

Thanks for the catch - added in b347bba

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/vector/Knn.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

…sql-knn-prefilter

...rg/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownConjunctionsToKnnPrefilters.java

ioanatia · 2025-07-11T12:43:46Z

...ql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

+
+ String query = """
+ from test
+ | where knn(dense_vector, [0, 1, 2], 10) and integer > 10


could we modify one of these tests such that we have 2 WHERE conditions, e.g. here:

| where knn(dense_vector, [0, 1, 2], 10) | integer > 10

it would help validate that combining the filters is done before we push down the prefilters to the KNN functions

Sure, done in a99e75f

ioanatia · 2025-07-11T12:47:47Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

+ */
+ var query = """
+ from test
+ | where ((knn(dense_vector, [0, 1, 2], 10) or integer > 10) and ((keyword == "test") or knn(dense_vector, [4, 5, 6], 10)))


can we get some tests where we use NOT.
for example:

((knn(dense_vector, [0, 1, 2], 10) or integer > 10) and NOT ((keyword == "test") or knn(dense_vector, [4, 5, 6], 10)))

Makes sense, added some in 4c6bb4d

…sql-knn-prefilter # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/knn-function.csv-spec

carlosdelest · 2025-07-14T11:44:42Z

After discussing with @jimczi , I removed knn functions as filters to other knn functions to avoid circular dependencies in ac7c3bc.

kderusso

Nice work @carlosdelest !

kderusso · 2025-07-14T19:54:35Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

 }

- protected abstract Query translate(TranslatorHandler handler);
+ protected abstract Query translate(LucenePushdownPredicates pushdownPredicates, TranslatorHandler handler);


Would it be easier to have two signatures for this, for a smaller file change count?

I think it's more confusing. LucenePushdownPredicates are part of the translation process even if they were not used until now.

kderusso · 2025-07-14T19:57:11Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

 * Support knn function
 */
- KNN_FUNCTION_V2(Build.current().isSnapshot()),
+ KNN_FUNCTION_V3(Build.current().isSnapshot()),


Super nitpick - I wonder if we could create an alias that this could reference (so when we go to V4 we only need to change one variable when testing for test compatibility)

Can you provide an example of what it would look like?

Something like KNN_FUNCTION_CURRENT = EsqlCapabilities.KNN_FUNCTION_V3?

Not sue if that works with CSV tests 🤔 . I'd say let's keep this as it's an established pattern in ES|QL.

Maybe I'll give it a try for the next one and see what it looks like from a changes perspective 👍

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/ExpressionWritables.java

…sql-knn-prefilter

…king * upstream/main: (91 commits) Mute org.elasticsearch.packaging.test.DockerTests test130JavaHasCorrectOwnership elastic#131369 Add exception logging when interrupted (elastic#131153) Mute org.elasticsearch.packaging.test.DockerTests test140CgroupOsStatsAreAvailable elastic#131372 Mute org.elasticsearch.packaging.test.DockerTests test070BindMountCustomPathConfAndJvmOptions elastic#131366 Mute org.elasticsearch.xpack.test.rest.XPackRestIT test {p0=ml/delete_expired_data/Test delete expired data with body parameters} elastic#131364 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectorAndField {functionName=v_cosine similarityFunction=COSINE} elastic#131363 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testDifferentDimensions {functionName=v_cosine similarityFunction=COSINE} elastic#131362 Mute org.elasticsearch.xpack.esql.vector.VectorSimilarityFunctionsIT testSimilarityBetweenConstantVectors {functionName=v_cosine similarityFunction=COSINE} elastic#131361 Check SCORE_FUNCTION capability in VerifierTests (elastic#131352) Replace deprecated routingTable table call in tests (elastic#131005) [DOCS] Remove misused applies_to tag (elastic#131349) Adj ivf postings list building (elastic#130843) [Transform] Read metadata from Project State (elastic#131205) Add note on o11y to architecture guide (elastic#131291) Upgrade AWS Java SDK to 2.31.78 (elastic#131050) Support Fields API in conditional ingest processors (elastic#121914) ESQL - KNN function uses prefilters when pushed down to Lucene (elastic#131004) Add docs for ES|QL query logs (elastic#131287) Simplify `expectedFinalRegisterValue` computation (elastic#131274) Mute org.elasticsearch.test.rest.yaml.RcsCcsCommonYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing, inner_hits and maxConcurrentGroupRequests} elastic#131348 ...

carlosdelest added 11 commits July 9, 2025 10:25

Add prefilters to Knn

b7a99fc

Add logical plan optimizer rule to add prefilters

61a1e4b

Adds filters to KnnVectorQueryBuilder when translating

fc0e54b

Update KNN capability name

6a52b63

Add tests

a73973f

Avoid infinite loop with multiple knn expressions, add tests

ef12ee5

Add test for not pushing down knn when there are non-pushable prefilters

616b725

Execute premapper after logical optimisation, to allow query builders…

3ae0e9b

… to be generated with prefilters

Fix generating QueryBuilder prefilters for knn when they cannot be pu…

50d6e19

…shed down

Update CSV tests now that knn uses prefilters

32715ce

Merge remote-tracking branch 'origin/main' into esql-knn-prefilter

7ebd1d9

# Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querydsl/query/KnnQuery.java

carlosdelest added :Search Relevance/Vectors Vector search :Analytics/ES|QL AKA ESQL Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.2.0 labels Jul 10, 2025

carlosdelest commented Jul 10, 2025

View reviewed changes

carlosdelest added the >non-issue label Jul 10, 2025

carlosdelest changed the title ~~ESQL - KNN function uses prefilters~~ ESQL - KNN function uses prefilters when pushed down to Lucene Jul 10, 2025

Merge branch 'main' into esql-knn-prefilter

9455f77

carlosdelest requested review from ChrisHegarty, afoucret, ioanatia, kderusso, svilen-mihaylov-elastic and tteofili July 10, 2025 14:35

carlosdelest marked this pull request as ready for review July 10, 2025 14:38

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 10, 2025

ioanatia reviewed Jul 11, 2025

View reviewed changes

carlosdelest added 5 commits July 11, 2025 13:07

Remove TransportVersion as knn is not released yet

62b0d1c

Merge remote-tracking branch 'origin/main' into esql-knn-prefilter

82fd2cc

Merge remote-tracking branch 'carlosdelest/esql-knn-prefilter' into e…

bfdd227

…sql-knn-prefilter

Add queryBuilder and filterExpresions to NodeInfo

b347bba

Remove unnecessary method

1083764

ioanatia reviewed Jul 11, 2025

View reviewed changes

carlosdelest and others added 8 commits July 11, 2025 16:20

Fix test for node info

6e4c1a9

Fix typo

91d4a0a

Add multiple filters test

a99e75f

Add negations tests

4c6bb4d

[CI] Auto commit changes from spotless

47a6d99

knn functions won't appear as prefilters for other knn functions

ac7c3bc

Merge remote-tracking branch 'carlosdelest/esql-knn-prefilter' into e…

32ada3d

…sql-knn-prefilter # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

Merge branch 'main' into esql-knn-prefilter

939705c

# Conflicts: # x-pack/plugin/esql/qa/testFixtures/src/main/resources/knn-function.csv-spec

carlosdelest requested a review from ioanatia July 14, 2025 11:45

carlosdelest added 2 commits July 14, 2025 15:20

Fix test

dd29fec

Fix test

b03f2a3

kderusso reviewed Jul 14, 2025

View reviewed changes

Merge branch 'main' into esql-knn-prefilter

20fe8fe

ioanatia approved these changes Jul 15, 2025

View reviewed changes

carlosdelest added 3 commits July 16, 2025 08:36

Merge remote-tracking branch 'origin/main' into esql-knn-prefilter

89c86e4

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/ExpressionWritables.java

Fix merge

8a550a1

Merge remote-tracking branch 'carlosdelest/esql-knn-prefilter' into e…

ea2fddd

…sql-knn-prefilter

carlosdelest merged commit 6ffe27d into elastic:main Jul 16, 2025
33 checks passed

This was referenced Jul 22, 2025

ESQL - KNN functions with non-pushed down filters #131708

Closed

ES|QL - dense_vector approximate nearest neighbour search support #126710

Open

ioanatia mentioned this pull request Jul 29, 2025

ES|QL: automatic prefiltering for semantic_text match queries #132068

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESQL - KNN function uses prefilters when pushed down to Lucene #131004

ESQL - KNN function uses prefilters when pushed down to Lucene #131004

Uh oh!

carlosdelest commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading

carlosdelest Jul 10, 2025

carlosdelest Jul 10, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 10, 2025

elasticsearchmachine commented Jul 10, 2025

Uh oh!

ioanatia Jul 11, 2025

carlosdelest Jul 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ioanatia Jul 11, 2025

carlosdelest Jul 11, 2025 •

edited

Loading

ioanatia Jul 11, 2025

carlosdelest Jul 11, 2025

carlosdelest commented Jul 14, 2025

kderusso left a comment

kderusso Jul 14, 2025

carlosdelest Jul 15, 2025

kderusso Jul 14, 2025

carlosdelest Jul 15, 2025

kderusso Jul 15, 2025

carlosdelest Jul 15, 2025

Uh oh!

Labels

5 participants

ESQL - KNN function uses prefilters when pushed down to Lucene #131004

ESQL - KNN function uses prefilters when pushed down to Lucene #131004

Uh oh!

Conversation

carlosdelest commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 10, 2025

elasticsearchmachine commented Jul 10, 2025

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

carlosdelest Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlosdelest commented Jul 14, 2025

kderusso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Labels

5 participants

carlosdelest commented Jul 10, 2025 •

edited

Loading

github-actions bot commented Jul 10, 2025 •

edited

Loading

carlosdelest Jul 11, 2025 •

edited

Loading