Add `sub_searches` to the search endpoint #96224

jdconrad · 2023-05-18T16:53:41Z

This change adds a new top-level element to the search endpoint called sub_searches. This top-level element allows for a list of additional searches where each "sub search" will have a query executed separately as part of ranking and later combined into a final single set of documents based on the ranking algorithm.

The sub_searches element is used instead of query (and they may not be used together) to allow a search using ranking to execute multiple queries.

An example search using sub_searches looks like:

GET example-index/_search { "sub_searches": [ { "query": { "term": { "text": "red shoes" } } }, { "query": { "term": { "text": "blue shoes" } } } ], "rank": { "rrf": { "window_size": 100, "rank_constant": 25 } } }

execute multiple queries individually

benwtrent · 2023-06-15T20:34:01Z

server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java

+ } else {
+ throw new XContentParseException(
+ parser.getTokenLocation(),
+ "malformed query within the [queries] field; found " + token


Needs a rename

benwtrent · 2023-06-15T20:36:36Z

server/src/main/java/org/elasticsearch/search/internal/ShardSearchRequest.java

- } else if (rankQueryBuilders.isEmpty() == false) {
- throw new IllegalArgumentException("cannot serialize [rank] to version [" + out.getTransportVersion() + "]");


did you mean to delete this?

Yes. This logic was moved to SearchSourceBuilder instead since where that's where subSearchBuilders lives.

Edit: I take that back, that was a different serialization error. I need to think about this.

Second edit: This is correct. We won't serialize multiple searches to anything prior to 8.8 now which is the equivalent to this with the new data structure for sub searches.

javanna

I left a bunch of new comments, mostly cosmetic things, or around hardening certain details. No blockers though, those could also be addressed as a follow-up especially if they require significant effort.

LGTM!

server/src/main/java/org/elasticsearch/search/SearchService.java

javanna · 2023-06-15T18:20:14Z

server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java

- public SearchSourceBuilder query(QueryBuilder query) {
- this.queryBuilder = query;
+ public SearchSourceBuilder query(QueryBuilder queryBuilder) {
+ subSearchSourceBuilders = new ArrayList<>();


this will reset the list no matter what was previously set to it? What happens if we assert that the list is empty?

I admit this is a bit fragile, but I think this should be addressed as a follow up. The prior behavior would replace the previous query so asserting there would break updates to this method.

I see, maybe just assert that there is a single element in the list then so it's 1:1 with the previous behaviour of replacing?

javanna · 2023-06-15T20:40:24Z

server/src/main/java/org/elasticsearch/search/builder/SearchSourceBuilder.java

+ * Sets the queries for this request.
+ */
+ public SearchSourceBuilder subSearches(List<SubSearchSourceBuilder> subSearchSourceBuilders) {
+ this.subSearchSourceBuilders = subSearchSourceBuilders;


should we enforce that the list can never be null?

Yes! Good call.

server/src/main/java/org/elasticsearch/search/internal/ShardSearchRequest.java

javanna · 2023-06-15T21:27:43Z

server/src/main/java/org/elasticsearch/search/SearchService.java

- // null query means match_all
- canMatch = aliasFilterCanMatch;
+ canMatch &= request.source()
+ .subSearches()


will subsearches here already contain the knn queries too? I think so, and that should not be an issue, but I wanted to double check.

there should never be a knn search here because can_match only works with query_then_fetch, but not dfs_query_Then_fetch.

oh good point. maybe that is something that we need to discuss changing.

server/src/main/java/org/elasticsearch/search/SearchService.java

javanna · 2023-06-15T21:33:45Z

server/src/main/java/org/elasticsearch/search/internal/ShardSearchRequest.java

+ if (in.getTransportVersion().onOrAfter(TransportVersion.V_8_8_0) && in.getTransportVersion().before(TransportVersion.V_8_500_004)) {
+ List<QueryBuilder> rankQueryBuilders = in.readNamedWriteableList(QueryBuilder.class);
+ for (QueryBuilder queryBuilder : rankQueryBuilders) {
+ source.queries().add(new SearchQueryWrapperBuilder(queryBuilder));


ok so that means that we don't need to add specific tests and we can rely on existing test coverage? Which test specifically are you referring to?

javanna · 2023-06-15T21:36:26Z

x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/100_rank_rrf.yml

+ body:
+ track_total_hits: true
+ fields: [ "text" ]
+ queries: [


Hear hear, Jack came up with an improved name, as promised back when we decided to add a new top level element :)

server/src/main/java/org/elasticsearch/action/search/DfsQueryPhase.java

elasticsearchmachine · 2023-06-15T21:45:35Z

Hi @jdconrad, I've updated the changelog YAML for you. Note that since this PR is labelled release highlight, you need to update the changelog YAML to fill out the extended information sections.

Missing word in changelog

jdconrad · 2023-06-16T01:05:47Z

@elasticsearchmachine run elasticsearch-ci/part-1

jdconrad · 2023-06-16T01:23:16Z

@elasticmachine run elasticsearch-ci/part-1

jdconrad · 2023-06-16T03:54:50Z

Thank you again @mayya-sharipova, @benwtrent, and @javanna for the reviews!

This adds basic documentation for the sub_searches top-level element in the search API. (#96224).

This adds basic documentation for the sub_searches top-level element in the search API. (elastic#96224).

This adds basic documentation for the sub_searches top-level element in the search API. (#96224).

jdconrad added 3 commits May 16, 2023 12:24

add wrapper object for a single query

e867b60

add wrapper for search queries to carry enough information to do

ef1a959

execute multiple queries individually

add paths for dfs and no dfs phases

3c2d6bd

jdconrad added the WIP label May 18, 2023

elasticsearchmachine added the v8.9.0 label May 18, 2023

jdconrad added 3 commits May 18, 2023 10:04

Merge branch 'main' into rrfmulti

97f465d

fix npe

541b6c1

another npe

e7971e1

jdconrad added the :Search Relevance/Ranking Scoring, rescoring, rank evaluation. label May 18, 2023

jdconrad added 21 commits May 18, 2023 10:52

fix test

3bb28ce

fix npe

17311de

make version high to avoid conflicts for now

56a5602

Merge branch 'main' into rrfmulti

3dc0b8b

fix transport version id

fdc27a7

add an it test for 2 bm25 queries as part of rrf

188c6f6

fix test

03568c1

fix test

649e778

Merge branch 'main' into rrfmulti

e0e4ee2

add restriction for query and queries used at the same time

ad95da3

Add tests for SearchQueryBuilder

70f673c

rename SearchQuery to SearchQueryWrapper

26f2883

remove dead code

5315d0e

do not use single shard shortcut when using rank

e0ec23e

Merge branch 'main' into rrfmulti

e18cf29

update shard counts

8fa9c52

add additional check for single shard optimization to not run with rank

807f654

spotless

9a06051

Merge branch 'main' into rrfmulti

0d6dbeb

Merge branch 'main' into rrfmulti

9424ef6

Merge branch 'main' into rrfmulti

6c9e87b

benwtrent reviewed Jun 15, 2023

View reviewed changes

jdconrad added 3 commits June 15, 2023 13:54

fix error messages

70b5dbd

replace more messages with sub searches in place of queries

60f1367

fix validation test

3a790b4

javanna approved these changes Jun 15, 2023

View reviewed changes

javanna added the release highlight label Jun 15, 2023

jdconrad added 3 commits June 15, 2023 14:45

Update docs/changelog/96224.yaml

e6f8591

response to pr comments

f89e085

cleaned up bwc deserialization

7289e37

jdconrad changed the title ~~Add multiple queries to search endpoint~~ Add sub_searches to the search endpoint Jun 16, 2023

jdconrad added 3 commits June 15, 2023 17:46

fix change log with highlight

48d6f5f

Merge branch 'main' into rrfmulti

6c991bb

Update 96224.yaml

f4655c1

Missing word in changelog

jdconrad added 2 commits June 15, 2023 18:40

no op

1b98e56

mute bad test

403185f

jdconrad mentioned this pull request Jun 16, 2023

[CI] GetDataStreamsTransportActionTests testGetTimeSeriesMixedDataStream failing #96672

Closed

jdconrad merged commit 52be52b into elastic:main Jun 16, 2023

slobodanadamovic mentioned this pull request Jun 16, 2023

[CI] SearchSourceBuilderTests testFromXContent failing #96896

Closed

martijnvg mentioned this pull request Jun 19, 2023

[CI] ShardSearchRequestTests testForceSyntheticUnsupported failing #96910

Closed

jdconrad mentioned this pull request Jun 22, 2023

Add basic documentation for sub searches #97025

Merged

jdconrad added a commit that referenced this pull request Jun 28, 2023

Add basic documentation for sub searches (#97025)

bca4995

This adds basic documentation for the sub_searches top-level element in the search API. (#96224).

jdconrad added a commit to jdconrad/elasticsearch that referenced this pull request Jun 28, 2023

Add basic documentation for sub searches (elastic#97025)

6307bbb

This adds basic documentation for the sub_searches top-level element in the search API. (elastic#96224).

elasticsearchmachine pushed a commit that referenced this pull request Jun 28, 2023

Add basic documentation for sub searches (#97025) (#97195)

f363a0d

This adds basic documentation for the sub_searches top-level element in the search API. (#96224).

ezimuel mentioned this pull request Jun 28, 2023

Add sub_searches to the search endpoint elastic/elasticsearch-specification#2162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `sub_searches` to the search endpoint #96224

Add `sub_searches` to the search endpoint #96224

Uh oh!

jdconrad commented May 18, 2023 •

edited

Loading

benwtrent Jun 15, 2023

jdconrad Jun 15, 2023

benwtrent Jun 15, 2023

jdconrad Jun 15, 2023 •

edited

Loading

javanna left a comment

Uh oh!

javanna Jun 15, 2023

jdconrad Jun 15, 2023

javanna Jun 15, 2023

javanna Jun 15, 2023

jdconrad Jun 15, 2023

Uh oh!

Uh oh!

javanna Jun 15, 2023

jdconrad Jun 15, 2023

javanna Jun 16, 2023

Uh oh!

javanna Jun 15, 2023

javanna Jun 15, 2023

Uh oh!

elasticsearchmachine commented Jun 15, 2023

jdconrad commented Jun 16, 2023

jdconrad commented Jun 16, 2023

jdconrad commented Jun 16, 2023

Labels

5 participants

		} else if (rankQueryBuilders.isEmpty() == false) {
		throw new IllegalArgumentException("cannot serialize [rank] to version [" + out.getTransportVersion() + "]");

Add sub_searches to the search endpoint #96224

Add sub_searches to the search endpoint #96224

Uh oh!

Conversation

jdconrad commented May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdconrad Jun 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

javanna left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jun 15, 2023

jdconrad commented Jun 16, 2023

jdconrad commented Jun 16, 2023

jdconrad commented Jun 16, 2023

Labels

5 participants

Add `sub_searches` to the search endpoint #96224

Add `sub_searches` to the search endpoint #96224

jdconrad commented May 18, 2023 •

edited

Loading

jdconrad Jun 15, 2023 •

edited

Loading