Add template parameters to Search Applications #95674

carlosdelest · 2023-04-28T13:37:51Z

Original PR carlosdelest#2
Continued work from kderusso#4

JSON Schema can be stored as part of the template for a Search Application:

PUT _application/search_application/<name>

{ "indices": ["test1"], "template": { "script": { "source": { "query": { "term": { "${field_name}": "${field_value}" } } }, "params": { "field_name": "hello", "field_value": "world" } }, "dictionary": { "properties": { "field_name": { "type": "string" }, "field_value": { "type": "string" } }, "required": ["field_name"], "additionalProperties": false } } }

dictionary is a JSON Schema used to validate parameters in the _search endpoint in the future.

The dictionary content must be a valid JSON Schema.

Demo script:

# Create indices PUT /test1 PUT /test2 # Add docs POST /test1/_doc/1 { "test": "value", "hello": "world", "value": 1 } POST /test2/_doc/2 { "test": "another", "hello": "world", "value": 2 } # Create search app PUT _application/search_application/test-app { "indices": [ "test1", "test2" ], "template": { "script": { "source": { "query": { "term": { "{{field_name}}": "{{field_value}}" } } }, "params": { "field_name": "hello", "field_value": "world" } }, "dictionary": { "properties": { "field_name": { "type": "string" }, "field_value": { "type": "string" } }, "required": ["field_name"], "additionalProperties": false } } } GET _application/search_application/test-app POST /_application/search_application/test-app/_search { "params": { "field_name": "hello", "field_value": "world" } } } POST /_application/search_application/test-app/_search { "params": { "field_name": "test", "field_value": "another" } } }

…ate as a separate field

…emplate parameters

… not specified

…uest

…purposes

carlosdelest · 2023-04-28T13:41:38Z

x-pack/plugin/ent-search/build.gradle

 api project(':modules:lang-mustache')

+ // JSON Schema dependencies
+ implementation "org.apache.commons:commons-lang3:${versions.commons_lang3}"


Added dependencies, which were OK according to core-infra team (Slack conversation)

carlosdelest · 2023-04-28T14:00:45Z

...est/src/yamlRestTest/resources/rest-api-spec/test/entsearch/55_search_application_search.yml

 params:
 field_name: field1
 field_value: value1
+ dictionary:


dictionary now allows the full JSON Schema syntax, including required parameters and limiting additionalProperties.

I think we should change the name from dictionary to param_schema. It is more descriptive and in line with using a JSON schema for validation.

I like your rename suggestion 👍

carlosdelest · 2023-04-28T14:02:10Z

x-pack/plugin/ent-search/build.gradle

 }

-addQaCheckDependencies(project)
+tasks.named("dependencyLicenses").configure {


Licenses for external dependencies needed to be added and some tweaking done to avoid repeating licenses for Jackson.

Some ignored classes to be added to audit as well, already done in other ES modules.

carlosdelest · 2023-04-28T14:02:21Z

x-pack/plugin/ent-search/licenses/commons-lang3-LICENSE.txt

Licenses were duplicated where possible from the codebase, or retrieved in the case of json-schema-validator to ensure the license check is passed.

kderusso

This looks great @carlosdelest , thank you! I have a few comments and questions mostly surrounding tests. Great work 💯

...a/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/20_search_application_put.yml

x-pack/plugin/ent-search/licenses/slf4j-api-NOTICE.txt

kderusso · 2023-04-28T14:38:21Z

...est/src/yamlRestTest/resources/rest-api-spec/test/entsearch/55_search_application_search.yml

 params:
 field_name: field1
 field_value: value1
+ dictionary:


I like your rename suggestion 👍

...est/src/yamlRestTest/resources/rest-api-spec/test/entsearch/55_search_application_search.yml

kderusso · 2023-04-28T14:40:17Z

...est/src/yamlRestTest/resources/rest-api-spec/test/entsearch/60_behavioral_analytics_list.yml

 name: my-test-analytics-collection
 ignore: 404

+ - do:


Thanks for taking care of this!

...arch/src/main/java/org/elasticsearch/xpack/application/search/SearchApplicationTemplate.java

kderusso · 2023-04-28T14:44:53Z

...c/main/java/org/elasticsearch/xpack/application/search/SearchApplicationTemplateService.java

+ public SearchSourceBuilder renderQuery(SearchApplication searchApplication, Map<String, Object> templateParams) throws IOException,
+ ValidationException {


Nitpick:

Suggested change

public SearchSourceBuilder renderQuery(SearchApplication searchApplication, Map<String, Object> templateParams) throws IOException,

ValidationException {

public SearchSourceBuilder renderQuery(SearchApplication searchApplication, Map<String, Object> templateParams)

throws IOException, ValidationException {

# Conflicts: # x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/60_behavioral_analytics_list.yml

elasticsearchmachine · 2023-04-28T19:16:59Z

Hi @carlosdelest, I've created a changelog YAML for you.

…hema-validation' into carlosdelest/add-params-schema-validation

elasticsearchmachine · 2023-05-03T08:45:59Z

Pinging @elastic/enterprise-search (Team:Enterprise Search)

kderusso

Nice work @carlosdelest 🙌

elasticsearchmachine · 2023-05-03T16:29:30Z

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

carlosdelest · 2023-05-08T07:45:32Z

@jimczi , @ioanatia , could I have your review on this? Are we OK with merging this PR and discuss applying the dictionary to default template params as a separate issue?

jimczi

LGTM to unblock

…ms-schema-validation # Conflicts: # x-pack/plugin/ent-search/build.gradle

ioanatia and others added 30 commits April 4, 2023 15:04

Add new template field to SearchApplication

9c047e3

Script should not be nullable

9b6abce

Add tests

7e77e3c

Rename to SearchApplicationTemplate

3dd4f22

Use mustache templates

afb0bc5

No need to modify the system index mapping and store the search templ…

499f22a

…ate as a separate field

Add rest api tests

825db42

More renames from SearchTemplate to SearchApplicationTemplates

dcd5a89

Initial API

39c8a17

simple template execution

916324a

Use SearchApplicationQueryParams to include query parameters, merge t…

bf864cc

…emplate parameters

Make the query work with GET and POST, and assume default params when…

c1119f5

… not specified

Add serialization tests

c53acb4

Add YAML tests

a7d1c66

Refactor TransportAction for legibility

026d5e6

Additional tests for Search Application Query

066d15e

Fix rebasing

9ad8327

Added javadoc, minor formatting changes

c09d6ff

Remove SearchApplicationQueryParams and included them directly in Req…

aa28795

…uest

Fix imports

5f745b4

Change search_application.query to search_application.search for API …

b674aed

…purposes

Add mustache template validation

9d76f5e

Add mustache template validation

d918ef0

Update module dependencies

a50f651

Update search application REST spec to reflect beta, no feature flag

2c7f3df

Apply some PR suggestions from previous PR

ae3c4a3

Store and retrieve JSON Schema

ae6a714

Add equals and hashcode

e4410cc

Formatting

e6a0169

Refactored dependencies

0ed6363

Merge branch 'main' into kderusso/carlos_add_template_schema

6183308

elasticsearchmachine added needs:triage Requires assignment of a team area label v8.9.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Apr 28, 2023

carlosdelest mentioned this pull request Apr 28, 2023

Add template parameters to Search Applications kderusso/elasticsearch#4

Closed

carlosdelest added Team:Enterprise Search Meta label for Enterprise Search team and removed external-contributor Pull request authored by a developer outside the Elasticsearch team labels Apr 28, 2023

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Apr 28, 2023

carlosdelest commented Apr 28, 2023

View reviewed changes

carlosdelest requested review from ioanatia, jimczi and kderusso April 28, 2023 14:03

kderusso reviewed Apr 28, 2023

View reviewed changes

carlosdelest added the >enhancement label Apr 28, 2023

carlosdelest added 2 commits April 28, 2023 20:40

Merge branch 'main' into carlosdelest/add-params-schema-validation

7c8887d

# Conflicts: # x-pack/plugin/ent-search/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/entsearch/60_behavioral_analytics_list.yml

PR review comments

5f9c89e

kderusso added the :EnterpriseSearch/Application Enterprise Search label Apr 28, 2023

Update docs/changelog/95674.yaml

8c2eb54

carlosdelest added 2 commits May 3, 2023 10:41

Fix duplicate security

65b06de

Merge remote-tracking branch 'carlosdelest/carlosdelest/add-params-sc…

12adeb6

…hema-validation' into carlosdelest/add-params-schema-validation

kderusso approved these changes May 3, 2023

View reviewed changes

Remove useless match

2ed00d1

jimczi approved these changes May 10, 2023

View reviewed changes

carlosdelest added 2 commits May 10, 2023 09:58

Merge remote-tracking branch 'origin/main' into carlosdelest/add-para…

faa68aa

…ms-schema-validation # Conflicts: # x-pack/plugin/ent-search/build.gradle

Fix merge

8b7dda9

carlosdelest merged commit 1c053b2 into elastic:main May 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add template parameters to Search Applications #95674

Add template parameters to Search Applications #95674

Uh oh!

carlosdelest commented Apr 28, 2023

carlosdelest Apr 28, 2023

carlosdelest Apr 28, 2023

kderusso Apr 28, 2023

carlosdelest Apr 28, 2023

carlosdelest Apr 28, 2023

kderusso left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kderusso Apr 28, 2023

Uh oh!

kderusso Apr 28, 2023

Uh oh!

kderusso Apr 28, 2023

elasticsearchmachine commented Apr 28, 2023

elasticsearchmachine commented May 3, 2023

kderusso left a comment

elasticsearchmachine commented May 3, 2023

carlosdelest commented May 8, 2023

jimczi left a comment

Labels

5 participants

		public SearchSourceBuilder renderQuery(SearchApplication searchApplication, Map<String, Object> templateParams) throws IOException,
		ValidationException {

Add template parameters to Search Applications #95674

Add template parameters to Search Applications #95674

Uh oh!

Conversation

carlosdelest commented Apr 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

elasticsearchmachine commented Apr 28, 2023

elasticsearchmachine commented May 3, 2023

kderusso left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented May 3, 2023

carlosdelest commented May 8, 2023

jimczi left a comment

Choose a reason for hiding this comment

Labels

5 participants