Add mappings for enrich fields #96056
Conversation
dnhatn added :Data Management/Ingest Node 
| Hi @dnhatn, I've created a changelog YAML for you. |
elasticsearchmachine added the Team:Data Management | Pinging @elastic/es-data-management (Team:Data Management) |
| @jbaiera @martijnvg I have updated this PR to maintain the current behavior. Populating the mapping for enrich fields is now a best-effort, which means we will ignore mapping conflicts that arise between the enrich fields of these source indices. I will have a follow-up to adjust this validation. Would you please take a look? Thank you! |
There was a problem hiding this comment.
LGTM
I think having the enrich fields mapped is a good change in general.
Enabling doc values by default does increase the size of the enrich index.
This shouldn't be too bad, given that an enrich index is designed to contain reference data set for lookups, an enrich index usually isn't that large. And it allows enrich to be used at query time for join like functionality. Down the line enrich indices could be configured to not store _source or the synthesise _source at runtime.
x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichPolicyRunner.java Outdated Show resolved Hide resolved
x-pack/plugin/enrich/src/test/java/org/elasticsearch/xpack/enrich/EnrichPolicyRunnerTests.java Show resolved Hide resolved
x-pack/plugin/enrich/src/main/java/org/elasticsearch/xpack/enrich/EnrichPolicyRunner.java Show resolved Hide resolved
| @nik9000 @martijnvg Thank you for reviews. |
This reverts commit 00afc5b.
We implemented a mapping change to enrich indices to support enrich in ESQL. Unfortunately, that change does not interact well with object fields. While we are actively addressing this issue, a comprehensive solution will take some time. To mitigate the impact of this bug, this PR will revert the mapping change for 8.9 and 8.10. The reason is that enrich for ESQL isn't required until 8.11. Relates #98019
We implemented a mapping change to enrich indices to support enrich in ESQL. Unfortunately, that change does not interact well with object fields. While we are actively addressing this issue, a comprehensive solution will take some time. To mitigate the impact of this bug, this PR will revert the mapping change for 8.9 and 8.10. The reason is that enrich for ESQL isn't required until 8.11. Relates elastic#98019
We implemented a mapping change to enrich indices to support enrich in ESQL. Unfortunately, that change does not interact well with object fields. While we are actively addressing this issue, a comprehensive solution will take some time. To mitigate the impact of this bug, this PR will revert the mapping change for 8.9 and 8.10. The reason is that enrich for ESQL isn't required until 8.11. Relates #98019
4 participants
We are developing a new component that enriches the search results during query time. The idea is to leverage the enrich policies and indices already used during indexing time. However, for the new component to function properly, we require the mapping types and doc_values of the enrich fields.
Currently, an enrich index can have multiple source indices, and any mapping conflicts that arise between the enrich fields of these source indices are ignored. However, with the upcoming changes, the enrich policy execution will fail if any mapping conflicts occur. We consider this change not as a breaking change, but rather as addressing a validation bug, as enrich fields should be consistent across source indices.