Avoid unnecessary determinization in index pattern conflict checks #128362
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
jimczi added >bug :Data Management/Indices APIs 
| Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine added the Team:Data Management | Hi @jimczi, I've created a changelog YAML for you. |
javanna reviewed May 23, 2025
pawankartik-elastic approved these changes May 27, 2025
| This should be backported to 9.0.x right? Its existed ever since the lucene 10 upgrade. |
jimczi added a commit to jimczi/elasticsearch that referenced this pull request Jun 2, 2025
…lastic#128362) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
elasticsearchmachine pushed a commit that referenced this pull request Jun 2, 2025
…128362) (#128759) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
mridula-s109 pushed a commit that referenced this pull request Jun 2, 2025
…128362) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jun 3, 2025
…lastic#128362) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
joshua-adams-1 pushed a commit to joshua-adams-1/elasticsearch that referenced this pull request Jun 3, 2025
…lastic#128362) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
Samiul-TheSoccerFan pushed a commit to Samiul-TheSoccerFan/elasticsearch that referenced this pull request Jun 5, 2025
…lastic#128362) Starting with Lucene 10, `CharacterRunAutomaton` is no longer determinized automatically. In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via `Regex#simpleMatchToAutomaton`). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization—an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion. This change removes the unnecessary determinization, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
cbuescher added a commit to cbuescher/elasticsearch that referenced this pull request Sep 5, 2025
We already fixed an issue with this in elastic#128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as elastic#128362 and adds a test that would have triggered this. Closes elastic#133652
cbuescher added a commit that referenced this pull request Sep 8, 2025
We already fixed an issue with this in #128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as #128362 and adds a test that would have triggered this. Closes #133652
cbuescher added a commit to cbuescher/elasticsearch that referenced this pull request Sep 8, 2025
We already fixed an issue with this in elastic#128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as elastic#128362 and adds a test that would have triggered this. Closes elastic#133652
cbuescher added a commit to cbuescher/elasticsearch that referenced this pull request Sep 8, 2025
We already fixed an issue with this in elastic#128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as elastic#128362 and adds a test that would have triggered this. Closes elastic#133652
elasticsearchmachine pushed a commit that referenced this pull request Sep 8, 2025
We already fixed an issue with this in #128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as #128362 and adds a test that would have triggered this. Closes #133652
elasticsearchmachine pushed a commit that referenced this pull request Sep 8, 2025
) * Fix exceptions in index pattern conflict checks (#134231) We already fixed an issue with this in #128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as #128362 and adds a test that would have triggered this. Closes #133652 * Fix test compilation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Starting with Lucene 10,
CharacterRunAutomatonis no longer determinized automatically.In Elasticsearch 9, we adapted to this by eagerly determinizing automatons early (via
Regex#simpleMatchToAutomaton). However, this introduced regression: operations like index template conflict checks, which only require intersection testing, now pay the cost of determinization, an expensive step that wasn’t needed before. In some cases, especially when many wildcard patterns are involved, determinization can even fail due to state explosion.This change removes the unnecessary determinization for index patterns conflict check, restoring the pre-9.0 behavior and allowing valid index templates with many patterns to be registered again.
closes: #127972