Merge branch 'main' into esql/lookup-join-grammar

Author: costin

build-tools-internal/src/main/groovy/elasticsearch.ide.gradle

@@ -145,7 +145,7 @@ if (providers.systemProperty('idea.active').getOrNull() == 'true') {
 
  doLast {
  ['main', 'test'].each { sourceSet ->
- modifyXml(".idea/modules/libs/native/elasticsearch.libs.${project.project(':libs:native').name}.${sourceSet}.iml") { xml ->
+ modifyXml(".idea/modules/libs/native/elasticsearch.libs.native.${sourceSet}.iml") { xml ->
  xml.component.find { it.'@name' == 'NewModuleRootManager' }?.'@LANGUAGE_LEVEL' = 'JDK_21_PREVIEW'
  }
  }

docs/changelog/115091.yaml

@@ -0,0 +1,7 @@
+pr: 115091
+summary: Added stricter range type checks and runtime warnings for ENRICH
+area: ES|QL
+type: bug
+issues:
+ - 107357
+ - 116799

docs/changelog/116944.yaml

@@ -0,0 +1,11 @@
+pr: 116944
+summary: "Remove support for type, fields, `copy_to` and boost in metadata field definition"
+area: Mapping
+type: breaking
+issues: []
+breaking:
+ title: "Remove support for type, fields, copy_to and boost in metadata field definition"
+ area: Mapping
+ details: The type, fields, copy_to and boost parameters are no longer supported in metadata field definition
+ impact: Users providing type, fields, copy_to or boost as part of metadata field definition should remove them from their mappings.
+ notable: false

docs/changelog/116970.yaml

@@ -0,0 +1,11 @@
+pr: 116970
+summary: Remove legacy params from range query
+area: Search
+type: breaking
+issues: []
+breaking:
+ title: Remove legacy params from range query
+ area: REST API
+ details: The deprecated range query parameters `to`, `from`, `include_lower`, and `include_upper` are no longer supported.
+ impact: Users should use `lt`, `lte`, `gt`, and `gte` query parameters instead.
+ notable: false

docs/changelog/116995.yaml

@@ -0,0 +1,5 @@
+pr: 116995
+summary: "Apm-data: disable date_detection for all apm data streams"
+area: Data streams
+type: enhancement
+issues: []

docs/changelog/90529.yaml

@@ -0,0 +1,26 @@
+pr: 90529
+summary: Output a consistent format when generating error json
+area: Infra/REST API
+type: "breaking"
+issues:
+ - 89387
+breaking:
+ title: Error JSON structure has changed when detailed errors are disabled
+ area: REST API
+ details: |-
+ This change modifies the JSON format of error messages returned to REST clients
+ when detailed messages are turned off.
+ Previously, JSON returned when an exception occurred, and `http.detailed_errors.enabled: false` was set,
+ just consisted of a single `"error"` text field with some basic information.
+ Setting `http.detailed_errors.enabled: true` (the default) changed this field
+ to an object with more detailed information.
+ With this change, non-detailed errors now have the same structure as detailed errors. `"error"` will now always
+ be an object with, at a minimum, a `"type"` and `"reason"` field. Additional fields are included when detailed
+ errors are enabled.
+ To use the previous structure for non-detailed errors, use the v8 REST API.
+ impact: |-
+ If you have set `http.detailed_errors.enabled: false` (the default is `true`)
+ the structure of JSON when any exceptions occur now matches the structure when
+ detailed errors are enabled.
+ To use the previous structure for non-detailed errors, use the v8 REST API.
+ notable: false

docs/reference/cluster/allocation-explain.asciidoc

@@ -159,6 +159,8 @@ node.
 <5> The decider which led to the `no` decision for the node.
 <6> An explanation as to why the decider returned a `no` decision, with a helpful hint pointing to the setting that led to the decision. In this example, a newly created index has <<indices-get-settings,an index setting>> that requires that it only be allocated to a node named `nonexistent_node`, which does not exist, so the index is unable to allocate.
 
+See https://www.youtube.com/watch?v=5z3n2VgusLE[this video] for a walkthrough of troubleshooting a node and index setting mismatch.
+
 [[maximum-number-of-retries-exceeded]]
 ====== Maximum number of retries exceeded
 
@@ -235,7 +237,9 @@ primary shard that was previously allocated.
 ----
 // NOTCONSOLE
 
-TIP: If a shard is unassigned with an allocation status of `no_valid_shard_copy`, then you should <<fix-cluster-status-recover-nodes,make sure that all nodes are in the cluster>>. If all the nodes containing in-sync copies of a shard are lost, then you can <<fix-cluster-status-restore,recover the data for the shard>>.
+If a shard is unassigned with an allocation status of `no_valid_shard_copy`, then you should <<fix-cluster-status-recover-nodes,make sure that all nodes are in the cluster>>. If all the nodes containing in-sync copies of a shard are lost, then you can <<fix-cluster-status-restore,recover the data for the shard>>. 
+
+See https://www.youtube.com/watch?v=6OAg9IyXFO4[this video] for a walkthrough of troubleshooting `no_valid_shard_copy`.
 
 ===== Unassigned replica shard
 

docs/reference/esql/esql-enrich-data.asciidoc

@@ -138,8 +138,33 @@ include::{es-ref-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=upda
 
 include::../ingest/enrich.asciidoc[tag=update-enrich-policy]
 
-==== Limitations
+==== Enrich Policy Types and Limitations
+The {esql} `ENRICH` command supports all three enrich policy types:
+
+`geo_match`::
+Matches enrich data to incoming documents based on a <<query-dsl-geo-shape-query,`geo_shape` query>>.
+For an example, see <<geo-match-enrich-policy-type>>.
+
+`match`::
+Matches enrich data to incoming documents based on a <<query-dsl-term-query,`term` query>>.
+For an example, see <<match-enrich-policy-type>>.
+
+`range`::
+Matches a number, date, or IP address in incoming documents to a range in the
+enrich index based on a <<query-dsl-term-query,`term` query>>. For an example,
+see <<range-enrich-policy-type>>.
+
 // tag::limitations[]
-The {esql} `ENRICH` command only supports enrich policies of type `match`.
-Furthermore, `ENRICH` only supports enriching on a column of type `keyword`.
+While all three enrich policy types are supported, there are some limitations to be aware of:
+
+* The `geo_match` enrich policy type only supports the `intersects` spatial relation.
+* It is required that the `match_field` in the `ENRICH` command is of the correct type.
+For example, if the enrich policy is of type `geo_match`, the `match_field` in the `ENRICH`
+command must be of type `geo_point` or `geo_shape`.
+Likewise, a `range` enrich policy requires a `match_field` of type `integer`, `long`, `date`, or `ip`,
+depending on the type of the range field in the original enrich index.
+* However, this constraint is relaxed for `range` policies when the `match_field` is of type `KEYWORD`.
+In this case the field values will be parsed during query execution, row by row.
+If any value fails to parse, the output values for that row will be set to `null`,
+an appropriate warning will be produced and the query will continue to execute.
 // end::limitations[]

docs/reference/esql/esql-language.asciidoc

@@ -14,6 +14,7 @@ Detailed reference documentation for the {esql} language:
 * <<esql-enrich-data>>
 * <<esql-process-data-with-dissect-and-grok>>
 * <<esql-implicit-casting>>
+* <<esql-time-spans>>
 
 include::esql-syntax.asciidoc[]
 include::esql-commands.asciidoc[]
@@ -23,3 +24,4 @@ include::multivalued-fields.asciidoc[]
 include::esql-process-data-with-dissect-grok.asciidoc[]
 include::esql-enrich-data.asciidoc[]
 include::implicit-casting.asciidoc[]
+include::time-spans.asciidoc[]

docs/reference/esql/esql-syntax.asciidoc

@@ -157,21 +157,15 @@ FROM employees
 ==== Timespan literals
 
 Datetime intervals and timespans can be expressed using timespan literals.
-Timespan literals are a combination of a number and a qualifier. These
-qualifiers are supported:
-
-* `millisecond`/`milliseconds`/`ms`
-* `second`/`seconds`/`sec`/`s`
-* `minute`/`minutes`/`min`
-* `hour`/`hours`/`h`
-* `day`/`days`/`d`
-* `week`/`weeks`/`w`
-* `month`/`months`/`mo`
-* `quarter`/`quarters`/`q`
-* `year`/`years`/`yr`/`y`
+Timespan literals are a combination of a number and a temporal unit. The
+supported temporal units are listed in <<esql-time-spans-table, time span unit>>.
+More examples of the usages of time spans can be found in
+<<esql-time-spans, Use time spans in ES|QL>>.
+
 
 Timespan literals are not whitespace sensitive. These expressions are all valid:
 
 * `1day`
 * `1 day`
 * `1 day`
+