You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[DOCS] Improve reroute processor documentation structure and clarity (#136189)
* [DOCS] Improve reroute processor documentation structure and clarity Add detailed explanation of document processing after rerouting, including: - How documents are processed through the new target's ingest pipeline - Support for chaining multiple reroutes across pipelines - Automatic cycle detection and failure handling Reorganize content with clear section headings for better scannability: - Operating modes (explicit destination vs data stream mode) - Pipeline execution behavior - Document processing after rerouting - Field updates - Permissions - Configuration options - Examples Improve readability throughout with clearer language and better formatting. Fixes feedback from issue raised by @ivosh requesting clarification on post-reroute document processing behavior.
The `reroute` processor routes a document to a different target index or data stream during ingestion. This allows you to dynamically direct documents to appropriate destinations based on their content or metadata.
9
13
10
-
The `reroute` processor allows to route a document to another target index or data stream. It has two main modes:
14
+
## Operating modes [reroute-operating-modes]
11
15
12
-
When setting the `destination` option, the target is explicitly specified and the `dataset` and `namespace` options can’t be set.
16
+
The reroute processor has two main modes of operation.
13
17
14
-
When the `destination` option is not set, this processor is in a data stream mode. Note that in this mode, the `reroute` processor can only be used on data streams that follow the [data stream naming scheme](docs-content://reference/fleet/data-streams.md#data-streams-naming-scheme). Trying to use this processor on a data stream with a non-compliant name will raise an exception.
The name of a data stream consists of three parts: `<type>-<dataset>-<namespace>`. See the [data stream naming scheme](docs-content://reference/fleet/data-streams.md#data-streams-naming-scheme) documentation for more details.
20
+
When you set the `destination` option, the processor routes the document to an explicitly specified target. In this mode, you cannot use the `dataset` and `namespace` options.
17
21
18
-
This processor can use both static values or reference fields from the document to determine the `dataset` and `namespace` components of the new target. See [Table 40](#reroute-options) for more details.
22
+
### Data stream mode [reroute-data-stream-mode]
23
+
24
+
When the `destination` option is not set, the processor operates in data stream mode. This mode requires that your data stream follows the [data stream naming scheme](docs-content://reference/fleet/data-streams.md#data-streams-naming-scheme).
25
+
26
+
Data stream names consist of three parts: `<type>-<dataset>-<namespace>`. See the [data stream naming scheme](docs-content://reference/fleet/data-streams.md#data-streams-naming-scheme) documentation for details.
27
+
28
+
In data stream mode, you can use static values or field references to determine the `dataset` and `namespace` components of the new target. The processor will automatically construct the full data stream name.
19
29
20
30
::::{note}
21
-
It’s not possible to change the `type` of the data stream with the `reroute` processor.
31
+
You cannot change the `type`component of a data stream with the `reroute` processor. Attempting to use this processor on a data stream with a non-compliant name will raise an exception.
When a `reroute` processor executes, it immediately stops processing the current pipeline:
37
+
38
+
- All subsequent processors in the current pipeline are skipped, including the final pipeline
39
+
- If the current pipeline is called from a [Pipeline processor](/reference/enrich-processor/pipeline-processor.md), the calling pipeline is also skipped
40
+
- Only one `reroute` processor can execute per document, allowing you to define mutually exclusive routing conditions similar to if-else logic
41
+
42
+
## Document processing after rerouting [reroute-document-processing]
43
+
44
+
After a document is rerouted, it is processed through the ingest pipeline associated with the new target destination (or dataset and namespace). This enables the document to be transformed according to the rules specific to its new destination.
45
+
46
+
If the new ingest pipeline also contains a `reroute` processor, the document can be rerouted again to yet another target. This chaining continues as long as each successive pipeline contains a reroute processor that matches the document.
24
47
25
-
After a `reroute` processor has been executed, all the other processors of the current pipeline are skipped, including the final pipeline. If the current pipeline is executed in the context of a [Pipeline](/reference/enrich-processor/pipeline-processor.md), the calling pipeline will be skipped, too. This means that at most one `reroute` processor is ever executed within a pipeline, allowing to define mutually exclusive routing conditions, similar to a if, else-if, else-if, … condition.
48
+
The system detects routing cycles automatically. If a document would be routed back to a target it has already been processed through, an exception is thrown and the document fails to be ingested.
26
49
27
-
The reroute processor ensures that the `data_stream.<type|dataset|namespace>` fields are set according to the new target. If the document contains a `event.dataset` value, it will be updated to reflect the same value as `data_stream.dataset`.
50
+
## Field updates [reroute-field-updates]
28
51
29
-
Note that the client needs to have permissions to the final target. Otherwise, the document will be rejected with a security exception which looks like this:
52
+
The reroute processor automatically updates document fields to reflect the new target:
53
+
54
+
- Sets `data_stream.type`, `data_stream.dataset`, and `data_stream.namespace` according to the new target
55
+
- If the document contains an `event.dataset` field, updates it to match `data_stream.dataset`
56
+
57
+
## Permissions [reroute-permissions]
58
+
59
+
The client ingesting the document must have permissions to write to the final target destination. Without proper permissions, the document will be rejected with a security exception:
30
60
31
61
```js
32
62
{"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [8-dt9H8BqGblnY2uSI--] of user [elastic/fleet-server] on indices [logs-foo-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}
33
63
```
34
64
65
+
## Configuration options [reroute-options]
66
+
35
67
$$$reroute-options$$$
36
68
37
69
| Name | Required | Default | Description |
38
70
| --- | --- | --- | --- |
39
-
|`destination`| no | - | A static value for the target. Can’t be set when the `dataset` or `namespace` option is set. |
71
+
|`destination`| no | - | A static value for the target. Cannot be set when the `dataset` or `namespace` option is set. |
40
72
|`dataset`| no |`{{data_stream.dataset}}`| Field references or a static value for the dataset part of the data stream name. In addition to the criteria for [index names](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create), cannot contain `-` and must be no longer than 100 characters. Example values are `nginx.access` and `nginx.error`.<br><br>Supports field references with a mustache-like syntax (denoted as `{{double}}` or `{{{triple}}}` curly braces). When resolving field references, the processor replaces invalid characters with `_`. Uses the `<dataset>` part of the index name as a fallback if all field references resolve to a `null`, missing, or non-string value.<br> |
41
73
|`namespace`| no |`{{data_stream.namespace}}`| Field references or a static value for the namespace part of the data stream name. See the criteria for [index names](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) for allowed characters. Must be no longer than 100 characters.<br><br>Supports field references with a mustache-like syntax (denoted as `{{double}}` or `{{{triple}}}` curly braces). When resolving field references, the processor replaces invalid characters with `_`. Uses the `<namespace>` part of the index name as a fallback if all field references resolve to a `null`, missing, or non-string value.<br> |
42
74
|`description`| no | - | Description of the processor. Useful for describing the purpose of the processor or its configuration. |
@@ -45,7 +77,11 @@ $$$reroute-options$$$
45
77
|`on_failure`| no | - | Handle failures for the processor. See [Handling pipeline failures](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md#handling-pipeline-failures). |
46
78
|`tag`| no | - | Identifier for the processor. Useful for debugging and metrics. |
47
79
48
-
The `if` option can be used to define the condition in which the document should be rerouted to a new target.
Use the `if` option to define conditions for when a document should be rerouted:
49
85
50
86
```js
51
87
{
@@ -57,9 +93,11 @@ The `if` option can be used to define the condition in which the document should
57
93
}
58
94
```
59
95
60
-
The dataset and namespace options can contain either a single value or a list of values that are used as a fallback. If a field reference evaluates to `null`, is not present in the document, the next value or field reference is used. If a field reference evaluates to a non-`String` value, the processor fails.
96
+
### Fallback values [reroute-fallback-values]
97
+
98
+
The `dataset` and `namespace` options accept either a single value or a list of values used as fallbacks. If a field reference evaluates to `null` or is missing, the processor tries the next value in the list.
61
99
62
-
In the following example, the processor would first try to resolve the value for the `service.name`field to determine the value for `dataset`. If that field resolves to `null`, is missing, or is a non-string value, it would try the next element in the list. In this case, this is the static value `"generic`". The `namespace` option is configured with just a single static value.
100
+
In this example, the processor first attempts to resolve `service.name`for the dataset. If that field is `null`, missing, or non-string, it falls back to the static value `"generic"`:
63
101
64
102
```js
65
103
{
@@ -73,3 +111,4 @@ In the following example, the processor would first try to resolve the value for
73
111
}
74
112
```
75
113
114
+
If a field reference evaluates to a non-string value, the processor fails.
0 commit comments