Date index name processor
The date_index_name processor is used to point documents to the correct time-based index based on the date or timestamp field within the document. The processor sets the _index metadata field to a date math index name expression. Then the processor fetches the date or timestamp from the field field in the document being processed and formats it into a date math index name expression. The extracted date, index_name_prefix value, and date_rounding value are then combined to create the date math index expression. For example, if the field field contains the value 2023-10-30T12:43:29.000Z and index_name_prefix is set to week_index- and date_rounding is set to w, then the date math index name expression is week_index-2023-10-30. You can use the date_formats field to specify how the date in the date math index expression should be formatted.
The following is the syntax for the date_index_name processor:
{ "date_index_name": { "field": "your_date_field or your_timestamp_field", "date_rounding": "rounding_value" } } Configuration parameters
The following table lists the required and optional parameters for the date_index_name processor.
| Parameter | Required/Optional | Description | |
|---|---|---|---|
field | Required | The date or timestamp field in the incoming document. Supports template snippets. | |
date_rounding | Required | The rounded date format within the index name . Valid values are y (year), M (month), w (week), d (day), h (hour), m (minute), and s (second). | |
date_formats | Optional | An array of date formats used to parse the date or timestamp field. Valid options include a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. Default is yyyy-MM-dd'T'HH:mm:ss.SSSXX. | |
index_name_format | Optional | The date format. Default is yyyy-MM-dd. Supports template snippets. | |
index_name_prefix | Optional | The index name prefix to append before the date. Supports template snippets. | |
description | Optional | A brief description of the processor. | |
if | Optional | A condition for running this processor. | |
ignore_failure | Optional | If set to true, failures are ignored. Default is false. | |
locale | locale | Optional | The locale to use when parsing the month name and week day of the date. Default is ENGLISH. Supports template snippets. |
on_failure | Optional | A list of processors to run if the processor fails. | |
tag | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | |
timezone | Optional | The time zone to use when parsing the date. Default is UTC. |
Using the processor
Follow these steps to use the processor in a pipeline.
Step 1: Create a pipeline.
The following query creates a pipeline, named date-index-name1, that uses the date_index_name processor to index logs into monthly indexes:
PUT /_ingest/pipeline/date-index-name1 { "description": "Create weekly index pipeline", "processors": [ { "date_index_name": { "field": "date_field", "index_name_prefix": "week_index-", "date_rounding": "w", "date_formats": ["YYYY-MM-DD"] } } ] } Step 2 (Optional): Test the pipeline.
It is recommended that you test your pipeline before you ingest documents.
To test the pipeline, run the following query:
POST _ingest/pipeline/date-index-name1/_simulate { "docs": [ { "_index": "testindex1", "_id": "1", "_source": { "date_field": "2023-10-30" } } ] } Response
The following example response confirms that the pipeline is working as expected:
{ "docs": [ { "doc": { "_index": "<week_index-{2023-10-01||/w{yyyy-MM-dd|UTC}}>", "_id": "1", "_source": { "date_field": "2023-10-30" }, "_ingest": { "timestamp": "2023-11-13T18:23:10.408593092Z" } } } ] } Step 3: Ingest a document.
The following query ingests a document into an index named testindex1:
PUT testindex1/_doc/1?pipeline=date-index-name1 { "date_field": "2023-10-30" } Response
The request indexes the document into the index week_index-2023-10-23 and will index all documents with a timestamp within that week into the same index because the pipeline rounds by week.
{ "_index": "week_index-2023-10-30", "_id": "1", "_version": 4, "result": "updated", "_shards": { "total": 2, "successful": 1, "failed": 0 }, "_seq_no": 3, "_primary_term": 1 } Step 4 (Optional): Retrieve the document.
To retrieve the document, run the following query:
GET week_index-2023-10-30/_doc/1