Skip to content

Conversation

@AlexisSouquiere
Copy link

@AlexisSouquiere AlexisSouquiere commented Dec 2, 2025

The GrokFilter is really useful to parse unstructured data. When I tried to use it with a *BytesArrayInputReader I missed 2 things:

  • A way to put the extracted fields in a given field and not always have to put these field at the top level.
    To solve this, I added support of the target field:

From:

{ "message": "1970-01-01 00:00:00,000 INFO a dummy log message", "timestamp": "1970-01-01 00:00:00,000", "level": "INFO" }

To:

"filters.grok.target": "parsed"
{ "message": "1970-01-01 00:00:00,000 INFO a dummy log message", "parsed": { "message": "a dummy log message", "timestamp": "1970-01-01 00:00:00,000", "level": "INFO" } }
  • I need to read a file as a whole to generate 1 single event. When working with *BytesArrayInputRead, SplitFilter to parse the file content (on \n for instance), we can have a string array on which we want to apply the GrokFilter.
    To solve this, I added support of array string input as well as the existing string input.

Note: The following example is not really relevant as we would generally read a log file line by line (it shows the objective)

From:

{ "message": ["1970-01-01 00:00:00,000 INFO a dummy log message", "1970-01-01 01:00:00,000 INFO a second dummy log message", "1970-01-01 02:00:00,000 INFO a third dummy log message"] }

To:

{ "parsed": [{ "message": "a dummy log message", "timestamp": "1970-01-01 00:00:00,000", "level": "INFO" }, { "message": "a second dummy log message", "timestamp": "1970-01-01 01:00:00,000", "level": "INFO" }, { "message": "a third dummy log message", "timestamp": "1970-01-01 00:00:00,000", "level": "INFO" }] }

This PR enhances the GrokFilter while keeping the original behaviour. Look forward to having your feedback !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant