Skip to content

EQL: Response format enhancements #52845

@costin

Description

@costin
  1. implicit sort

Currently, there's no sorting meaning the documents are returned based on their doc_id. Under the assumption that the docs are sorted based on @timestamp, the return order is correct however what if the index is not sorted?

Returning documents always in a predefined order gives predictability and the expense of some performance in case of misconfiguration (no insert time sorting).

  1. Are fields ever needed in the response?

Events always return entire documents. So do sequences and joins and while matching on the field requires the engine to interpret that, response wise, do we ever need to process the returned data?
It seems always the search hits are returned verbatim - there's no way to perform projection and thus no extra processing is needed.
This is important since the engine can simply return the data as is.
A separate point is whether we need to care of whether a field is in the source or not for the join-key since when we do the querying, we can just simply return the _source or, if we have to do field extraction, use doc_values for exact fields.

Related to #49634

  1. How is pagination suppose to work?

Currently our response returns only results but not information on resuming.
The default page seems to be 50 items - how do we handle multiple pages?
Do we want to support pagination or not? I suppose we do but that requires some kind of a cursor.

And search_after parameter is inadequate for sequences and joins since information such as what is the last key we matched on, how to extract such information from the response cannot be passed around.

See: #49594

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions