Eql Sampling #85206

astefan · 2022-03-22T09:48:12Z

A sample searches for events matching the declared filters in all possible permutations. The result of a sample is identical in structure with the one of a sequence, but for each combination of join key values, if there is at least one match, the result will contain only one events combination matching the sample (as opposed to sequences where all results are returned).

costin

Left some comments.

costin · 2022-03-28T07:09:45Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/BaseEqlSpecTestCase.java

 final long[] ids = new long[len];
 for (int i = 0; i < len; i++) {
- Object field = events.get(i).sourceAsMap().get(tiebreaker());
+ Object field = events.get(i).sourceAsMap().get(tiebreaker() == null ? idField() : tiebreaker());


tiebreaker() could be cached and the condition evaluated outside the loop - just like the previous change.

costin · 2022-03-28T07:20:34Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/plan/logical/Join.java

 import static java.util.Collections.singletonList;

-public class Join extends LogicalPlan {
+public class Join extends Sampling {


Join and Sampling share common properties so it makes sense to reuse code between them however due to the difference in semantics, it's better the classes don't extend one another.
Sequence extends Join so if Join extends Sampling it means Sequence is a type of sampling.
Also Join currently exists as a logical plan without being exposed in the query .
My suggestion is to extract the common join properties into a separate class (say AbstractJoin) and keep Sequence/Join separately from Sample.

costin · 2022-03-28T07:21:36Z

x-pack/plugin/eql/src/main/antlr/EqlBase.g4

 : sequence
 | join
 | eventQuery
+ | sampling


Sample I think works better than sampling - we use sequence as oppose to sequencing, join instead of joining.

costin · 2022-03-28T07:25:47Z

...ql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/AggregatedQueryRequest.java

+ return new SearchSourceBuilder(in);
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException(e);


Use EQL exceptions to wrap the underlying exception - adds consistency and proper handling.

costin · 2022-03-28T07:27:37Z

...ql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/AggregatedQueryRequest.java

+ * Sets keys / terms to filter on in the final stage filtering (where actual events are gathered).
+ * Can be removed through null.
+ */
+ public void singleKeysPair(final List<Object> compositeKeyValues, int maxStages) {


singleKeyPair

costin · 2022-03-28T07:27:46Z

...ql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/AggregatedQueryRequest.java

+ * Sets keys / terms to filter on in an intermediate stage filtering.
+ * Can be removed through null.
+ */
+ public void multipleKeysPairs(List<Map<String, Object>> values, List<String> previousCriterionKeys) {


multipleKeyPairs

costin · 2022-03-28T07:29:22Z

...ql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/AggregatedQueryRequest.java

+ /*
+ * Not a great way of getting a copy of a SearchSourceBuilder
+ */
+ private SearchSourceBuilder copySource() {


Better to make this static and parameterize it - not for performance but to indicate that it copies any given source as oppose to copying the existing source and then returning a new instance.
Make the copy/modify/reassign code flow better.

costin · 2022-03-28T07:32:28Z

...gin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/SamplingCriterion.java

+
+import java.util.List;
+
+public class SamplingCriterion<Q extends QueryRequest> {


A common base Criterion class which contains the keys, key extraction and the key size is useful not just in sharing code but also in ExecutionManager.

Add EqlSampleDataLoader Introduce AbstractJoin having Join and Sample as subclasses Rename Sampling to Sample Rename some methods Parametrize copySource

…o sampling_in_eql

costin

Left another round of comments.

costin · 2022-04-19T18:33:26Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/DataLoader.java

A better name might be samples (since sampling means the process of selecting the sample).

costin · 2022-04-19T18:42:59Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/analysis/PostAnalyzer.java

Why is this relevant only for the Sequence but not for Sample?

Because in a Sample there is no notion of limit (so far at least) and what this is doing is to add a limit plan to the tree.

does that mean that fetch_size is ignored for sample? Is there a global limit for join keys then or will sample just always return all the keys?

fetch_size is different. What you probably meant is size. And yes, for now, there is no limit and it will return all keys.

I will add fetch_size support in a new commit. This one is about the size of the results page returned by the composite aggregation and it has a very similar meaning in sequences as well.

yes, I had size in mind. Isn't support for size somewhat crucial? Otherwise a query might easily OOM due to too many samples.

+1 on addressing limit/size in a separate PR if it's non-simplistic - this one is already quite big.
Potentially open a separate branch from sample to avoid any impact on releases.

costin · 2022-04-19T18:49:41Z

...lugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/SampleCriterion.java

SampleCriterion extends Criterion<AggregatedQueryRequest>
Since the AggregatedQueryRequest is not used anywhere else it could be renamed to SampleQueryRequest.

costin · 2022-04-19T18:52:17Z

...gin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/SequenceCriterion.java

SequenceCriterion extends Criterion<BoxedQueryRequest>

costin · 2022-04-19T18:55:52Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

final var hits = ..

costin · 2022-04-19T18:57:20Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

could use better names to avoid the comments - int reponseIndex / currentResponse, int groupIndex, currentGroup

costin · 2022-04-19T18:59:33Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

var next = page.size() == MAX_PAGE_SIZE ? page : stack.pop(); log.trace("Final stage... getting next page of the " + (next == page ? "current" : "previous") + " page"); nextPage(listener, next);

costin · 2022-04-19T19:01:33Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SamplePayload.java

Does it make sense to introduce a new type for Sample?

It may make sense in the future, not sure. I added it. But the payload is still a List of Sequences (the result of a Sample has an identical structure to a Sequence) and it's a bit more involved to add a Sample payload.

costin · 2022-04-19T19:05:52Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/planner/QueryFolder.java

This seems better suited as a logical optimization rule as a oppose to a folding one.

Actually, this is a leftover from one of the initial ideas on how to introduce bucket extractors. The name of this rule is a misnomer, it doesn't actually propagate the composite keys, but more creating extractors. I've removed this rule, the logic that's adding the extractors is in ExecutionManager.

Added tests

…o sampling_in_eql

elasticmachine · 2022-05-04T15:57:56Z

Pinging @elastic/es-ql (Team:QL)

elasticsearchmachine · 2022-05-04T15:58:17Z

Hi @astefan, I've created a changelog YAML for you.

astefan · 2022-05-04T16:24:33Z

@elasticmachine update branch

…o sampling_in_eql

luigidellaquila · 2022-05-10T14:11:30Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/Criterion.java

-
-import java.util.List;

 public class Criterion<Q extends QueryRequest> {


This class does not seem to have much meaning after this refactoring, probably we can just remove it.

Actually, I've moved to this class a common part of SampleCriterion and SequenceCriterion - the keySize.

Luegg

Lots to digest here so I'm only leaving some preliminary comments for now.

Luegg · 2022-05-10T06:38:44Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/BaseEqlSpecTestCase.java

+ String[] splitNames = index.split(",");
+ int i = 0;
+
+ while (shouldLoadData && i < splitNames.length) {


I think Stream.allMatch would help a lot here to communicate the intent. As in:

boolean shouldLoadData = Arrays.stream(index.split(",")) .allMatch( indexName -> provisioningClient.performRequest(new Request("HEAD", "/" + unqualifiedIndexName(indexName))) .getStatusLine() .getStatusCode() == 200 );

Luegg · 2022-05-10T06:51:56Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/BaseEqlSpecTestCase.java

 private long[] extractIds(List<Map<String, Object>> events) {
 final int len = events.size();
 final long[] ids = new long[len];
+ String idField = tiebreaker() == null ? idField() : tiebreaker();


I think this could be clearer if this logic is pushed to idField(). The default implementation can return tiebreaker() and EqlSampleTestCase.idField() stays as is.

Luegg · 2022-05-10T07:04:34Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/EqlSampleTestCase.java

+
+ @Override
+ protected String tiebreaker() {
+ return null;


Why is tiebreaker not used for sampling queries?

In general, a sample has no sense of chronological status. A sample is an unordered sequence of events and, in a way, is a less restrictive sequence.

In this particular case (EqlSampleTestCase) the id (which should be unique for all documents in the testing scenario) makes more sense for identifying the matching documents than a tiebreaker.

Luegg · 2022-05-11T06:41:18Z

x-pack/plugin/eql/qa/common/src/main/resources/test_sample.toml

@@ -0,0 +1,252 @@
+[[queries]]


One case I'm missing (and currently fails) is a sample without a join key as in sample [any where true] [any where true].

another one is composition with head/tail. It looks like these two are ignored.

You're right. I'll add them.

@Luegg thinking about this some more, samples do not make sense without a join key. Without the restriction of chronological order and without an element that ties the two events together (the join key), the returned events are just a collection of random (technically not random) events.
I've added 0-join keys as a restriction.
Same for head and tail (or pipes in general) - a limitation for the moment. head and tail may be part of the size/limit story but these details haven't been ironed out yet.

Luegg · 2022-05-11T07:06:31Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/analysis/PostAnalyzer.java

+ });
+ }

+ hasJoin.set(hasJoin.get() || plan.anyMatch(Sample.class::isInstance));


Suggested change

hasJoin.set(hasJoin.get() || plan.anyMatch(Sample.class::isInstance));

boolean hasJoin = plan.anyMatch(AbstractJoin.class::isInstance)

I've done it the way it is in the PR because the first anyMatch is already traversing the tree and I wanted to take advantage of that step and not do the traversal twice in all cases.

Luegg · 2022-05-11T11:24:43Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ // get results through search (to keep using PIT)
+ client.fetchHits(hits(samples), ActionListeners.map(listener, listOfHits -> {
+ SamplePayload payload = new SamplePayload(samples, listOfHits, false, timeTook());
+ return payload;


payload can be inlined and the curly braces of the lambda can be dropped

Luegg · 2022-05-11T11:53:34Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ * Starting point of the iterator, which also goes through all the criterions and gathers initial results pages.
+ */
+ private void advance(ActionListener<Payload> listener) {
+ int currentStage = stack.size();


"stage" and "criteria" seems to be used interchangeably. I think it would be easier to understand if only one of the two terms is used in the code base. Stage is quite an overloaded term.

Luegg · 2022-05-11T12:31:51Z

...in/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/SampleQueryRequest.java

+ * Sets keys / terms to filter on in an intermediate stage filtering.
+ * Can be removed through null.
+ */
+ public void multipleKeyPairs(List<Map<String, Object>> values, List<String> previousCriterionKeys) {


The use of "key" and "value" is somewhat inconsistent which makes it hard to follow what's being passed. As far as I understand, values is a list of join keys and previousCriterionKeys is the list of join key field names. Also, on the call site both arguments are referred to as just "keys": request.multipleKeyPairs(previousCriterion.keys(previousResults.hits), previousResults.keys).

In general, "key" is the name of the field, "value" is its value.

Luegg · 2022-05-11T12:41:42Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/analysis/PostAnalyzer.java

does that mean that fetch_size is ignored for sample? Is there a global limit for join keys then or will sample just always return all the keys?

Luegg · 2022-05-11T12:53:25Z

...in/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/SampleQueryRequest.java

+ }
+ }
+
+ SearchSourceBuilder newSource = copySource(searchSource);


Maybe it would be easier to create new search source builders from scratch instead of hacky copy. This way it would also be more explicit what needs to end up in the source for the according query. E.g. currently it also inherits the fields from the original query which is not really needed as far as I understand.

When I took the decision on hacky copying the source, I realized it was close to impossible to copy a SearchSourceBuilder because of queryBuilder (which has many implementations). You'd have to copy a BoolQueryBuilder and ExistsQueryBuilder and other possible such builders we use in functions and operators.

astefan · 2022-05-12T05:21:00Z

@elasticmachine update branch

astefan · 2022-05-12T05:38:05Z

@elasticmachine run elasticsearch-ci/part-2

astefan · 2022-05-12T05:38:17Z

@elasticmachine run elasticsearch-ci/docs-check

luigidellaquila

Adding some comments

luigidellaquila · 2022-05-12T13:56:08Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ request = criterion.firstQuery();
+ }
+
+ final SampleQueryRequest rr = request;


This can be avoided by making request final (it's never reassigned)

luigidellaquila · 2022-05-12T13:59:03Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ InternalComposite composite = (InternalComposite) a;
+ log.trace("Advancing.... found [{}] hits", composite.getBuckets().size());
+ Page nextPage = new Page(composite, rr);
+ if (nextPage != null && nextPage.size() > 0) {


Suggested change

if (nextPage != null && nextPage.size() > 0) {

if (nextPage.size() > 0) {

nextPage cannot be null here

luigidellaquila · 2022-05-12T14:11:32Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ final SampleQueryRequest rr = request;
+ log.trace("Querying stage [{}] {}", currentStage, request);
+ client.query(request, wrap(r -> {
+ Aggregation a = r.getAggregations().get(COMPOSITE_AGG_NAME);


If I'm not wrong, this logic is exactly the same as nextPage() (apart from the log message), probably it's worth extracting it as a method and use it in both contexts

luigidellaquila · 2022-05-12T14:18:15Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SamplePayload.java

+
+class SamplePayload extends AbstractPayload {
+
+ private final List<org.elasticsearch.xpack.eql.action.EqlSearchResponse.Sequence> values;


Are values conceptually Sequences?
Structurally, for transport purposes, Sequence does the job, even though the events are supposed to be a Set (not a List).
Just wondering if it makes sense to keep the two concepts separate and have a specialized EqlSearchResponse.Sample

Yes, the response of a "sample" is identical in structure with the one of a "sequence".

luigidellaquila · 2022-05-12T14:46:38Z

...ugin/eql/src/test/java/org/elasticsearch/xpack/eql/execution/sample/SampleIteratorTests.java

+ public void testMatchSample() {
+ assertEquals(
+ asSearchHitsList(2, 1, 3),
+ matchSample(asList(asSearchHitsList(1, 1, 2), asSearchHitsList(1, 1, 1), asSearchHitsList(1, 1, 3)), 3)


Can these search hits contain duplicates in practice?
It makes sense to test the algorithm in this case as well, just wondering if the algorithm can be simplified if it's not true (ie. avoid the backtracking by sorting the searchHits lists by size)

I thought a bit about this, IMHO we should not test duplicates here. They are not allowed by construction (each query in the final reduce step should not produce duplicates), and they would also invalidate the general approach (ie. if you had duplicates, you could not rely on size=n and terminate_after=n in the final queries, without losing results).
The backtracking algorithm tolerates duplicate results, but IMHO it can be improved, and this test could fail with a different, yet valid, algorithm that does not tolerate duplicates.

luigidellaquila · 2022-05-16T08:04:55Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ int initialSize = samples.size();
+ client.multiQuery(searches, ActionListener.wrap(r -> {
+ List<List<SearchHit>> finalSamples = new ArrayList<>();
+ List<List<SearchHit>> sample = new ArrayList<>(maxStages);


Probably we can use List<Set<SearchHit>> here. It would make some operations faster (ie. contains()) and it would make the intention more clear.

I don't think a Set can be used here. The order of each sub-list of SearchHits should be kept. Each sub-list matches one filter of the query. The position of the SearchHit should correspond to the position of the query filter.

Isn't the order of requests kept in the first List...? The Set contains the results for a single query filter, that should not be so relevant as long as you can decide which filter you run first

You're right. That order shouldn't matter.

…o sampling_in_eql

costin

LGTM! This looks great!

costin · 2022-05-20T11:32:07Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/EqlRestTestCase.java

+ assert400BadRequest(test[0], test[1]);
 }

+ bulkIndex("""


👍 for using """

costin · 2022-05-20T11:34:01Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/EqlSampleTestCase.java

+ @Override
+ protected int requestFetchSize() {
+ // a more relevant fetch_size value for Samples, from algorithm point of view, so we'll mostly test this value
+ if (frequently()) {


nit - the ternary expression makes this a one-liner: return frequently() ? 2 : super.requestFetchSize()

costin · 2022-05-20T11:36:54Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/analysis/PostAnalyzer.java

+1 on addressing limit/size in a separate PR if it's non-simplistic - this one is already quite big.
Potentially open a separate branch from sample to avoid any impact on releases.

costin · 2022-05-20T11:40:17Z

...ugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/assembler/ExecutionManager.java

+ // search query
+ if (query instanceof EsQueryExec esQueryExec) {
+ SampleQueryRequest firstQuery = new SampleQueryRequest(
+ () -> wrapAsFilter(esQueryExec.source(session, false)),


Please create an issue for it so it doesn't get lost.

costin · 2022-05-20T11:53:20Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/planner/Mapper.java

- s.direction(),
- s.maxSpan()
- );
+ if (p instanceof Sequence sequence) {


Nit, switch the check: check on Sample and return it otherwise fallback to the existing code - less code modified.

rw-access · 2022-05-25T17:14:44Z

x-pack/plugin/eql/qa/common/src/main/resources/test_sample.toml

+ [any where port > 100] by op_sys 
+ [any where bool == true] by os
+'''
+expected_event_ids = [17,26,16,


hey @astefan! it's been a while! 🙂
cool feature, looks similar to Endgame EQL join but without all the baggage that word brings, so I'm glad to see the name change.

I think it behaves how I would intuitively expect, just want to double check one property: how are the resulting events ordered? Is it (1) in declaration order, parallel to the structure of the query? Or is it (2) chronological order, not mirroring the structure.

I can understand arguments for both, and I think (1) makes the most intuitive sense to me and that was how the Endgame join syntax behaved. From a quick glance, it looks like this is the current behavior, which sounds like the right call.

Glad to see this happen, it's exciting!

hey @rw-access. Nice to see you in the PR :-) and thank you for the interest.
The events mirror the structure of the query. The position of one event corresponds to the same position of the filter it is matching.

excellent! that's what it looked like. nice work with the feature!

bpintea

Impressive work, LGTM!

bpintea · 2022-05-25T18:22:04Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/parser/LogicalPlanBuilder.java

+
+ queries.add(joinTerm);
+ int numberOfQueries = queries.size();
+ if (numberOfQueries > 5) {


Nit: might be nice to extract this value into a constant. Also, I guess a doc PR will follow.

Then, given the "narrowing" nature of the keys-searching algorithm and capped composite page size (vs. just permutations search), it might be too conservative. But I guess that's future work.

bpintea · 2022-05-25T18:42:05Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/session/Results.java

 @SuppressWarnings("unchecked")
 public List<Sequence> sequences() {
- return type == Type.SEQUENCE ? (List<Sequence>) results : null;
+ return (type == Type.SEQUENCE || type == Type.SAMPLE) ? (List<Sequence>) results : null;


Not strictly related to this line: I understand that the sample has practically the same format as a sequence and that a sample is a list of Sequences, but the result isn't a sequence at all - the order of events is influenced by the order of the rules, but that's an implementation detail - so I'd find it natural for the result to reflect that: i.e. the result's hits should contain samples instead of sequences, imo. Otoh, keeping it like this might be easier to implement by the clients.

bpintea · 2022-05-25T18:58:18Z

x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/plan/logical/Sample.java

+
+public class Sample extends AbstractJoin {
+
+ public Sample(Source source, List<KeyedFilter> queries, KeyedFilter... query) {


The varadic argument is provided nowhere.

Nice catch! I've removed it.

bpintea · 2022-05-25T19:22:55Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/BaseEqlSpecTestCase.java

 public void setup() throws Exception {
 RestClient provisioningClient = provisioningClient();
- if (provisioningClient.performRequest(new Request("HEAD", "/" + unqualifiedIndexName())).getStatusLine().getStatusCode() == 404) {
+ boolean shouldLoadData = false == Arrays.stream(index.split(","))


Optional alternative: do away with false == ... by renaming var to dataLoaded.

bpintea · 2022-05-25T19:37:33Z

x-pack/plugin/eql/qa/common/src/main/java/org/elasticsearch/test/eql/EqlRestTestCase.java

- assertThat(response.getHeader("Content-Type"), containsString(contentType));
- assertThat(EntityUtils.toString(response.getEntity()), containsString(test[1]));
- assertThat(response.getStatusLine().getStatusCode(), is(400));
+ assert400BadRequest(test[0], test[1]);


nit: method's only called here, could be replaced by assertBadRequest(..., 400)

bpintea · 2022-05-25T20:17:58Z

...ck/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/execution/sample/SampleIterator.java

+ }
+
+ /*
+ * Starting point of the iterator, which also goes through all the criterions and gathers initial results pages.


luigidellaquila

LGTM, great work!

…o sampling_in_eql

astefan · 2022-05-30T13:24:11Z

@elasticmachine update branch

Eql Sampling feature

a8fee5e

elasticsearchmachine added the v8.2.0 label Mar 22, 2022

salvatore-campagna added v8.3.0 and removed v8.2.0 labels Mar 30, 2022

costin reviewed Apr 4, 2022

View reviewed changes

astefan added 3 commits April 15, 2022 19:13

Move the aggregation results handling to KeyExtractor infra.

18b0492

Add EqlSampleDataLoader Introduce AbstractJoin having Join and Sample as subclasses Rename Sampling to Sample Rename some methods Parametrize copySource

Minor fix in test infra

f27a383

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

1f02fcd

…o sampling_in_eql

costin reviewed Apr 19, 2022

View reviewed changes

astefan added 4 commits April 28, 2022 17:24

Address reviews

b74f698

Added support for optional fields

88c910a

Added tests

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

6d25a15

…o sampling_in_eql

Fix after upstream merge

8075b2b

astefan force-pushed the sampling_in_eql branch from 4b8cdf9 to 8075b2b Compare April 29, 2022 18:47

astefan requested a review from costin April 29, 2022 18:49

astefan marked this pull request as ready for review May 4, 2022 15:57

astefan requested review from Luegg, bpintea and luigidellaquila May 4, 2022 15:57

astefan added >feature :Analytics/EQL EQL querying labels May 4, 2022

elasticmachine added the Team:QL (Deprecated) Meta label for query languages team label May 4, 2022

Update docs/changelog/85206.yaml

26deaf3

elasticmachine and others added 3 commits May 5, 2022 01:54

Merge branch 'master' into sampling_in_eql

014927e

Remove workaround for elastic#85928

1371bac

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

ef33659

…o sampling_in_eql

luigidellaquila reviewed May 10, 2022

View reviewed changes

Luegg reviewed May 11, 2022

View reviewed changes

Add support for fetch_size.

cc41c99

Merge branch 'master' into sampling_in_eql

757acde

Cleaning up in line with elastic#86626

79ca0a8

luigidellaquila reviewed May 12, 2022

View reviewed changes

luigidellaquila reviewed May 16, 2022

View reviewed changes

astefan added 2 commits May 16, 2022 16:59

Address reviews

281e3f8

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

dc883d5

…o sampling_in_eql

astefan requested a review from Luegg May 17, 2022 11:12

costin approved these changes May 20, 2022

View reviewed changes

rw-access reviewed May 25, 2022

View reviewed changes

bpintea approved these changes May 25, 2022

View reviewed changes

craigtaverner added v8.4.0 and removed v8.3.0 labels May 25, 2022

luigidellaquila approved these changes May 26, 2022

View reviewed changes

astefan added 2 commits May 26, 2022 19:22

Address reviews

b717563

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

9cfa68e

…o sampling_in_eql

Merge branch 'master' into sampling_in_eql

56467b1

astefan changed the base branch from master to feature/eql_samples May 30, 2022 13:35

astefan merged commit 83422a0 into elastic:feature/eql_samples May 30, 2022

luigidellaquila mentioned this pull request Nov 10, 2022

EQL samples #91312

Merged


		import java.util.List;

		public class SamplingCriterion<Q extends QueryRequest> {


		import java.util.List;

		public class Criterion<Q extends QueryRequest> {

	hasJoin.set(hasJoin.get() \|\| plan.anyMatch(Sample.class::isInstance));
	boolean hasJoin = plan.anyMatch(AbstractJoin.class::isInstance)

	if (nextPage != null && nextPage.size() > 0) {
	if (nextPage.size() > 0) {


		class SamplePayload extends AbstractPayload {

		private final List<org.elasticsearch.xpack.eql.action.EqlSearchResponse.Sequence> values;


		public class Sample extends AbstractJoin {

		public Sample(Source source, List<KeyedFilter> queries, KeyedFilter... query) {

Eql Sampling #85206

Eql Sampling #85206

Uh oh!

Conversation

astefan commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 4, 2022

elasticsearchmachine commented May 4, 2022

astefan commented May 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Luegg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astefan commented May 12, 2022

astefan commented May 12, 2022

astefan commented May 12, 2022

luigidellaquila left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

astefan commented Mar 22, 2022 •

edited

Loading