Check failure store when available during system tests

[jsoriano]

Enable failure store and check if it has any documents after executing system tests.

Failure store is used if the version of the stack is 8.15.0 or newer. In lower versions there are unexpected failures on tests.

Fixes #1832.

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jsoriano]

Failures in CI seem legit.

[jsoriano]

Ok, it looks like even when the failure store is available in older versions, it has some issues with time series before 8.15. Reopening for review after enabling failure store starting on 8.15, and pinning the stack version for the related test package.

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[jlind23]

LGTM but waiting for @flash1293 review and test anyway

[flash1293]

The code looks good to me, however I'm not sure how to test locally anymore - I used to

elastic-package test system -v --setup --config-file ./my/config.yml, then elastic-package test system -v --no-provision.

The first command still works and finds the docs in the failure store, however I don't know how to see the test failing locally (as the setup routine is not running the validation bits). Seems like this code was recently restructured.

Not a blocker to merge though, I read through the code and it seems to work exactly like the ignored docs stuff.

[jsoriano]

@flash1293 do you mean that it doesn't fail when running with --no-provision? It should fail the same as when running directly with elastic-package test system -v.

[flash1293]

It never finds any hits when running with --no-provision and doesn't terminate

[jsoriano]

It never finds any hits when running with --no-provision and doesn't terminate

Have you tried with elastic-package test system -v? Could you share the package you are testing with?

[flash1293]

@jsoriano When I'm running just with test system it doesn't get to that point because it fails during teardown:

Error: failed to tear down system runner: there was an apply error: can't remove the package: could not remove package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Unable to remove package with existing package policy(s) in use by agent(s)"}

I'm testing with the test/packages/false_positives/failure_store package from this PR.

I don't think it's something wrong with this PR, seems like a general thing.

[flash1293]

nvm, I think it was just some stale state in my local stack. Tested again with a fresh stack and it works as expected 👍

One nit: Maybe we should show more details of the failure than just the error message (e.g. which processor was involved and so on) to make it easier to debug what's happening.

[jsoriano]

Maybe we should show more details of the failure than just the error message (e.g. which processor was involved and so on) to make it easier to debug what's happening.

Thanks @flash1293 for the feedback. I have pushed another commit including more information in the error messages, they include the error type, the processor and the pipelines that caused the issue:

--- Test results for package: failure_store - START --- FAILURE DETAILS: failure_store/test fail: [0] illegal_argument_exception: unable to convert [eight] to integer (processor: convert, pipelines: logs-failure_store.test-0.0.1) [1] illegal_argument_exception: unable to convert [five] to integer (processor: convert, pipelines: logs-failure_store.test-0.0.1) [2] illegal_argument_exception: unable to convert [seven] to integer (processor: convert, pipelines: logs-failure_store.test-0.0.1) [3] illegal_argument_exception: unable to convert [six] to integer (processor: convert, pipelines: logs-failure_store.test-0.0.1) 

Additionally, the stack trace is logged at the debug level when collecting results.

Wdyt? Is it too much?

[flash1293]

Love it! Thanks for adding

[jsoriano]

test integrations

[elasticmachine]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#10492

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b911556

History

• 💔 Build #3628 failed 69ffa36
• 💔 Build #3627 failed a8c71d5
• 💔 Build #3626 failed 0f63308
• 💔 Build #3625 failed f9bcf0e
• 💔 Build #3624 failed 9fe6090

cc @jsoriano