Fix pipeline and tests for 8.7 #5910

bhapas · 2023-04-18T13:27:10Z

What does this PR do?

This PR fixes strict json input for different integrations that have faulty json inputs.
This also fixes the pipeline for ti_cybersixgill integration where the grok processor is appending the values that it has processed.
This PR also includes change from type:float to type:double for citrix_adc integration

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Testing Checklist

Manually trigger CI testing of this PR and set the stack parameter to 8.7.1-SNAPSHOT (https://fleet-ci.elastic.co/job/Ingest-manager/job/integrations/job/PR-5910/build?delay=0sec).

Related Issues

Making JsonProcessor stricter so that it does not silently drop data elasticsearch#93179

elasticmachine · 2023-04-18T14:19:30Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-18T21:44:03.585+0000
Duration: 52 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	328
Skipped	4
Total	332

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-04-18T14:19:41Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`37/37`)	💚
Files	97.368% (`37/38`)	👎 -2.632
Classes	97.368% (`37/38`)	👎 -2.632
Methods	91.589% (`490/535`)	👎 -8.411
Lines	87.754% (`10985/12518`)	👎 -7.431
Conditionals	100.0% (`0/0`)	💚

bhapas · 2023-04-18T20:11:49Z

/test

elasticmachine · 2023-04-18T20:56:19Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

bhapas · 2023-04-18T20:57:48Z

Relates #5641

efd6

Queries:

packages/mimecast/data_stream/siem_logs/_dev/test/pipeline/test-siem-logs.log

.../ceph/data_stream/cluster_status/_dev/test/pipeline/test-cluster-status-metrics-nautilus.log

efd6 · 2023-04-18T22:12:16Z

packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.json

@@ -1,31 +0,0 @@
-{


What was the issue here?

Changed the filename from .json to .log as the input file contained a bunch of json inputs to the data_stream

So essentially clarity?

efd6

LGTM for the SEI packages. Please wait for other teams.

efd6 · 2023-04-18T22:22:26Z

packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.json

@@ -1,31 +0,0 @@
-{


So essentially clarity?

bhapas · 2023-04-18T22:24:33Z

So essestially clarity?
This also goes in hand with the strict json processor thing. So fixed the inputs to be strict jsons

bhapas · 2023-04-19T04:47:30Z

TEST Result here

citrix_adc

The Citrix_adcintegration has a decimal truncation which is different in 8.4 and 8.7 and hence this integration failed in 8.7 CI build. If min Kibana version is moved to 8.7.1 , it works

Solution

Converting the type:float to type:double

endorama

👍 from Obs Cloud Monitoring (aws.route53_resolver_logs test JSON comes directly from the initial PR and looks like a copy paste error)

lalit-satapathy

Approving for ciricx_adc and ceph changes.

elasticmachine · 2023-04-19T11:27:56Z

Package citrix_adc - 0.5.2 containing this change is available at https://epr.elastic.co/search?package=citrix_adc

elasticmachine · 2023-04-19T11:32:44Z

Package ti_cybersixgill - 1.10.1 containing this change is available at https://epr.elastic.co/search?package=ti_cybersixgill

ebeahan · 2023-04-21T14:04:16Z

The Citrix_adcintegration has a decimal truncation which is different in 8.4 and 8.7 and hence this integration failed in 8.7 CI build

@bhapas is this also related to a change in Elasticsearch 8.7, or something else?

bhapas · 2023-04-21T14:41:57Z

The Citrix_adcintegration has a decimal truncation which is different in 8.4 and 8.7 and hence this integration failed in 8.7 CI build

@ebeahan Yes , a change in ES 8.7 , but not related to the Grok Processor change

* [aws]Fix json input for pipeline tests * [ceph]fix input for pipeline tests * [citrix_adc]Fix ijson input for pipeline tests * [okta]Fix json input for pipeline tests * [citrix_adc]fix pipeline to have double type instead of float

bhapas added 6 commits April 18, 2023 14:58

Fix Pipeline issues

5e37fa8

Fix json input for pipeline tests

fa2a7b0

[aws]Fix json input for pipeline tests

e59ef51

[ceph]fix input for pipeline tests

392a27c

[citrix_adc]Fix ijson input for pipeline tests

65a1502

[okta]Fix json input for pipeline tests

e1cc983

bhapas added the bug Something isn't working, use only for issues label Apr 18, 2023

bhapas self-assigned this Apr 18, 2023

Fix changelog

a21a5b4

bhapas added the Team:Security-External Integrations label Apr 18, 2023

Fix pipeline test values for 8.4

323f454

bhapas marked this pull request as ready for review April 18, 2023 20:56

bhapas requested review from a team as code owners April 18, 2023 20:56

[citrix_adc]fix pipeline to have double type instead of float

cd73b53

efd6 reviewed Apr 18, 2023

View reviewed changes

efd6 approved these changes Apr 18, 2023

View reviewed changes

packages/okta/data_stream/system/_dev/test/pipeline/test-okta-system-events.json

@@ -1,31 +0,0 @@

{

Copy link

Contributor

efd6 Apr 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So essentially clarity?

endorama approved these changes Apr 19, 2023

View reviewed changes

andrewkroh approved these changes Apr 19, 2023

View reviewed changes

lalit-satapathy approved these changes Apr 19, 2023

View reviewed changes

bhapas merged commit fd27f43 into elastic:main Apr 19, 2023

bhapas deleted the 8.7-issues-fixes branch April 19, 2023 11:06

andrewkroh added Integration:aws AWS Integration:ceph Ceph Integration:citrix_adc Citrix ADC Integration:mimecast Mimecast (Partner supported) Integration:okta Okta Integration:ti_cybersixgill Cybersixgill labels Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix pipeline and tests for 8.7 #5910

Fix pipeline and tests for 8.7 #5910

Uh oh!

bhapas commented Apr 18, 2023 •

edited

Loading

elasticmachine commented Apr 18, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Apr 18, 2023 •

edited

Loading

bhapas commented Apr 18, 2023

elasticmachine commented Apr 18, 2023

bhapas commented Apr 18, 2023

efd6 left a comment

Uh oh!

Uh oh!

efd6 Apr 18, 2023

bhapas Apr 18, 2023

efd6 Apr 18, 2023

efd6 left a comment

efd6 Apr 18, 2023

bhapas commented Apr 18, 2023

bhapas commented Apr 19, 2023

endorama left a comment

lalit-satapathy left a comment

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

ebeahan commented Apr 21, 2023

bhapas commented Apr 21, 2023

Labels

7 participants

Fix pipeline and tests for 8.7 #5910

Fix pipeline and tests for 8.7 #5910

Uh oh!

Conversation

bhapas commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Testing Checklist

Related Issues

elasticmachine commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

bhapas commented Apr 18, 2023

elasticmachine commented Apr 18, 2023

bhapas commented Apr 18, 2023

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 Apr 18, 2023

Choose a reason for hiding this comment

bhapas Apr 18, 2023

Choose a reason for hiding this comment

efd6 Apr 18, 2023

Choose a reason for hiding this comment

efd6 left a comment

Choose a reason for hiding this comment

efd6 Apr 18, 2023

Choose a reason for hiding this comment

bhapas commented Apr 18, 2023

bhapas commented Apr 19, 2023

citrix_adc

Solution

endorama left a comment

Choose a reason for hiding this comment

lalit-satapathy left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 19, 2023

elasticmachine commented Apr 19, 2023

ebeahan commented Apr 21, 2023

bhapas commented Apr 21, 2023

Labels

7 participants

bhapas commented Apr 18, 2023 •

edited

Loading

elasticmachine commented Apr 18, 2023 •

edited

Loading

elasticmachine commented Apr 18, 2023 •

edited

Loading