enhancement - modify incident handling to match defender for endpoint

[agmic]

Enhancement

Proposed commit message

Modify incident handling to match Defender for Endpoint.

• Change fingerprint to use creation_time and modification_time to generate a unique id
• Add event.created set to original incident creation time
• Set timestamp to modification_time
• Change search cursor to use modification_time instead of creation_time in order to only fetch newest incidents.
• Add severity:critical=5 to allow for events marked as 5 in Cortex XDR

This change will align the integration with the defender for endpoint integration, enable correct fetching of the latest incidents and incident changes, and allow for ingestion of modified incidents.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[jkomara]

buildkite test this

[agmic]

I see the build is failing, but don't have access to buildkite to see why. Is there anything I can help with to facilitate this?

[kcreddy]

@agmic CI is failing on this line inside changelog: link: https://github.com/elastic/integrations/pull/ with following error:
Error: checking package failed: linting package failed: found 1 validation error:   | 1. https://github.com/elastic/integrations/pull/: issue number in changelog link should be a positive number

Can you update the line to link: https://github.com/elastic/integrations/pull/9246?

[agmic]

thx @kcreddy . I've made the update - any last things needed to trigger the buildcheck?

[kcreddy]

/test

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[kcreddy]

Hey @agmic,
IIUC you are trying add support for ingesting modified incidents.
But the problem I see is that it gets difficult to analyse documents with dashboards/aggregations. For example, when an incident has changed its severity from critical to high, you will have 2 documents now inside the datastream with 2 states.
When you try to find how many incidents have severity: critical -> this leads to 1.
Also, how many incidents have severity: high -> this also leads to 1.

The better way to handle this is using Elasticsearch Latest Transform where you can store latest copy in separate index. But, transforms are usually computationally expensive as every time it has to query all the data in the source datastream to find and store latest copies.

@jamiehynds Do you think adding this support (modified incidents) using transforms is worth it? Currently based on the fingerprint processor, modifications to incidents are not ingested.

[agmic]

We find value in the suggested behavior as it allows the modifications to come into elastic, enabling us to follow the changes. The behavior also follows how incidents from M365 Defender are being handled by that integration.
I checked the M365 Incident dashboard, and it looks like it is just uses a simple filter on severity, so there is the potential for counting the same incident twice here as well. But I believe this could be handled by taking the last value of a given incident ID.

integrations/packages/m365_defender/kibana/dashboard/m365_defender-ac54d310-44ab-11ed-8375-0168a9970c06.json

Line 318 in 2e66957

"query": "m365_defender.incident.alert.severity : \"high\""

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[agmic]

👍

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[andrewkroh]

The update to include mod time in the fingerprint seem good, because otherwise the integration mostly won't index any updates. I suggest adding a small blurb in the Alerts section of the package's readme to state that a new document is indexed when alerts are modified.

[kcreddy]

Hey @agmic , if you are okay with the suggestion here, could you please add to this PR?

As for the other approach, if you would like to add latest transform, you can take an example from here.

[agmic]

@kcreddy I'll update the readme and add it to the PR. Just to be clear, I'll add it to the Incidents section, as this is what is changed.
Since there have been changes to the cortex integration since my initial PR (we are now on version 1.27.0) Do I need to resolve the conflicts and resubmit all changes - or can I just change the readme file?

[kcreddy]

Hey @agmic, thanks for taking care of this.

Just to be clear, I'll add it to the Incidents section, as this is what is changed.

Yes, that sounds good.

Do I need to resolve the conflicts and resubmit all changes - or can I just change the readme file?

Yes, the merge conflicts need to be resolved as well.

[kcreddy]

/test

[kcreddy]

@agmic The CI is failing on outdated README.
Can you regenerate the README file and commit it.

elastic-package build && elastic-package format && elastic-package lint && elastic-package check && elastic-package build

[agmic]

@kcreddy - I'd forgotten that the README.md was generated from a template. I've modified the template in _dev/build/docs and run the elastic-package commands above. The regenerated README.md appears to be unchanged.

[kcreddy]

/test

[agmic]

@kcreddy Failing on expected outcome of incident parsing due to change of timestamp handling - I'll get that remedied.

[kcreddy]

/test

[kcreddy]

@agmic You may have to re-run the pipeline tests:
eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v

There is a missing comma , in one of the lines, causing an invalid JSON.

[agmic]

/test

[kcreddy]

/test

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
47.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1c5a37a

History

• 💔 Build #13592 failed 4398272
• 💔 Build #13587 failed fadac77
• 💔 Build #13525 failed 126ec97
• 💔 Build #9879 failed d790b79
• 💔 Build #9347 failed fbe76b2

[kcreddy]

LGTM

[elasticmachine]

Package panw_cortex_xdr - 1.28.0 containing this change is available at https://epr.elastic.co/search?package=panw_cortex_xdr