darktrace: Handle nested defeat conditions in Darktrace models #15552

arvchristos · 2025-10-06T09:43:22Z

Proposed commit message

Darktrace model defeats are not 1 level only (List) as currently implemented. Latest API provides functionality for nested conditions. If not nested, the defeatID is an integer and the defeats list is a list of conditions. However, if nested, defeats list is a list of lists of conditions and defeatIDs can either be integers (e.g. 1) or strings (e.g. 1-1, 1-2 etc.).

Is there darktrace documentation to back this up? It would be helpful to include that in the commit message for future maintenance.

Definition of the field from Darktrace documentation:

defeats.defeatID	numeric/string	1	An id for the defeat within the context of the model. If defeats are conditional, their id will indicate the relationship - e.g., a condition for defeat with id 1 will have ID 1-1. In this case, the id will be a string.

An example result for the model schema:

{ "name": "Anomalous File::Anomalous Octet Stream", "pid": 12, "phid": 2842, "uuid": "80010119-6d7f-0000-0305-5e0000000420", "logic": { "data": [ 3621 ], "type": "componentList", "version": 1 }, "throttle": 3600, "sharedEndpoints": false, "actions": { "alert": true, "antigena": {}, "breach": true, "model": true, "setPriority": false, "setTag": false, "setType": false }, "tags": [ "AP: Tooling", "DNS Server" ], "interval": 0, "delay": 0, "sequenced": false, "active": true, "modified": "2023-08-15 12:27:32", "activeTimes": { "devices": {}, "tags": {}, "type": "exclusions", "version": 2 }, "priority": 2, "autoUpdatable": false, "autoUpdate": true, "autoSuppress": true, "description": "A device has downloaded a rare data stream which is not specifying a specific data type...", "behaviour": "decreasing", "defeats": [ [ { "arguments": { "value": "6" }, "comparator": "is", "defeatID": 1, "filtertype": "Internal source device type" }, { "defeatID": "1-1", "filtertype": "Destination port", "comparator": "!=", "arguments": { "value": 80 } } ], { "arguments": { "value": "7" }, "comparator": "is", "defeatID": 2, "filtertype": "Internal source device type" }, { "arguments": { "value": "7" }, "comparator": "is", "defeatID": 3, "filtertype": "Internal source device type" }, { "arguments": { "value": "1" }, "comparator": "is", "defeatID": 4, "filtertype": "Internal source device type" } ], "created": { "by": "System" }, "edited": { "by": "Sarah", "userID": 24 }, "history": [ { "modified": "2023-08-15 12:27:32", "active": true, "message": "Improved rarity filters, and merged exclusion filters", "by": "Sarah", "phid": 2842 }, { "modified": "2023-05-31 13:47:22", "active": false, "message": "Updating regex filters", "accepted": 1686578003000, "acceptedBy": "Sarah", "by": "System", "phid": 2675 }, ... ], "message": "Altered priority value.", "mitre": { "tactics": [ "command-and-control", "execution", "lateral-movement" ], "techniques": [ "T1203", "T1210", "T1219" ] }, "version": 53, "priority": 2, "category": "Critical", "compliance": false }

Title of PR is suggested as commit message.

This is labeled as bugfix as the incorrect parsing of defeatID leads to drop of any event that has a nested model defeat, leading to logs being lost.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

cla-checker-service · 2025-10-06T09:43:27Z

💚 CLA has been signed

arvchristos · 2025-10-06T10:30:41Z

❌ Author of the following commits did not sign a Contributor Agreement: 16b44dd, 6d2f6de

Please, read and sign the above mentioned agreement if you want to contribute to this project

Done

elasticmachine · 2025-10-06T12:01:07Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

Is there darktrace documentation to back this up? It would be helpful to include that in the commit message for future maintenance.

...ages/darktrace/data_stream/model_breach_alert/_dev/test/pipeline/test-model-breach-alert.log

efd6 · 2025-10-08T23:09:57Z

packages/darktrace/data_stream/model_breach_alert/fields/fields.yml

 description: The filter the defeat is made from.
 - name: id
- type: long
+ type: keyword


This would be a breaking change. I agree that it should have been a keyword when written, but do we need to do this?

I agree with you on this point. However, at the moment, if the id is not numeric but a string, the message is incorrectly parsed and dropped with the following error:

Cannot index event * (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [darktrace.model_breach_alert.model.defeats.defeatID] cannot be changed from type [long] to [keyword]\"}

As the initial value can be a string, how would you handle it without stepping into this breaking change?

This comment was in conjunction with the comment here. If one changes they both change. Do we need to make this a keyword (and change the type of the convert processor? If it's the case that we will see non-numeric IDs, then the answer is yes. If that is the case this becomes a breaking change and the changelog would need to be updated to include that.

As the defeatID (and by extension the id field here) can take either numeric or string, I do not see another way that could accommodate bot 1 and "1-1" values in a single definition so from my point of view yes.

Replied before your last edit in the thread. Yes, this is needed as we ingest non numeric IDs.

I updated the changelog type to breaking-change

efd6 · 2025-10-08T23:10:47Z

packages/darktrace/data_stream/model_breach_alert/elasticsearch/ingest_pipeline/default.yml

 on_failure:
 - remove:


String conversion cannot fail, so the on_failure is no longer needed, though see comment below about this being a breaking change.

Removed this processor. Thank you for the feedback

arvchristos · 2025-10-09T07:01:52Z

Is there darktrace documentation to back this up? It would be helpful to include that in the commit message for future maintenance.

As you may know, Darktrace is a proprietary software, hence I cannot share a direct link to the documentation. However, I can share an excerpt of it for the specific field.

I am interested in how you usually tackle cases like this one when the documentation is fenced but the implementation open source.

For now, I will include the definition of the field from Darktrace documentation (also in the Commit message) :

defeats.defeatID	numeric/string	1	An id for the defeat within the context of the model. If defeats are conditional, their id will indicate the relationship - e.g., a condition for defeat with id 1 will have ID 1-1. In this case, the id will be a string.

An example result for the model schema:

{ "name": "Anomalous File::Anomalous Octet Stream", "pid": 12, "phid": 2842, "uuid": "80010119-6d7f-0000-0305-5e0000000420", "logic": { "data": [ 3621 ], "type": "componentList", "version": 1 }, "throttle": 3600, "sharedEndpoints": false, "actions": { "alert": true, "antigena": {}, "breach": true, "model": true, "setPriority": false, "setTag": false, "setType": false }, "tags": [ "AP: Tooling", "DNS Server" ], "interval": 0, "delay": 0, "sequenced": false, "active": true, "modified": "2023-08-15 12:27:32", "activeTimes": { "devices": {}, "tags": {}, "type": "exclusions", "version": 2 }, "priority": 2, "autoUpdatable": false, "autoUpdate": true, "autoSuppress": true, "description": "A device has downloaded a rare data stream which is not specifying a specific data type...", "behaviour": "decreasing", "defeats": [ [ { "arguments": { "value": "6" }, "comparator": "is", "defeatID": 1, "filtertype": "Internal source device type" }, { "defeatID": "1-1", "filtertype": "Destination port", "comparator": "!=", "arguments": { "value": 80 } } ], { "arguments": { "value": "7" }, "comparator": "is", "defeatID": 2, "filtertype": "Internal source device type" }, { "arguments": { "value": "7" }, "comparator": "is", "defeatID": 3, "filtertype": "Internal source device type" }, { "arguments": { "value": "1" }, "comparator": "is", "defeatID": 4, "filtertype": "Internal source device type" } ], "created": { "by": "System" }, "edited": { "by": "Sarah", "userID": 24 }, "history": [ { "modified": "2023-08-15 12:27:32", "active": true, "message": "Improved rarity filters, and merged exclusion filters", "by": "Sarah", "phid": 2842 }, { "modified": "2023-05-31 13:47:22", "active": false, "message": "Updating regex filters", "accepted": 1686578003000, "acceptedBy": "Sarah", "by": "System", "phid": 2675 }, ... ], "message": "Altered priority value.", "mitre": { "tactics": [ "command-and-control", "execution", "lateral-movement" ], "techniques": [ "T1203", "T1210", "T1219" ] }, "version": 53, "priority": 2, "category": "Critical", "compliance": false }

efd6 · 2025-10-09T07:35:21Z

/test

…model.defeats.id type

packages/darktrace/changelog.yml

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

efd6 · 2025-10-09T19:57:28Z

/test

efd6 · 2025-10-09T20:21:04Z

Tests should be fixed by

diff --git a/packages/darktrace/changelog.yml b/packages/darktrace/changelog.yml index 352ff55219..af681bc3ad 100644 --- a/packages/darktrace/changelog.yml +++ b/packages/darktrace/changelog.yml @@ -4,7 +4,7 @@ - description: Fix model defeat handling when nested conditions exist and so defeat ID can be non-numeric. type: bugfix link: https://github.com/elastic/integrations/pull/15552 - - description: `darktrace.model_breach_alert.model.defeats.id` field type is changed from `long` to `keyword`.  + - description: '`darktrace.model_breach_alert.model.defeats.id` field type is changed from `long` to `keyword`.' type: breaking-change link: https://github.com/elastic/integrations/pull/15552 - version: "1.23.0" diff --git a/packages/darktrace/manifest.yml b/packages/darktrace/manifest.yml index 290de70035..ab7dc8b2ff 100644 --- a/packages/darktrace/manifest.yml +++ b/packages/darktrace/manifest.yml @@ -1,7 +1,7 @@ format_version: "3.0.3" name: darktrace title: Darktrace -version: "1.23.1" +version: "2.0.0" description: Collect logs from Darktrace with Elastic Agent. type: integration categories:

arvchristos · 2025-10-10T06:38:20Z

- description: '`darktrace.model_breach_alert.model.defeats.id` field type is changed from `long` to `keyword`.'

Changes pushed, thank you!

efd6 · 2025-10-12T21:44:28Z

/test

elastic-vault-github-plugin-prod · 2025-10-12T22:14:12Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-10-12T22:14:15Z

💚 Build Succeeded

Buildkite Build
Commit: f109596

History

💔 Build #32565 failed 7640a1b
💔 Build #32545 failed 2c909b5

efd6

Thanks

elastic-vault-github-plugin-prod · 2025-10-13T01:54:10Z

Package darktrace - 2.0.0 containing this change is available at https://epr.elastic.co/package/darktrace/2.0.0/

Handle nested defeat conditions in Darktrace models

16b44dd

arvchristos requested a review from a team as a code owner October 6, 2025 09:43

Update changelog with new PR id

6d2f6de

Merge branch 'main' into dt_model_breach_defeat_id

1ab1214

andrewkroh added Integration:darktrace Darktrace Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Oct 6, 2025

efd6 added the bugfix Pull request that fixes a bug issue label Oct 8, 2025

efd6 reviewed Oct 8, 2025

View reviewed changes

Fix review feedback points

2c909b5

arvchristos added 2 commits October 9, 2025 09:44

Change changelog type to breaking-change

701ba3c

Update Darktrace documentation with new darktrace.model_breach_alert.…

2a3c746

…model.defeats.id type

arvchristos requested a review from efd6 October 9, 2025 08:41

efd6 reviewed Oct 9, 2025

View reviewed changes

packages/darktrace/changelog.yml Outdated Show resolved Hide resolved

Issue new major version for breaking change

7640a1b

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

arvchristos requested a review from efd6 October 9, 2025 11:27

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 9, 2025

Fix tests

f109596

efd6 approved these changes Oct 13, 2025

View reviewed changes

efd6 merged commit 7051042 into elastic:main Oct 13, 2025
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

darktrace: Handle nested defeat conditions in Darktrace models #15552

darktrace: Handle nested defeat conditions in Darktrace models #15552

Uh oh!

arvchristos commented Oct 6, 2025 •

edited

Loading

cla-checker-service bot commented Oct 6, 2025 •

edited

Loading

arvchristos commented Oct 6, 2025

elasticmachine commented Oct 6, 2025

efd6 left a comment

Uh oh!

efd6 Oct 8, 2025

arvchristos Oct 9, 2025

efd6 Oct 9, 2025 •

edited

Loading

arvchristos Oct 9, 2025

arvchristos Oct 9, 2025

efd6 Oct 8, 2025

arvchristos Oct 9, 2025

arvchristos commented Oct 9, 2025

efd6 commented Oct 9, 2025

Uh oh!

efd6 commented Oct 9, 2025

efd6 commented Oct 9, 2025

arvchristos commented Oct 10, 2025

efd6 commented Oct 12, 2025

elastic-vault-github-plugin-prod bot commented Oct 12, 2025

elasticmachine commented Oct 12, 2025

efd6 left a comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 13, 2025

Labels

4 participants

darktrace: Handle nested defeat conditions in Darktrace models #15552

darktrace: Handle nested defeat conditions in Darktrace models #15552

Uh oh!

Conversation

arvchristos commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

cla-checker-service bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

arvchristos commented Oct 6, 2025

elasticmachine commented Oct 6, 2025

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Oct 8, 2025

Choose a reason for hiding this comment

arvchristos Oct 9, 2025

Choose a reason for hiding this comment

efd6 Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

arvchristos Oct 9, 2025

Choose a reason for hiding this comment

arvchristos Oct 9, 2025

Choose a reason for hiding this comment

efd6 Oct 8, 2025

Choose a reason for hiding this comment

arvchristos Oct 9, 2025

Choose a reason for hiding this comment

arvchristos commented Oct 9, 2025

efd6 commented Oct 9, 2025

Uh oh!

efd6 commented Oct 9, 2025

efd6 commented Oct 9, 2025

arvchristos commented Oct 10, 2025

efd6 commented Oct 12, 2025

elastic-vault-github-plugin-prod bot commented Oct 12, 2025

🚀 Benchmarks report

elasticmachine commented Oct 12, 2025

💚 Build Succeeded

History

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 13, 2025

Labels

4 participants

arvchristos commented Oct 6, 2025 •

edited

Loading

cla-checker-service bot commented Oct 6, 2025 •

edited

Loading

efd6 Oct 9, 2025 •

edited

Loading