Add ransomware alert schema #96

marshallmain · 2020-10-16T15:02:14Z

No description provided.

magermark

Looks pretty good, just need to update some descriptions and change metrics to an array of keywords

magermark · 2020-10-16T15:56:26Z

custom_schemas/custom_ransomware.yml

+ - name: version
+ level: custom
+ type: keyword
+ description: Ransomware detection version.


description: Ransomware artifact version

magermark · 2020-10-16T15:57:18Z

custom_schemas/custom_ransomware.yml

+ - name: files
+ level: custom
+ type: object
+ description: Information about each file targeted by the ransomware. Expected to be an array.


description: Information about each file event attributed to the ransomware. Expected to be an array.

magermark · 2020-10-16T16:00:03Z

custom_schemas/custom_ransomware.yml

+ - name: files.entropy
+ level: custom
+ type: double
+ description: TODO write description and check that type is indeed double


description: "Entropy of file contents"

magermark · 2020-10-16T16:02:48Z

custom_schemas/custom_ransomware.yml

+ - name: files.data
+ level: custom
+ type: binary
+ description: Combined header_string + data buffer field


description: File header or MBR bytes

magermark · 2020-10-16T16:06:29Z

custom_schemas/custom_ransomware.yml

+ - name: files.operation
+ level: custom
+ type: keyword
+ description: Operation applied to file.


@wburgess1 Operation is string-based here. So I take it we'll need to map the file operation dword values to their string equivalents before forming the alert?

@magermark operation is only ever one int value right? So this is fine just being a keyword and not say an array?

Yes, that's correct

magermark · 2020-10-16T16:10:14Z

custom_schemas/custom_ransomware.yml

+ - name: files.metrics
+ level: custom
+ type: double
+ description: TODO idk what this is


metrics should be an array of strings (keywords?), so something like this:

name: files.metrics level: custom type: keyword description: Characteristics that describe uniqueness of file event normalize: - array

I would be tempted to go with a description like:

description: Suspicious ransomware behaviours associated with the file event.

magermark · 2020-10-16T16:13:41Z

custom_schemas/custom_ransomware.yml

+ - name: files.score
+ level: custom
+ type: double
+ description: Ransomware score for this particular file.


description: Ransomware score for this particular file event

magermark · 2020-10-16T16:21:10Z

custom_subsets/elastic_endpoint/alerts/ransomware_event.yaml

+ fields:
+ id:
+ exceptionable: true
+ rule:


Would rule apply for a ransomware alert? This seems out of place here but I see it's being used for malware alerts so maybe it's safe just to leave it in just in case it's needed: https://github.com/elastic/endpoint-package/blob/master/custom_subsets/elastic_endpoint/alerts/malware_event.yaml

Initially when making the malware schema I thought we might use rule to capture high level information about the malware module and make it easily accessible - basically treating the malware detection routine as one large rule. That hasn't happened so the fields are unused afaik. It's a core ECS fieldset though so maintaining them is free for us.

wburgess1 · 2020-10-19T11:06:22Z

custom_schemas/custom_ransomware.yml

+ - name: files.path
+ level: custom
+ type: keyword
+ description: File path after being ransomed


I feel like this description (and ones above) might be slightly misleading. It's mainly the before/being ransomed that I'm uncomfortable with (and may be misleading in some circumstances).

I would be tempted to put these descriptions in more neutral language e.g. File path for suspicious file activity / File path before suspicious file activity etc.. or maybe even just File path?

Thoughts @magermark ?

I think File path is sufficient. There are circumstances (ransom notes, benign activity) where a file path isn't necessarily being encrypted, so the description should be more general.

wburgess1 · 2020-10-19T11:40:37Z

custom_schemas/custom_ransomware.yml

+ - name: feature
+ level: custom
+ type: keyword
+ description: Specific ransomware feature, e.g. MBR/Lua/Canaries.


description: Ransomware feature which triggered the alert.

wburgess1 · 2020-10-19T11:50:53Z

custom_schemas/custom_ransomware.yml

+ - name: score
+ level: custom
+ type: double
+ description: Total score of all files ransomed.


description: Total ransomware score for aggregated file events.

wburgess1 · 2020-10-19T12:15:19Z

custom_schemas/custom_ransomware.yml

+ - name: child_pids
+ level: custom
+ type: long
+ description: Array of PIDs for cases where ransomware spawns numerous child processes (e.g. lockergoga)


description: Array of child PIDs for ransomware which spawns numerous processes to handle encryption.

jonathan-buttner

Just did a quick look over, need to review more in depth.

jonathan-buttner · 2020-10-19T17:38:17Z

custom_schemas/custom_ransomware.yml

+ type: long
+ description: Array of PIDs for cases where ransomware spawns numerous child processes (e.g. lockergoga)
+ normalize:
+ - array


What does the normalize do here?

It doesn't do anything for elasticsearch right now, but it's used in some ECS core fields as well for documentation that a field is expected to be an array.

jonathan-buttner · 2020-10-19T17:40:53Z

custom_subsets/elastic_endpoint/alerts/ransomware_event.yaml

@@ -0,0 +1,383 @@
+---


Could you add these:

# these fields are needed in the mapping so the maps page of the security app does not throw a bunch of errors source: fields: geo: fields: "*" destination: fields: geo: fields: "*"

It's a bit of a hack to get around this issue: elastic/kibana#73304
#87

jonathan-buttner · 2020-10-20T17:01:45Z

custom_schemas/custom_ransomware.yml

+
+ - name: score
+ level: custom
+ type: double


does this need to be a double or could it be a smaller float? In your example @wburgess1 they only had one decimal places I think. Will that always be the case?

@jonathan-buttner We typically require up to three decimal places at the moment

jonathan-buttner · 2020-10-20T17:05:24Z

custom_schemas/custom_ransomware.yml

+ normalize:
+ - array
+
+ - name: files


Do we imagine the user searching across these files? Would we need to switch it to type: nested?

@jonathan-buttner Yes, users would likely be interested in searching / scanning through this list to determine the extent of the breach. type: nested seems like it would be appropriate in this use case

I feel like if we can make it searchable it sounds like a good idea but not sure if that comes at a performance cost. Note that max number of objects will be 200, on average though its probs around ~50. @jonathan-buttner how would these numbers affect nested limitations ?

Yeah so the advantage of nested is being able to query the objects in the array independently of each other, which is probably what we'd want to do. At endgame we ran into issues with nested objects because they have these limits by default:

https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html#:~:text=Default%20is%2050%20.&text=The%20maximum%20number%20of%20nested,Default%20is%2010000%20.

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html

If worst case is 200, I think we'll be fine.

@magermark Would users want to query for a specific alert based on files that are breached? Or would they only need be able to scan through the list of files after picking a specific alert based on other criteria? Even without nested they'll be able to see all the information in files once they are looking at a specific alert document.

The use case that nested enables is a query like "find ransomware alerts where a certain ransomed file had (property 1 AND property 2)".

@jared-day Do you have any opinion on the above? Is it desirable for users to be able to query for a specific file that may have been encrypted/affected?

@marshallmain that was the main use case @magermark and I were thinking about - a user may want to search for a specific file that may have been affected by this kind of incident (e.g. for incident response).

Is it easy to change this in the future? My view is still the same, if we can do it and it's fine performance wise then it makes sense to gives users as much flexibility as possible.

It's not easy to change the type of a field in the future so I went ahead and changed it to nested.

jonathan-buttner · 2020-10-20T17:54:36Z

custom_schemas/custom_ransomware.yml

+
+ - name: files.score
+ level: custom
+ type: double


Same question here does this need to be a full double?

Up to three decimal places should suffice

Gotcha, if we wanted to save some space we could probably switch it to like a float or maybe even half_float I think:

https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html#number

jonathan-buttner · 2020-10-20T18:09:00Z

custom_subsets/elastic_endpoint/alerts/ransomware_event.yaml

@@ -0,0 +1,392 @@
+---
+name: ransomware_event


Just double checking but we don't need any of the core file fields right?

Entries in the files array (Ransomware.files) here will greatly differ from typical core file entries, so we do not need to inherit any fields from file.

wburgess1 · 2020-10-20T20:26:53Z

custom_schemas/custom_ransomware.yml

+ - name: files.previous_extension
+ level: custom
+ type: keyword
+ description: Previous file extension.


Another nitpick and feel free to ignore but I noticed in the file fields schema (

endpoint-package/schemas/v1/file/file.yaml

Line 793 in 9ca1713

description: Original file path prior to a modification event

) it uses the following:

description: Original file path prior to a modification event

To remain consistent we could also use something like (basing the below on descriptions taken from that schema):

description: Previous file extension prior to a ransomware file event. description: Previous file path prior to a ransomware file event. description: Full path to the file, including the file name. It should include the drive letter, when appropriate

We might also want to use files.original.path to remain consistent with the custom file fieldset. Thoughts? Would that mean the same thing?

I think files.path is still fine for this. If anything previous_path / previous_extension could be renamed to original.path / original.extension to be more consistent with ECS 👍

…l.extension

wburgess1 · 2020-10-22T12:03:18Z

This LGTM now. The only possible change which I will defer to you @marshallmain is changing the description of Ransomware.files[].original.path/extension respectively to:

description: Original file path prior to the file event. description: Original file extension prior to the file event. Also could do `files.path` --> description: Full path to the file, including the file name.

As an FYI, I sanity checked custom_ransomware.yml to ensure everything we want is collected there (and ensured all superfluous fields were removed) and it looks good to me 👍. I also double checked all the field types (and cross referenced to the old endgame schema) and that looks good too. Lastly, I checked ransomware_event.ymland cross referenced it to an alert currently being generated. All the top level fields listed in ransomware_event.yml were present apart from Dll, rule, threat, user, group (+ source/destination) which I believe are all to be expected ( as FYI user info is collected as part of process).

marshallmain added 6 commits September 15, 2020 17:23

Add ransomware fields and event

008f672

Remove file and malware_classification fieldsets from ransomware event

b645f8d

Add child_pids and files.previous_extension to ransomware

441d18d

Add normalize to child_pids

f67a5fc

merge master

350a23c

Run make with updated tooling

1b647bc

magermark reviewed Oct 16, 2020

View reviewed changes

wburgess1 reviewed Oct 19, 2020

View reviewed changes

Update ransomware descriptions

546de84

jonathan-buttner reviewed Oct 19, 2020

View reviewed changes

marshallmain added 2 commits October 19, 2020 14:11

Add source and destination fields to ransomware event

0a3134d

Regenerate files with ransomware event changes

656a032

jonathan-buttner approved these changes Oct 20, 2020

View reviewed changes

wburgess1 reviewed Oct 20, 2020

View reviewed changes

files.metrics double -> keyword, rename previous_extension to origina…

a851b4d

…l.extension

marshallmain added 2 commits October 22, 2020 10:45

Make files nested for searching

961b4a6

Update path and extension descriptions

c8b4972

marshallmain merged commit 576bf76 into elastic:master Oct 26, 2020

ferullo mentioned this pull request Jun 21, 2022

Add more Endpoint fields #265

Merged

5 tasks

Add ransomware alert schema #96

Add ransomware alert schema #96

Uh oh!

Conversation

marshallmain commented Oct 16, 2020

magermark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magermark Oct 16, 2020 • edited by wburgess1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

wburgess1 Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wburgess1 Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wburgess1 Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wburgess1 Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-buttner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marshallmain Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wburgess1 Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wburgess1 commented Oct 22, 2020

Labels

4 participants

magermark Oct 16, 2020 •

edited by wburgess1

Loading

wburgess1 Oct 19, 2020 •

edited

Loading

wburgess1 Oct 19, 2020 •

edited

Loading

wburgess1 Oct 19, 2020 •

edited

Loading

wburgess1 Oct 19, 2020 •

edited

Loading

marshallmain Oct 19, 2020 •

edited

Loading

wburgess1 Oct 20, 2020 •

edited

Loading