mimecast: log processing stage and improve document fingerprinting

[efd6]

Proposed commit message

The current ingest pipeline does not effectively distinguish documents that have come from the stages/log types of Mimecast email processing[1]: receipt, process and delivery. This can result in documents from different types being given the same document fingerprint and a subsequent ingest version error. So detect the log type of the event and use it and a small set of distinguishing fields to ensure we don't collide documents. The heuristics for log type detection and the set of fields chosen for fingerprinting are based on the documentation at [1] and the summary at [2].

[1]https://integrations.mimecast.com/documentation/tutorials/understanding-siem-logs/
[2]https://docs.google.com/spreadsheets/d/1zspKE-LjrlFztsguB3z5wCIFrN6X2yx5ZgC01mnLzuY/

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Fixes [mimecast] Events missing Message-ID header value cause _id fingerprint collisions #9048

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mbudge]

I've added some Go code I used to determine each Mimecast log type to this support ticket #01554878. I used more than 1 field for redundancy in case the field was missing. This code was running for 3-4 years and we didn't spot any problems.

Journal logs are important as they are for internal emails between staff, but journal logs aren't documented on the "Understanding SIEM logs" page. The code I've provided shows how to detect journal logs.

Thanks

[efd6]

I've added some Go code I used

Where is this?

[mbudge]

I've added some Go code I used

Where is this?

In support ticket #01554878 or emailed to Jamie.

[efd6]

Summary:

headerFrom|SpamLimit|Error -> receipt Hld|AttNames -> process Delivered|Snt|ReceiptAck|Latency -> delivery urlCategory|md5|sha1|fileName -> protection RcptActType -> journal 

[mbudge]

Hi,

Just seen they have added a few more log types which makes this more complicated. It's more work but it does make the logs easier to use.

I'll email mimecast to ask if they will add the log type field, but they normally ignore customer requests.

headerFrom|SpamLimit|Error -> receipt
Hld|AttNames|Act|AttCnt|MsgSize -> process
Delivered|Snt|ReceiptAck|Latency -> delivery
RcptActType -> journal
Virus and not (Rcpt or headerFrom) -> avlog
SourceIP and not (Reason or urlCategory or headerFrom or Rcpt) -> spam
ScanResultInfo -> internal-email-protect
CustomName or CustomThreatDictionary SimilarCustomExternalDomain or SimilarInternalDomain or SimilarMimecastExternalDomain or TaggedExternal or TaggedMalicious or ThreatDictionary -> impersonation-protect
UrlCategory and reason and not (ScanResultInfo or ScanResultInfo) -> email-protect
fileName or md5 or sha1 or sha256 and not (Virus or SenderDomainInternal) -> attachment-protect

I recommend full regression testing using the sample logs on this page. We use Mimecast to support Data Loss Prevention and phishing incident response processes so we need to guarantee there's no data loss.

https://integrations.mimecast.com/documentation/tutorials/understanding-siem-logs/

[mbudge]

We were getting all the logs listed in the Understanding SIEM logs page through this endpoint > /api/audit/get-siem-logs

However I can see mimecast also have several other endpoints for url, impersonation and attachment protect. If using those endpoints then setting the log type field can be done in the respective ingest pipeline. However there siem logs endpoint also might send the same TTP logs. It's not clear from their documentation.

https://integrations.mimecast.com/documentation/endpoint-reference/logs-and-statistics/

I've got a python script I can use to try and work this out, and maybe provide some sample data. Might be later today.

[efd6]

Test case generator here https://go.dev/play/p/U6YWZHJHKHJ

[chrisberkhout]

I see the log_type value is extracted from the file name in the Content-Disposition header of the response. We request compressed data and can get a zip file of json log files, which HTTP JSON will merge, losing those log file names and with them their log types. So then we use heuristics to determine the type, and store that in stage.

It would have been nice if they had a type field (and possibly a stage field) in the contents of their log messages.

If I was building this I would strongly consider not requesting zip files, so we always get the type from the file name. There could still be http compression.

Although three "stages" are discussed in the documentation, those stages can be inferred from the type, and we're setting stage to a larger number of values that seem to be types.

Should our stage field really be separate from type?

[efd6]

The background for the change is largely in the issue. This is where the justification for the larger number of categorisations lives.

[chrisberkhout]

The background for the change is largely in the issue. This is where the justification for the larger number of categorisations lives.

I skimmed the issue before but re-reading it I'm still not sure. In the following table of what I could see in the PR and the documentation, aren't we only adding stage because we're getting incomplete data in log_type?

I just want to be clear about whether they're trying to identify the same categories or not. If they are the same thing, it might still be good to keep both because they are populated using different methods and results may vary slightly.

mimecast.log_type	mimecast.stage	Name in documentation
receipt	receipt	Receipt logs
process	process	Process logs
delivery	delivery	Delivery logs
?	avlog	AV logs
?	spam	Spam Event Thread logs
?	internal-email-protect	Target Threat Protection - Internal Email Protect logs
?	impersonation-protect	Targeted Threat Protection - Impersonation Protect logs
ttp_url	protection	Targeted Threat Protection - URL Protect logs
?	attachment-protect	Targeted Threat Protection - Attachment Protect logs
jrnl	journal	?
?	email-protect	?

Update: the table above isn't quite right. Please note:

• journal is an undocumented type
• protection matches several of the documented types

[jamiehynds]

@efd6 Mimecast have given us access to their API. If you need access to test against, happy to provide.

[mbudge]

Just wondering if this is fixed?

[efd6]

This has not been merged yet.

[chrisberkhout]

I'll take a proper look at this on Thursday next week.

Feel free to ignore these until I've done a proper review, but here's what I saw so far:

• in general the approach looks good
• some typos: avlog vs avlogs, rprocess vs process
• naming inconsistency: log type vs stage
• maybe better to ignore the field key case?
• maybe HashSet operations would be helpful?

[efd6]

Addressed all bar the naming consistency; given the information we have, I'm not sure what the approach should be for that.

[efd6]

/test

[chrisberkhout]

I like the way you reconciled stage and log_type.

In the spreadsheet I see that the "Information" column would be 1.0 if the type had equal numbers of present and absent fields and a lower value if it's less balanced in either direction. I'd be interested to know what exactly this measure is or what it's adapted from or inspired by.

Some changes

• The stage field definition can go away and the README should be regenerated.
• The proposed commit message may mentioned stages as something we need to distinguish between, but it should be updated to focus on log_type.
• The proposed commit message should mention sample_event.json changes in other data streams as an additional change.
• It would be good to have a test case for the jrnl type.
• In the spreadsheet I didn't see right away that there are hidden columns. I think it's better to show them and let the reader hide things if necessary.

Classification code

The classification code seems like it'll work. What follows is more about my though process and some commentary rather than problems that need to be fixed.

Reading the code for classification, it was clear immediately how the definite positives were handled. After that I could see that based on other fields there was some penalty and then some positive score, but it took a bit to see how the data and logic for these are related. I guessed that if I'd understood it correctly it could be written with more compact data and code (although a bit more computation) by using more set operations and I came up with this equivalent:

### NOTE LOG TYPE - script: lang: painless params: known_shared_keys: [acode, act, attcnt, attsize, cphr, dir, fileext, filemime, headerfrom, ip, md5, rcpt, recipient, rejcode, rejinfo, rejtype, route, senderdomain, sha1, sha256, size, sourceip, sourceip,, tlsver, url, urlcategory, virus] types: attachment-protect: unique_keys: [filename] shared_keys: [fileext, filemime, ip, md5, recipient, route, senderdomain, sha1, sha256, size] avlog: unique_keys: [customerip, mimecastip, senderdomaininternal] shared_keys: [fileext, filemime, ip, md5, recipient, route, senderdomain, sha1, sha256, size, virus] delivery: unique_keys: [attempt, delivered, err, latency, receiptack, snt, usetls] shared_keys: [acode, attcnt, attsize, cphr, dir, ip, rcpt, rejcode, rejinfo, rejtype, route, tlsver] impersonation-protect: unique_keys: [customname, customthreatdictionary, definition, hits, internalname, newdomain, replymismatch, similarcustomexternaldomain, similarinternaldomain, similarmimecastexternaldomain, taggedexternal, taggedmalicious, threatdictionary] shared_keys: [acode, ip, recipient, route] internal-email-protect: unique_keys: [scanresultinfo] shared_keys: [acode, recipient, route, url, urlcategory] jrnl: unique_keys: [rcptacttype] shared_keys: [acode, dir, rcpt] process: unique_keys: [attnames, hld, ipinternalname, ipnewdomain, ipreplymismatch, ipsimilardomain, ipthreaddict, msgsize] shared_keys: [acode, act, attcnt, attsize] receipt: unique_keys: [action, error, spaminfo, spamlimit, spamprocessingdetail, spamscore] shared_keys: [acode, act, cphr, dir, headerfrom, ip, rcpt, rejcode, rejinfo, rejtype, tlsver, virus] url-protect: unique_keys: [reason] shared_keys: [recipient, route, senderdomain, sourceip, url, urlcategory] spam: unique_keys: [] shared_keys: [acode, headerfrom, recipient, route, senderdomain, sourceip] if: ctx.mimecast instanceof Map source: | // Canonicalise keys to lowercase. If this causes issues in future // because case becomes significant, this table space optimisation // will need to be reverted. def keys = new HashSet(); for (def k: ctx.mimecast.keySet()) { keys.add(k.toLowerCase()); } for (typeEntry in params.types.entrySet()) { def uniqueKeysPresent = typeEntry.getValue().unique_keys.clone(); uniqueKeysPresent.retainAll(keys); if (uniqueKeysPresent.size() > 0) { ctx.mimecast.log_type = typeEntry.getKey(); return; } } def maxNumSharedKeysPresent = -1; def bestTypes = []; for (typeEntry in params.types.entrySet()) { def excessKnownSharedKeys = keys.clone(); excessKnownSharedKeys.retainAll(params.known_shared_keys); excessKnownSharedKeys.removeAll(typeEntry.getValue().shared_keys); if (excessKnownSharedKeys.size() > 0) { continue; } def sharedKeysPresent = typeEntry.getValue().shared_keys.clone(); sharedKeysPresent.retainAll(keys); if (sharedKeysPresent.size() == maxNumSharedKeysPresent) { bestTypes.add(typeEntry.getKey()); } else if (sharedKeysPresent.size() > maxNumSharedKeysPresent) { maxNumSharedKeysPresent = sharedKeysPresent.size(); bestTypes = [typeEntry.getKey()]; } } ctx.mimecast.log_type = bestTypes; return; 

Thinking about robustness, if types are changed, there would be problems with:

• adding a previously unique key to another type (both types would be identified as the first one in the types list)
• adding a shared key to another type (the altered type would be rejected because the document has an excess known shared key)
• adding a new type with new fields (I think it would match all known types)

Maybe this is an acceptable trade-off. Adding unknown keys would probably be the more common case and should not cause problems.

Alternative approaches would be:

• Matching lists of fields exactly: assuming all fields are sent for a given type, this would match perfectly. New types or modified types would be not matched rather than mismatched.
• Calculating similarity scores once rather than having a 3-step selection process: perhaps by counting excess and missing fields compared to each type, and weighting those counts heavier for fields expected in fewer types. Below a certain minimal similarity threshold it would be considered a new type. I think it's more elegant but also more complicated and could miss some easy cases (like the definite positive cases) unless parameters are tuned correctly.

[efd6]

In the spreadsheet I see that the "Information" column would be 1.0 if the type had equal numbers of present and absent fields and a lower value if it's less balanced in either direction. I'd be interested to know what exactly this measure is or what it's adapted from or inspired by.

This is purely a heuristic that I was using to help me understand/direct where I should look first. It's not directly used in the rules here. I don't completely remember what I was thinking, but from the name and the shape of the formula, I imagine that I wrote something that's close to S for the field set for each type (this is a pretty common thing for me to use when making cut decisions).

The stage field definition can go away and the README should be regenerated.

Yep, missed this. Removed.

The proposed commit message may mentioned stages as something we need to distinguish between, but it should be updated to focus on log_type.

New proposed commit message:

The current ingest pipeline does not effectively distinguish documents that have come from the stages/log types of Mimecast email processing[1]: receipt, process and delivery. This can result in documents from different types being given the same document fingerprint and a subsequent ingest version error. So detect the log type of the event and use it and a small set of distinguishing fields to ensure we don't collide documents. The heuristics for log type detection and the set of fields chosen for fingerprinting are based on the documentation at [1] and the summary at [2]. [1]https://integrations.mimecast.com/documentation/tutorials/understanding-siem-logs/ [2]https://docs.google.com/spreadsheets/d/1zspKE-LjrlFztsguB3z5wCIFrN6X2yx5ZgC01mnLzuY/ 

It would be good to have a test case for the jrnl type.

I believe this is already tested, though this is done via the file path.

In the spreadsheet I didn't see right away that there are hidden columns. I think it's better to show them and let the reader hide things if necessary.

Yep, this was just the working state. Unhidden.

Thinking about robustness, if types are changed, there would be problems with:

• adding a previously unique key to another type (both types would be identified as the first one in the types list)
• adding a shared key to another type (the altered type would be rejected because the document has an excess known shared key)
• adding a new type with new fields (I think it would match all known types)

Yes, all these are intentional. The rationale is that the vendor data is intrinsically brittle, so we make the choice to maintain the maximum data and be obviously misformed (scalar v array — not actually malformed which would lose the data) while still being queryable.

WRT the alternative, the approach taken is intended to balance being reasonably clear (explicit) against being onerously long. The successive approach is taken to avoid the more expensive work that's required for score calculations unless it's demonstrated to be necessary.

Ideally, the vendor would just provide the log type explicitly; a customer has filed a request with them for this, so if that is accepted and implemented (frankly, it should be; requiring the user to jump though hoops to effectively use the data is unreasonably brittle), then this all becomes moot.

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ad45dd8

History

• 💚 Build #10208 succeeded 6b55398
• 💔 Build #10206 failed c9690209cae623cd159874392df18ab0a6362d6c
• 💔 Build #10203 failed 76a9a8729ce3b9be4ec9584e5960510dea29136e
• 💚 Build #10147 succeeded 35ad8bf
• 💚 Build #9505 succeeded e6a0d72
• 💔 Build #9504 failed 592cea81c36e2443f62267763ab37bf97c074579

cc @efd6

[chrisberkhout]

Yep, all sounds reasonable.

[elasticmachine]

Package mimecast - 1.24.0 containing this change is available at https://epr.elastic.co/search?package=mimecast