Skip to content

Commit b845b1d

Browse files
efd6efd6
authored
cisco_secure_email_gateway: fix timeout in grok for consolidated events (#6879)
This works around the timeout by splitting the initial section of the pattern from the final message part. The second part is more expensive because of the potential of significant backtracking.
1 parent 18ff909 commit b845b1d

File tree

5 files changed

+317
-13
lines changed

5 files changed

+317
-13
lines changed

‎packages/cisco_secure_email_gateway/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.10.1"
3+
changes:
4+
- description: Fix grok timeout on expensive consolidated events logs.
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/6879
27
- version: "1.10.0"
38
changes:
49
- description: Convert dashboard to lens.

‎packages/cisco_secure_email_gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,4 +9,6 @@
99
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-023|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=423A4DF759243122B64F-7941F28E57A4 ESAMID=4086421 ESAICID=13956459 ESADCID=2522340 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH endTime=Thu Nov 24 13:39:24 2022 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'image002.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '30bf618599d8784ebcf38769f8b524b40dc20d2ba262a1e4052d24711abcd064'}, 'BodyScanner': {}}, 'image001.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7de9d8514c142887d11821fd30faddc693d192efdd19dfb6459872a1be63dcfa'}, 'BodyScanner': {}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Nov 24 13:39:16 2022 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=MATCH act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=716707 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=3.5 ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Favorable sourceHostName=example.cisco.com ESASenderGroup=ACCEPTLIST sourceAddress=1.128.3.4 msg='RE: SR 312312 : consolidate event log' ESATLSInCipher=KWLDS-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESATLSOutCipher=HDKWA-RSA-AES256-JMB-SHA384 ESATLSOutConnStatus=Success ESATLSOutProtocol=TLSv1.2 ESADaneHost=testdomain.com ESADaneStatus=success ESADHASource=1.128.3.4 ESADMARCVerdict=TempFailure cs5Label=ESAMsgLanguage cs5=English ESAMARAction={'action':'<>';'succesful_rcpts'='<>';'failed_recipients'='<>';'filename'='<>'} ESAMsgTooBigFromSender=true ESARateLimitedIP=1.128.3.4
1010
<166>Apr 03 12:20:40 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=164229 ESAICID=62908 ESADCID=47845 ESAAMPVerdict=SKIPPED ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE ESACFVerdict=NO_MATCH endTime=Mon Nov 14 15:40:48 2022 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Mon Nov 14 15:40:47 2022 deviceInboundInterface=IncList deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=NOT_EVALUATED act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=1411 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 ESAReplyTo=example.com cfp1Label=SBRSScore cfp1=5.2 ESASDRDomainAge=1 month cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral sourceHostName=example.cisco.com ESASenderGroup=UNKNOWNLIST sourceAddress=1.128.3.4 msg="Demande d'achat Econocom Products and Solutions, ref: SSAY-MEDECIN3"
1111
<14>Jun 12 14:01:33 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=4235746AE34E6DD44EB8-CE101C994AA5 ESAMID=5181473 ESAICID=17267358 ESADCID=3036000 endTime=Mon Jun 12 14:01:31 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAFriendlyFrom=jean sistin <jsistin@gmail.com> ESAGMVerdict=NEGATIVE startTime=Mon Jun 12 14:01:27 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=jsistin@gmail.com cs1Label=MailPolicy cs1=Test quarantaine utilisateur cs2Label=SenderCountry cs2=United States ESAMFVerdict=MATCH act=DELIVERED cs4Label=ExternalMsgID cs4='<CAKEaRS+qOfSncg8ppnitd\\=jqiqDs28MGVOLwYCdb3PZh7S-3Yw@mail.gmail.com>' ESAMsgSize=3078 ESAOFVerdict=NEGATIVE duser=fpenigaud@exaprobe.com ESAHeloDomain=mail-oi1-f182.google.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=3.4 ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'jsistin@gmail.com'}} sourceHostName=mail-oi1-f182.google.com ESASenderGroup=ACCEPTLIST sourceAddress=1.128.3.4 msg='test url' ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESATLSOutCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSOutConnStatus=Success ESATLSOutProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=NEGATIVE ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH
12-
12+
<14>Jul 04 06:21:54 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=E7DEF468022C4EB09683-9A331A42E1F7 ESAMID=54376810 ESAICID=43587623 endTime=Tue Jul 4 06:21:54 2023 ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAAttachmentDetails={'meeting.ics': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '17ae79446b4ec3baf161704831970aac49457d32935c6383c2a45aed136a99df'}, 'BodyScanner': {}}} ESAFriendlyFrom=River <river@this.example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul 4 16:12:44 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=river@this.example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=New Zealand ESAMFVerdict=MATCH act=DQ ESAFinalActionDetails=Message held temporarily in Delay Quarantine cs4Label=ExternalMsgID cs4='<2403354681.734500.1688449973515.mail.lion@example.com>' ESAMsgSize=18675 ESAOFVerdict=NEGATIVE duser=smith@example.com ESAHeloDomain=vm-lion.dmz ESAHeloIP=89.160.20.128 cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'river@this.example.com'}, 'helo': {'result': 'None', 'sender': 'postmaster@vm-lion.dmz'}} sourceHostName=company.example.com sourceAddress=89.160.20.128 msg='Accept: Cisco - SOLUTIONS' ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=FA_PENDING ESAASVerdict=NEGATIVE ESAAVVerdict=NOT_EVALUATED ESACFVerdict=MATCH
13+
<14>Jul 04 06:21:54 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=30543A3AB9E54FA8AAC1-FB812C95028D ESAMID=238746 ESAICID=435897324 ESADCID=34809573 endTime=Tue Jul 4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.142 ESAFriendlyFrom=Will <irobot@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul 4 15:14:29 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=irobot@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<MDlhMDg0MjY0NmE2OWFkNTZhMzA2NDA0MDVkZWNlZWVlYzI3MjMyYmI5YWJlNDMxM2UxOGVjZTBiNGZmOGZmYSAgLQo@hotmail.com>' ESAMsgSize=12312 ESAOFVerdict=NEGATIVE duser=alfombra@example.com ESAHeloDomain=mail-q6by9-a42.google.com ESAHeloIP=81.2.69.192 cfp1Label=SBRSScore cfp1=2.7 ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'irobot@example.com'}, 'helo': {'result': 'None', 'sender': 'postmaster@mail-q6by9-a42.google.com'}} sourceHostName=mail-q6by9-a42.google.com ESASenderGroup=ACCEPTLIST sourceAddress=81.2.69.192 msg='IE : Crayons' ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=MATCH
14+
<14>Jul 04 06:21:54 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-032|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=4FEF3A4372664BCCB404-20EE1767D434 ESAMID=786324 ESAICID=35635425 ESADCID=970897 endTime=Tue Jul 4 06:21:54 2023 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=81.2.69.144 ESAFriendlyFrom=Beaches <playas@example.com> ESAGMVerdict=NEGATIVE startTime=Tue Jul 4 14:42:34 2023 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=westinghouse-thoreau\\=example.com@example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=NZ ESAMFVerdict=MATCH act=QUARANTINED ESAFinalActionDetails=To SPAM cs4Label=ExternalMsgID cs4='<490b2a15fa4742331779cdaa4e@example.com>' ESAMsgSize=20668 ESAOFVerdict=NEGATIVE duser=thoreau@example.com ESAHeloDomain=example.com ESAHeloIP=89.160.20.112 ESAReplyTo=lane@example.com cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral ESASPFVerdict={'mailfrom': {'result': 'Pass', 'sender': 'westinghouse-thoreau=example.com@example.com'}, 'helo': {'result': 'Pass', 'sender': 'postmaster@example.com'}} sourceHostName=example.com sourceAddress=89.160.20.112 msg=\"Totally not suspicious email subject\" ESATLSInCipher=ECDHE-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESAAMPVerdict=SKIPPED ESAASVerdict=SUSPECT ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH

0 commit comments

Comments
 (0)