![Big Data Analytics 14 Security for Business Innovation Council report, “When Advanced Persistent Threats Go Mainstream,” Chuck Hollis VP – CTO, EMC Corporation “The core of the most effective [advanced threat] response appears to be a new breed of security analytics that help quickly detect anomalous patterns -- basically power tools in the hands of a new and important sub-category of data scientists: the security analytics expert..” “[Security teams need] an analytical engine to sift through massive amounts of real-time and historical data at high speeds to develop trending on user and system activity and reveal anomalies that indicate compromise.” 14](https://image.slidesharecdn.com/using-big-data-to-counteract-advanced-threats-140521161152-phpapp01/75/Using-Big-Data-to-Counteract-Advanced-Threats-14-2048.jpg)
Uploaded byZivaro Inc
Using Big Data to Counteract Advanced Threats
The document discusses the use of big data in cybersecurity to address advanced threats that traditional security methods struggle to detect. It emphasizes the importance of collecting and analyzing large datasets to identify anomalies indicative of security breaches and offers a three-step approach to enhance threat detection and remediation. The presentation showcases Splunk's capabilities in machine data analytics and highlights the need for skilled practitioners to leverage big data effectively.
Using Big Data to Counteract Advanced Threats
- 1. 1 GOOD GUYS VSBAD GUYS: USING BIG DATA TO COUNTERACT ADVANCED THREATS Presented at RMISC 2014 by Kelly Feagans – Senior Sales Engineer, Splunk Dave Herrald – Principal Security Consultant, GTRI Content by Joe Goldberg, Splunk
- 2.
- 3. Security Presentation Template 3 BigData Advanced Threats
- 4. Here Comes theScary Part….. 4
- 5. Advanced Threats Outpacethe Defenders 5 Adversary You Time Technical Capabilities
- 6. Advanced Threats AreHard to Detect 6 100% Valid credentials were used 40 Average # of systems accessed 243 Median # of days before detection 63% Of victims were notified by external entity Source: Mandiant M-Trends Report 2012 and 2013
- 7. Advanced Threat Pattern– Not Signature Based 7 Infiltration Back Door Exfiltration Data GatheringRecon Phishing or web drive-by. Email has attached malware or link to malware Malware installs remote access toolkit(s) Malware obtains credentials to key systems and identifies valuable data Data is acquired and staged for exfiltration Data is exfiltrated as encrypted files via HTTP/S, FTP, DNS
- 8. 8 Traditional SIEMs MissThe Threats Limited view of security threats. Difficult to collect all data sources. Costly, custom collectors. Datastore w/schema. Inflexible search/reporting hampers investigations and threat detection Scale/speed issues impede ability to do fast analytics Difficult to deploy and manage; often multiple products
- 9. Better Defensive CybersecurityTools Needed 9
- 10. Here Comes TheSolution 10 Big Data
- 11. Big Data isUsed Across IT and the Business 11 IT Ops Security Compliance App Mgmt Fraud Business Intelligence Big Data
- 12. “Big Data” Definition Wikipedia: Collection of data sets so large and complex that it becomes difficult to process using database management tools Gartner: The Three Vs Data volume Data variety Data velocity Security has always been a Big Data problem; now it has a solution 12
- 13. Machine Data /Logs are Big Data 13 2013-08-09 16:21:38 10.11.36.29 98483 148 TCP_HIT 200 200 0 622 - - OBSERVED GET www.neverbeenseenbefore.com HTTP/1.1 0 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; InfoPath.1; MS-RTC LM 8; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; ) User John Doe," 08/09/2013 16:23:51.0128event_status="(0)The operation completed successfully. "pid=1300 process_image="John DoeDeviceHarddiskVolume1WindowsSystem32neverseenbefore.exe“ registry_type ="CreateKey"key_path="REGISTRYMACHINESOFTWAREMicrosoftWindows NTCurrentVersion Printers PrintProviders John Doe-PCPrinters{} NeverSeenbefore" data_type"" Endpoint Logs Web Proxy Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer name: ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and Settings/smithe/Local Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time: 2009-01-23 03:19:12,Inserted: 2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My CompanyACME Remote,Server: acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20 20130806041221.000000Caption=ACME-2975EBAdministrator Description=Built-in account for administering the computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20 TrueName=Administrator SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1 Status=Degradedwmi_ type=UserAccounts Anti-virus Authentications
- 14. Big Data Analytics 14 Securityfor Business Innovation Council report, “When Advanced Persistent Threats Go Mainstream,” Chuck Hollis VP – CTO, EMC Corporation “The core of the most effective [advanced threat] response appears to be a new breed of security analytics that help quickly detect anomalous patterns -- basically power tools in the hands of a new and important sub-category of data scientists: the security analytics expert..” “[Security teams need] an analytical engine to sift through massive amounts of real-time and historical data at high speeds to develop trending on user and system activity and reveal anomalies that indicate compromise.” 14
- 15. Step 1: CollectALL The Data in One Location Servers Service Desk Storage DesktopsEmail Web Call Records Network Flows DHCP/ DNS Hypervisor Custom Apps Industrial Control / HVAC Badges Databases Mobile Intrusion Detection Firewall Data Loss Prevention Anti- Malware Vulnerability Scans Traditional SIEM Authentication 15
- 16. Need Both Networkand Endpoint And Inbound/Outbound! 16
- 17. Enrich Indexed Datawith External Data / Lookups 17 Geo-IP Mapping 3rd-party threat intel Asset Info Prohibited Services / Apps Critical Network Segments / Honeypots Employee Info
- 18. Step 2: IdentifyThreat Activity 18 What‟s the M.O. of the attacker? (think like a criminal) What/who are the most critical assets and employees? What minute patterns/correlations in „normal‟ IT activities would represent „abnormal‟ activity? What in my environment is different/new/changed? What is rarely seen or standard deviations off the norm?
- 19. Big Data Solution 19 BigData Architecture Data Inclusion Model All the original data from any source No database schema to limit investigations/detection Lookups against external data sources Search & reporting flexibility Advanced correlations Math/statistics to baseline and find outliers/anomalies Real-time indexing and alerting “Known” and “Unknown” threat detection Scales horizontally to 100 TB+ a day on commodity H/W One product, UI, and datastore
- 20. Big Data Solutions Flat file datastore (not database), distributed search, commodity H/W More than a SIEM; can use outside security/compliance 20 Incident investigations/forensics, custom reporting, correlations, APT detection, fraud detection
- 21. Sample Correlation ofUnknown Threats 21 2013-08-09 16:21:38 10.11.36.29 98483 148 TCP_HIT 200 200 0 622 - - OBSERVED GET www.neverbeenseenbefore.com HTTP/1.1 0 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; InfoPath.1; MS-RTC LM 8;.NET CLR 3.0.4506.2152; ) User John Doe," 08/09/2013 16:23:51.0128event_status="(0)The operation completed successfully. "pid=1300 process_image="John DoeDeviceHarddiskVolume1WindowsSystem32neverseenbefore.exe“ registry_type ="CreateKey"key_path="REGISTRYMACHINESOFTWAREMicrosoftWindows NTCurrentVersion Printers PrintProviders John Doe-PCPrinters{} NeverSeenbefore" data_type"" 2013-08-09T12:40:25.475Z,,exch-hub-den-01,,exch-mbx-cup- 00,,,STOREDRIVER,DELIVER,79426,<20130809050115.18154.11234@acme.com>,johndoe@acme.com, ,685191,1,,, hacker@neverseenbefore.com , Please open this attachment with payroll information,, ,2013-08-09T22:40:24.975Z Endpoint Logs Web Proxy Email Server All three occurring within a 24-hour period Example Correlation - Spearphishing User Name User Name Rarely seen email domain Rarely visited web site User Name Rarely seen service
- 22. Fingerprints of anAdvanced Threat 22 What to Look For Why Data Source Attack Phase Rarely seen registry, service, DLL. Or they fail hash checks. Malware or remote access toolkit OS Back door Account creation or privilege escalation without corresponding IT service desk ticket Creating new admin accounts AD/ Service Desk logs Lateral movement A non-IT machine logging directly into multiple servers. Or chained logins. Threat accessing multiple machines AD /asset info Lateral movement For single employee: Badges in at one location, then logs in countries away Stealing credentials Badge/ VPN/ Auth Data gathering Employee makes standard deviations more data requests from file server with confidential data than normal Gathering confidential data for theft OS Data gathering Standard deviations larger traffic flows (incl DNS) from a host to a given IP Exfiltration of info NetFlow Exfiltration
- 23. Step 3: Remediateand Automate Where else in my environment do I see the “Indicators of Compromise” (IOC)? Remediate infected machines Fix weaknesses, including employee education Turn IOC into a real-time search for future threats 23
- 24. Security Realities… BigData is only as good as the data in it and people behind the UI No replacement for capable practitioners Put math and statistics to work for you Encourage IT Security creativity and thinking outside the box Fine tuning needed; always will be false positives 24
- 25. Recap 25 Step 1:Collect ALL The Data in One Location Step 2: Identify Threat Activity Step 3: Remediate and Automate
- 26. About Splunk BigData platform for ingesting machine data; desktop to 100+ TB/day Many use cases within security; also outside security Over 6500 customers total; 2800+ security customers Free download and tutorial at www.splunk.com 26
- 27. GTRI Splunk PracticeOverview Highlights: Splunk‟s 1st Elite Partner and one of only two Splunk Certified Training Centers in the U.S. GTRI provides end-to-end support for Splunk from pre-sales engineering to post- sales professional services, implementation, training and optimization Splunk‟s most credentialed partner in N. America: GTRI holds over 60 Splunk Certifications: 5 Certified Architects 6 Certified Solutions Engineers (SE-I & SE-2)
- 28.
Editor's Notes
- #21 What is not big data - data warehouses/database. They cannot take in all the original data. Also often batch-oriented, not real-time. Explain the key security use cases