Spark Summit, profile picture
Uploaded bySpark Summit
PDF, PPTX6,576 views

Analyzing Log Data With Apache Spark

The document discusses analyzing log data using Apache Spark. It covers challenges with log data like schema mediation and feature engineering to transform log records into vectors. It also discusses visualizing the structured data using dimensionality reduction techniques like PCA and self-organizing maps to find outliers. The document provides examples of analyzing log data to identify the most frequent error levels and applications generating logs.

Data & Analytics◦
In this document
Powered by AI

Introduction to analyzing log data using Apache Spark, presented by William Benton from Red Hat.

Overview of challenges faced with log data, including SQL queries for critical messages and error levels across different services.

Methods for collecting and ingesting log data, including normalization, warehousing in Elasticsearch, and caching for analysis.

Discussion on schema mediation in log data management, highlighting key metadata elements such as timestamp and log level.

Techniques for exploring structured log data using aggregation and group operations on log levels and application names with numerical counts.

Transforming log records into feature vectors and understanding categorical feature similarities.

Mathematical approaches to measure similarity and distances between feature sets in log data analysis.

Discussion on identifying interesting features from log data, including significant info and critical messages.

Presentation of multidimensional data through visualizations, with specific examples highlighting data points.

Introduction to Principal Component Analysis (PCA) for dimensionality reduction in data visualization.

Overview of tree-based methods for analyzing and classifying log data.

Techniques for using Self-Organizing Maps to find patterns and outliers in log data.

Methodology for detecting outliers in log data, including statistical approaches and percentage metrics.

Process for performing on-line Self-Organizing Map training to continuously learn from log data.

Batch training methods for Self-Organizing Maps with a focus on parallel processing.

Discussion on strategies and best practices for sharing machine learning models and feature encoders.

Best practices for integrating Spark with ElasticSearch, managing memory, and effective feature engineering.

Download as PDF, PPTX
1 / 112
2 / 112
3 / 112
4 / 112
5 / 112
6 / 112
7 / 112
8 / 112
9 / 112
10 / 112
11 / 112
12 / 112
13 / 112
14 / 112
15 / 112
16 / 112
17 / 112
18 / 112
19 / 112
20 / 112
21 / 112
22 / 112
23 / 112
24 / 112
25 / 112
26 / 112
27 / 112
28 / 112
29 / 112
30 / 112
31 / 112
32 / 112
33 / 112
34 / 112
35 / 112
36 / 112
37 / 112
38 / 112
39 / 112
40 / 112
41 / 112
42 / 112
43 / 112
44 / 112
45 / 112
46 / 112
47 / 112
48 / 112
49 / 112
50 / 112
51 / 112
52 / 112
53 / 112
54 / 112
55 / 112
56 / 112
57 / 112
58 / 112
59 / 112
60 / 112
61 / 112
62 / 112
63 / 112
64 / 112
65 / 112
66 / 112
67 / 112
68 / 112
69 / 112
70 / 112
71 / 112
72 / 112
73 / 112
74 / 112
75 / 112
76 / 112
77 / 112
78 / 112
79 / 112
80 / 112
81 / 112
82 / 112
83 / 112
84 / 112
85 / 112
86 / 112
87 / 112
88 / 112
89 / 112
90 / 112
91 / 112
92 / 112
93 / 112
94 / 112
95 / 112
96 / 112
97 / 112
98 / 112
99 / 112
100 / 112
101 / 112
102 / 112
103 / 112
104 / 112
105 / 112
106 / 112
107 / 112
108 / 112
109 / 112
110 / 112
111 / 112
112 / 112
Analyzing log data with Apache Spark William Benton Red Hat Emerging Technology
BACKGROUND
Challenges of log data
Challenges of log data SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
Challenges of log data 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
Challenges of log data postgres httpd syslog INFO INFO WARN CRIT DEBUG INFO GET GET GET POST WARN WARN INFO INFO INFO GET (404) INFO (ca. 2000)
Challenges of log data postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN (ca. 2016)
Challenges of log datapostgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN How many services are generating logs in your datacenter today?
DATA INGEST
Collecting log data collecting Ingesting live log data via rsyslog, logstash, fluentd normalizing Reconciling log record metadata across sources warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Collecting log data warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Collecting log data warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
Schema mediation
Schema mediation
Schema mediation
Schema mediation timestamp, level, host, IP addresses, message, &c. rsyslog-style metadata, like app name, facility, &c.
logs .select("level").distinct .map { case Row(s: String) => s } .collect Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … debug, notice, emerg, err, warning, crit, info, severe, alert
Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert
Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert This class must be declared outside the REPL!
FEATURE ENGINEERING
From log records to vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010
From log records to vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010 red pancakes orange waffles -> 00010000 -> 00101000
Similarity and distance
Similarity and distance
Similarity and distance (q - p) • (q - p)
Similarity and distance pi - qi i=1 n
Similarity and distance pi - qi i=1 n
Similarity and distance p • q p q
WARN INFO INFOINFO WARN DEBUGINFOINFOINFO WARN WARNINFO INFO INFO WAR INFO INFO Other interesting features host01 host02 host03
INFO INFOINFO DEBUGINFOINFO WARNNFO INFO INFO WARN INFO INFOINFO INFO INFO INFOINFOINFO INFOINFO INFO INFO INFO INFO WARN DEBUG Other interesting features host01 host02 host03
INFO INFOINFO INFO INFO INFOINFO INFO INFO INFO WARN DEBUG WARN INFO INFO EBUG WARN INFO INFO INFO INFO INFO INFO INFO INFO WARN WARN INFO WARN INFO Other interesting features host01 host02 host03
Other interesting features : Great food, great service, a must-visit! : Our whole table got gastroenteritis. : This place is so wonderful that it has ruined all other tacos for me and my family.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes.
Other interesting features INFO: Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes. See https://links.freevariable.com/nlp-logs/ for more!
VISUALIZING STRUCTURE and FINDING OUTLIERS
Multidimensional data
Multidimensional data [4,7]
Multidimensional data [4,7]
Multidimensional data [4,7] [2,3,5]
Multidimensional data [4,7] [2,3,5]
Multidimensional data [4,7] [2,3,5] [7,1,6,5,12,
 8,9,2,2,4, 7,11,6,1,5]
Multidimensional data [4,7] [2,3,5] [7,1,6,5,12,
 8,9,2,2,4, 7,11,6,1,5]
A linear approach: PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
A linear approach: PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
Tree-based approaches
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
Tree-based approaches yes no yes no if orange if !orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
Self-organizing maps
Self-organizing maps
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Finding outliers with SOMs
Outliers in log data
Outliers in log data 0.95
Outliers in log data 0.95 0.97
Outliers in log data 0.95 0.97 0.92
Outliers in log data 0.95 0.97 0.92 0.37 An outlier is any record whose best match was at least 4σ below the mean. 0.94 0.89 0.91 0.93 0.96
Out of 310 million log records, we identified 0.0012% as outliers.
Thirty most extreme outliers 10 Can not communicate with power supply 2. 9 Power supply 2 failed. 8 Power supply redundancy is lost. 1 Drive A is removed. 1 Can not communicate with power supply 1. 1 Power supply 1 failed.
SOM TRAINING in SPARK
On-line SOM training
On-line SOM training
On-line SOM training
On-line SOM training
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt at each step, we update each unit by adding its value from the previous step…
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt to the example that we considered…
On-line SOM training while t < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt scaled by a learning factor and the distance from this unit to its best match
On-line SOM training
On-line SOM training sensitive to learning rate not parallel sensitive to example order
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods)
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) update the state of every cell in the neighborhood of the best matching unit, weighting by distance
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) keep track of the distance weights we’ve seen for a weighted average
Batch SOM training for t in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) since we can easily merge multiple states, we can train in parallel across many examples
Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
over all partitions Batch SOM training
driver (using aggregate) workers
driver (using aggregate) workers
driver (using aggregate) workers
driver (using aggregate) workers What if you have a 3 mb model and 2,048 partitions?
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
driver (using treeAggregate) workers
SHARING MODELS BEYOND SPARK
Sharing models class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here }
Sharing models class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here } case class FrozenModel(entries: Array[Double], /* ... */ ) { }
Sharing models case class FrozenModel(entries: Array[Double], /* ... */ ) { } class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here def freeze: FrozenModel = // ... } object Model { def thaw(im: FrozenModel): Model = // ... }
Sharing models import org.json4s.jackson.Serialization import org.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) }
Sharing models import org.json4s.jackson.Serialization import org.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) } Also consider how you’ll share feature encoders and other parts of your learning pipeline!
PRACTICAL MATTERS
Spark and ElasticSearch Data locality is an issue and caching is even more important than when running from local storage. If your data are write-once, consider exporting ES indices to Parquet files and analyzing those instead.
Structured queries in Spark Always program defensively: mediate schemas, explicitly convert null values, etc. Use the Dataset API whenever possible to minimize boilerplate and benefit from query planning without (entirely) forsaking type safety.
Memory and partitioning Large JVM heaps can lead to appalling GC pauses and executor timeouts. Use multiple JVMs or off-heap storage (in Spark 2.0!) Tree aggregation can save you both memory and execution time by partially aggregating at worker nodes.
Interoperability Avoid brittle or language-specific model serializers when sharing models with non-Spark environments. JSON is imperfect but ubiquitous. However, json4s will serialize case classes for free! See also SPARK-13944, merged recently into 2.0.
Feature engineering Favor feature engineering effort over complex or novel learning algorithms. Prefer approaches that train interpretable models. Design your feature engineering pipeline so you can translate feature vectors back to factor values.
@willb • willb@redhat.com
 https://chapeau.freevariable.com THANKS!

More Related Content

ODP
Elasticsearch for beginners
byNeil Baker
 
PPTX
ELK Stack
byPhuc Nguyen
 
PDF
Designing Structured Streaming Pipelines—How to Architect Things Right
byDatabricks
 
PDF
What is in a Lucene index?
bylucenerevolution
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
PDF
Efficiently Building Machine Learning Models for Predictive Maintenance in th...
byDatabricks
 
PPTX
Mongo DB 성능최적화 전략
byJin wook
 
PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
Elasticsearch for beginners
byNeil Baker
 
ELK Stack
byPhuc Nguyen
 
Designing Structured Streaming Pipelines—How to Architect Things Right
byDatabricks
 
What is in a Lucene index?
bylucenerevolution
 
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
Efficiently Building Machine Learning Models for Predictive Maintenance in th...
byDatabricks
 
Mongo DB 성능최적화 전략
byJin wook
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 

What's hot

PDF
ClickHouse Intro
byYegor Andreenko
 
PDF
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
PDF
Introduction to elasticsearch
byhypto
 
PDF
Introduction to MongoDB
byMike Dirolf
 
PDF
EMR 플랫폼 기반의 Spark 워크로드 실행 최적화 방안 - 정세웅, AWS 솔루션즈 아키텍트:: AWS Summit Online Ko...
byAmazon Web Services Korea
 
PDF
Hyperspace for Delta Lake
byDatabricks
 
PDF
SQOOP PPT
byDushhyant Kumar
 
PDF
Delta Architecture
byPaulo Gutierrez
 
PDF
Technical Introduction to PostgreSQL and PPAS
byAshnikbiz
 
PDF
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
byYohei Onishi
 
PPTX
Singer, Pinterest's Logging Infrastructure
byDiscover Pinterest
 
PPTX
Process mining
byMhamed Ali Ben Mahmoud
 
PPTX
Introduction to Apache Spark
byRahul Jain
 
PPTX
Snowflake Architecture.pptx
bychennakesava44
 
PPTX
Elk
byCaleb Wang
 
PPTX
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
byDatabricks
 
PPTX
MongoDB presentation
byHyphen Call
 
PDF
Apache Spark Introduction
bysudhakara st
 
PDF
Inside MongoDB: the Internals of an Open-Source Database
byMike Dirolf
 
PDF
Deep dive into stateful stream processing in structured streaming by Tathaga...
byDatabricks
 
ClickHouse Intro
byYegor Andreenko
 
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
Introduction to elasticsearch
byhypto
 
Introduction to MongoDB
byMike Dirolf
 
EMR 플랫폼 기반의 Spark 워크로드 실행 최적화 방안 - 정세웅, AWS 솔루션즈 아키텍트:: AWS Summit Online Ko...
byAmazon Web Services Korea
 
Hyperspace for Delta Lake
byDatabricks
 
SQOOP PPT
byDushhyant Kumar
 
Delta Architecture
byPaulo Gutierrez
 
Technical Introduction to PostgreSQL and PPAS
byAshnikbiz
 
Building a Data Pipeline using Apache Airflow (on AWS / GCP)
byYohei Onishi
 
Singer, Pinterest's Logging Infrastructure
byDiscover Pinterest
 
Process mining
byMhamed Ali Ben Mahmoud
 
Introduction to Apache Spark
byRahul Jain
 
Snowflake Architecture.pptx
bychennakesava44
 
Elk
byCaleb Wang
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
byDatabricks
 
MongoDB presentation
byHyphen Call
 
Apache Spark Introduction
bysudhakara st
 
Inside MongoDB: the Internals of an Open-Source Database
byMike Dirolf
 
Deep dive into stateful stream processing in structured streaming by Tathaga...
byDatabricks
 

Similar to Analyzing Log Data With Apache Spark

PDF
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
byBig Data Joe™ Rossi
 
PPTX
Build a DataWarehouse for your logs with Python, AWS Athena and Glue
byMaxym Kharchenko
 
PDF
Polyglot Persistence in the Real World: Cassandra + S3 + MapReduce
bythumbtacktech
 
PDF
Tues 115pm cassandra + s3 + hadoop = quick auditing and analytics_yazovskiy
byAnton Yazovskiy
 
PDF
Scaling Security Threat Detection with Apache Spark and Databricks
byDatabricks
 
PDF
Monitoring Error Logs at Databricks
byDatabricks
 
PPTX
Observability - the good, the bad, and the ugly
byAleksandr Tavgen
 
PDF
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
byEuangelos Linardos
 
PDF
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
byNETWAYS
 
PDF
Hadoop, hive和scribe在运维方面的应用
byxshadowxc
 
PDF
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
byVictoriaMetrics
 
PDF
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
PDF
Data Science
byAhmet Bulut
 
PDF
A Practical Guide to Anomaly Detection for DevOps
byBigPanda
 
PDF
More Than Just The Tip Of The Iceberg.pdf
byMichal Gancarski
 
PPTX
System Logs Anomaly Detection Using Deep Learning
bySrinivas Kanakala
 
PDF
Cassandra Talk: Austin JUG
byStu Hood
 
PDF
Fluentd meetup #3
byTreasure Data, Inc.
 
PPTX
Challenges of monitoring distributed systems
byNenad Bozic
 
PDF
Data Science Lab Meetup: Cassandra and Spark
byChristopher Batey
 
OC Big Data Monthly Meetup #5 - Session 2 - Sumo Logic
byBig Data Joe™ Rossi
 
Build a DataWarehouse for your logs with Python, AWS Athena and Glue
byMaxym Kharchenko
 
Polyglot Persistence in the Real World: Cassandra + S3 + MapReduce
bythumbtacktech
 
Tues 115pm cassandra + s3 + hadoop = quick auditing and analytics_yazovskiy
byAnton Yazovskiy
 
Scaling Security Threat Detection with Apache Spark and Databricks
byDatabricks
 
Monitoring Error Logs at Databricks
byDatabricks
 
Observability - the good, the bad, and the ugly
byAleksandr Tavgen
 
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
byEuangelos Linardos
 
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
byNETWAYS
 
Hadoop, hive和scribe在运维方面的应用
byxshadowxc
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
byVictoriaMetrics
 
Keeping the fun in functional w/ Apache Spark @ Scala Days NYC
byHolden Karau
 
Data Science
byAhmet Bulut
 
A Practical Guide to Anomaly Detection for DevOps
byBigPanda
 
More Than Just The Tip Of The Iceberg.pdf
byMichal Gancarski
 
System Logs Anomaly Detection Using Deep Learning
bySrinivas Kanakala
 
Cassandra Talk: Austin JUG
byStu Hood
 
Fluentd meetup #3
byTreasure Data, Inc.
 
Challenges of monitoring distributed systems
byNenad Bozic
 
Data Science Lab Meetup: Cassandra and Spark
byChristopher Batey
 

More from Spark Summit

PDF
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
PDF
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
PDF
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
PDF
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
bySpark Summit
 
PDF
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
PDF
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
PDF
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
PDF
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
PDF
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
PDF
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
PDF
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
PDF
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
PDF
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
PDF
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
PDF
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
PDF
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
PDF
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
bySpark Summit
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
bySpark Summit
 
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
bySpark Summit
 
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
bySpark Summit
 
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
bySpark Summit
 
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
Apache Spark and Tensorflow as a Service with Jim Dowling
bySpark Summit
 
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
bySpark Summit
 
Next CERN Accelerator Logging Service with Jakub Wozniak
bySpark Summit
 
Powering a Startup with Apache Spark with Kevin Kim
bySpark Summit
 
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
bySpark Summit
 
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
bySpark Summit
 
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
bySpark Summit
 
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
bySpark Summit
 
Goal Based Data Production with Sim Simeonov
bySpark Summit
 
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
bySpark Summit
 
Getting Ready to Use Redis with Apache Spark with Dvir Volk
bySpark Summit
 
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
bySpark Summit
 
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
bySpark Summit
 

Recently uploaded

DOCX
Embolic Protection Devices Market 2025.docx
byshilpathakur2787
 
PPTX
Expanded_Value_Addition_Shifting_Project
bysauravbaranwal9001
 
PPTX
RADIATION SOURCE OF UV-SPECTROSCOPY.pptx
byvarshachandgudesvpm
 
PPTX
一比一原版澳洲西悉尼大学毕业证(Western Sydney University毕业证书)学历认证
by【一比一日本学位记日本工业大学成绩单】【Q微号:1954 292 140】(NIT学历认证)学位证
 
PDF
ML-Lecture 1- Decision Tree and Random Forest.pdf
byZahra Sadeghi
 
PPTX
G12-Practical-research-lesson 2 stem.pptx
byAtheenaFayeSeadan1
 
PPTX
加拿大萨斯卡彻温大学学历认证范本{U of S offer学费单U of S成绩单}补办学历认证
by西澳大利亚大学学位证书快速办理【Q微号:1954 292 140】UWA买毕业证
 
PPTX
Security Design Principles of Information Security.pptx
bybahadurali6795
 
PPTX
COMPARATIVE STUDY OF MPPT TECHNIQUES FOR PHOTOVOLTAIC SYSTEM
byssuser588d36
 
PDF
Something about Software Testing: taxonomies and more
byNunoSilva238662
 
PDF
IFRS17 Risk Adjustment Worked Example Part 1.pdf
bySyed Danish Ali
 
PDF
SFBA Splunk Usergroup meeting October 1, 2025
byBecky Burwell
 
DOCX
_Is it legal to buy Google Ads accounts_.docx
byBEST IT SMM
 
PPTX
Feed Formulation PEARSON'S SQUARE METHOD.pptx
bybaginareca
 
PDF
DP 900 Azure Data Fundamentals Quesions Microsoft
bynnakasone
 
PDF
fairer_finance_how-can-housing-wealth-bridge-the-later-life-funding-gap.pdf
byHenry Tapper
 
PPTX
module 1 on exploratory data analysis.pptx
byFathimaFareeda3
 
PPTX
ORDINAL UTILITY in the world of Finances' Fundamentals
byARJUN5330
 
PPTX
Industrial Maintenance Meeting by Slidesgo.pptx
bymohaamro
 
PDF
LS-BL273310-20C 10GBASE-BX20-U BiDi 1270nm-TX/1330nm-RX SFP+ 20km with DDM
byLINK-PP
 
Embolic Protection Devices Market 2025.docx
byshilpathakur2787
 
Expanded_Value_Addition_Shifting_Project
bysauravbaranwal9001
 
RADIATION SOURCE OF UV-SPECTROSCOPY.pptx
byvarshachandgudesvpm
 
一比一原版澳洲西悉尼大学毕业证(Western Sydney University毕业证书)学历认证
by【一比一日本学位记日本工业大学成绩单】【Q微号:1954 292 140】(NIT学历认证)学位证
 
ML-Lecture 1- Decision Tree and Random Forest.pdf
byZahra Sadeghi
 
G12-Practical-research-lesson 2 stem.pptx
byAtheenaFayeSeadan1
 
加拿大萨斯卡彻温大学学历认证范本{U of S offer学费单U of S成绩单}补办学历认证
by西澳大利亚大学学位证书快速办理【Q微号:1954 292 140】UWA买毕业证
 
Security Design Principles of Information Security.pptx
bybahadurali6795
 
COMPARATIVE STUDY OF MPPT TECHNIQUES FOR PHOTOVOLTAIC SYSTEM
byssuser588d36
 
Something about Software Testing: taxonomies and more
byNunoSilva238662
 
IFRS17 Risk Adjustment Worked Example Part 1.pdf
bySyed Danish Ali
 
SFBA Splunk Usergroup meeting October 1, 2025
byBecky Burwell
 
_Is it legal to buy Google Ads accounts_.docx
byBEST IT SMM
 
Feed Formulation PEARSON'S SQUARE METHOD.pptx
bybaginareca
 
DP 900 Azure Data Fundamentals Quesions Microsoft
bynnakasone
 
fairer_finance_how-can-housing-wealth-bridge-the-later-life-funding-gap.pdf
byHenry Tapper
 
module 1 on exploratory data analysis.pptx
byFathimaFareeda3
 
ORDINAL UTILITY in the world of Finances' Fundamentals
byARJUN5330
 
Industrial Maintenance Meeting by Slidesgo.pptx
bymohaamro
 
LS-BL273310-20C 10GBASE-BX20-U BiDi 1270nm-TX/1330nm-RX SFP+ 20km with DDM
byLINK-PP
 

Analyzing Log Data With Apache Spark

  • 1.
    Analyzing log datawith Apache Spark William Benton Red Hat Emerging Technology
  • 2.
  • 3.
  • 4.
    Challenges of logdata SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
  • 5.
    Challenges of logdata 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 SELECT hostname, DATEPART(HH, timestamp) AS hour, COUNT(msg) FROM LOGS WHERE level='CRIT' AND msg LIKE '%failure%' GROUP BY hostname, hour
  • 6.
    Challenges of logdata postgres httpd syslog INFO INFO WARN CRIT DEBUG INFO GET GET GET POST WARN WARN INFO INFO INFO GET (404) INFO (ca. 2000)
  • 7.
    Challenges of logdata postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN (ca. 2016)
  • 8.
    Challenges of logdatapostgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN postgres httpd syslog INFO WARN GET (404) CRIT INFO GET GET GET POST INFO INFO INFO WARN CouchDB httpd Django INFO CRITINFO GET POST INFO INFO INFO WARN haproxy k8s INFO INFO WARN CRITDEBUG WARN WARN INFO INFOINFO INFO Cassandra nginx Rails INFO CRIT INFO GET POST PUT POST INFO INFO INFOWARN INFO redis INFO CRIT INFOINFO PUT (500)httpd syslog GET PUT INFO INFO INFOWARN How many services are generating logs in your datacenter today?
  • 9.
  • 10.
    Collecting log data collecting Ingestinglive log data via rsyslog, logstash, fluentd normalizing Reconciling log record metadata across sources warehousing Storing normalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 11.
    Collecting log data warehousing Storingnormalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 12.
    Collecting log data warehousing Storingnormalized records in ES indices analysis cache warehoused data as Parquet files on Gluster volume local to Spark cluster
  • 13.
  • 14.
  • 15.
  • 16.
    Schema mediation timestamp, level,host, IP addresses, message, &c. rsyslog-style metadata, like app name, facility, &c.
  • 17.
    logs .select("level").distinct .map { caseRow(s: String) => s } .collect Exploring structured data logs .groupBy($"level", $"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … debug, notice, emerg, err, warning, crit, info, severe, alert
  • 18.
    Exploring structured data logs .groupBy($"level",$"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert
  • 19.
    Exploring structured data logs .groupBy($"level",$"rsyslog.app_name") .agg(count("level").as("total")) .orderBy($"total".desc) .show info kubelet 17933574 info kube-proxy 10961117 err journal 6867921 info systemd 5184475 … logs .select("level").distinct .as[String].collect debug, notice, emerg, err, warning, crit, info, severe, alert This class must be declared outside the REPL!
  • 20.
  • 21.
    From log recordsto vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010
  • 22.
    From log recordsto vectors What does it mean for two sets of categorical features to be similar? red green blue orange -> 000 -> 010 -> 100 -> 001 pancakes waffles aebliskiver omelets bacon hash browns -> 10000 -> 01000 -> 00100 -> 00001 -> 00000 -> 00010 red pancakes orange waffles -> 00010000 -> 00101000
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29.
    WARN INFO INFOINFO WARNDEBUGINFOINFOINFO WARN WARNINFO INFO INFO WAR INFO INFO Other interesting features host01 host02 host03
  • 30.
    INFO INFOINFO DEBUGINFOINFO WARNNFO INFOINFO WARN INFO INFOINFO INFO INFO INFOINFOINFO INFOINFO INFO INFO INFO INFO WARN DEBUG Other interesting features host01 host02 host03
  • 31.
    INFO INFOINFO INFO INFO INFOINFO INFOINFO INFO WARN DEBUG WARN INFO INFO EBUG WARN INFO INFO INFO INFO INFO INFO INFO INFO WARN WARN INFO WARN INFO Other interesting features host01 host02 host03
  • 32.
    Other interesting features :Great food, great service, a must-visit! : Our whole table got gastroenteritis. : This place is so wonderful that it has ruined all other tacos for me and my family.
  • 33.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK.
  • 34.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers.
  • 35.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes.
  • 36.
    Other interesting features INFO:Everything is great! Just checking in to let you know I’m OK. CRIT: No requests in last hour; suspending running app containers. INFO: Phoenix datacenter is on fire; may not rise from ashes. See https://links.freevariable.com/nlp-logs/ for more!
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42.
  • 43.
  • 44.
  • 45.
    A linear approach:PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
  • 46.
    A linear approach:PCA 0 0 0 1 1 0 1 0 1 0 0 0 1 0 0 0 1 1 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 0 0 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1
  • 48.
  • 49.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray
  • 50.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
  • 51.
    Tree-based approaches yes no yes no if orange if!orange if red if !red if !gray if !gray yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no yes no no yes yes no yes no
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
    Outliers in logdata 0.95
  • 61.
    Outliers in logdata 0.95 0.97
  • 62.
    Outliers in logdata 0.95 0.97 0.92
  • 63.
    Outliers in logdata 0.95 0.97 0.92 0.37 An outlier is any record whose best match was at least 4σ below the mean. 0.94 0.89 0.91 0.93 0.96
  • 65.
    Out of 310million log records, we identified 0.0012% as outliers.
  • 68.
    Thirty most extremeoutliers 10 Can not communicate with power supply 2. 9 Power supply 2 failed. 8 Power supply redundancy is lost. 1 Drive A is removed. 1 Can not communicate with power supply 1. 1 Power supply 1 failed.
  • 69.
  • 70.
  • 71.
  • 72.
  • 73.
  • 74.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt
  • 75.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt at each step, we update each unit by adding its value from the previous step…
  • 76.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt to the example that we considered…
  • 77.
    On-line SOM training whilet < iterations: for ex in examples: t = t + 1 if t == iterations: break bestMatch = closest(somt, ex) for (unit, wt) in neighborhood(bestMatch, sigma(t)): somt+1[unit] = somt[unit] + ex * alpha(t) * wt scaled by a learning factor and the distance from this unit to its best match
  • 78.
  • 79.
    On-line SOM training sensitiveto learning rate not parallel sensitive to example order
  • 80.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods)
  • 81.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) update the state of every cell in the neighborhood of the best matching unit, weighting by distance
  • 82.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) keep track of the distance weights we’ve seen for a weighted average
  • 83.
    Batch SOM training fort in (1 to iterations): state = newState() for ex in examples: bestMatch = closest(somt-1, ex) hood = neighborhood(bestMatch, sigma(t)) state.matches += ex * hood state.hoods += hood somt = newSOM(state.matches / state.hoods) since we can easily merge multiple states, we can train in parallel across many examples
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94.
    driver (using aggregate) workers Whatif you have a 3 mb model and 2,048 partitions?
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 101.
    Sharing models class Model(privatevar entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here }
  • 102.
    Sharing models class Model(privatevar entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here } case class FrozenModel(entries: Array[Double], /* ... */ ) { }
  • 103.
    Sharing models case classFrozenModel(entries: Array[Double], /* ... */ ) { } class Model(private var entries: breeze.linalg.DenseVector[Double], /* ... lots of (possibly) mutable state ... */ ) implements java.io.Serializable { // lots of implementation details here def freeze: FrozenModel = // ... } object Model { def thaw(im: FrozenModel): Model = // ... }
  • 104.
    Sharing models import org.json4s.jackson.Serialization importorg.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) }
  • 105.
    Sharing models import org.json4s.jackson.Serialization importorg.json4s.jackson.Serialization.{read=>jread, write=>jwrite} implicit val formats = Serialization.formats(NoTypeHints) def toJson(m: Model): String = { jwrite(som.freeze) } def fromJson(json: String): Try[Model] = { Try({ Model.thaw(jread[FrozenModel](json)) }) } Also consider how you’ll share feature encoders and other parts of your learning pipeline!
  • 106.
  • 107.
    Spark and ElasticSearch Datalocality is an issue and caching is even more important than when running from local storage. If your data are write-once, consider exporting ES indices to Parquet files and analyzing those instead.
  • 108.
    Structured queries inSpark Always program defensively: mediate schemas, explicitly convert null values, etc. Use the Dataset API whenever possible to minimize boilerplate and benefit from query planning without (entirely) forsaking type safety.
  • 109.
    Memory and partitioning LargeJVM heaps can lead to appalling GC pauses and executor timeouts. Use multiple JVMs or off-heap storage (in Spark 2.0!) Tree aggregation can save you both memory and execution time by partially aggregating at worker nodes.
  • 110.
    Interoperability Avoid brittle orlanguage-specific model serializers when sharing models with non-Spark environments. JSON is imperfect but ubiquitous. However, json4s will serialize case classes for free! See also SPARK-13944, merged recently into 2.0.
  • 111.
    Feature engineering Favor featureengineering effort over complex or novel learning algorithms. Prefer approaches that train interpretable models. Design your feature engineering pipeline so you can translate feature vectors back to factor values.
  • 112.