Hadoop and Spark for the SAS Developer

Hadoop and Spark for the SAS Developer

• 1.
  Hadoop and Sparkfor the SAS Developer Richard Williamson | @superhadooper 10 June 2015
• 2.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience
• 3.
  3@SVDataScience © 2015SILICON VALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED.  My Background  Overview: SAS vs. Spark  Spark DataFrame vs. SAS Dataset  Spark SQL vs. SAS Proc SQL  Spark MLlib vs. SAS Stats  Spark Streaming  Questions? AGENDA
• 4.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience * http://en.wikipedia.org/wiki/SAS_%28software%29 ** http://techcrunch.com/2015/03/19/on-the-growth-of-apache-spark OVERVIEW: SAS vs. Spark SAS • SAS is the largest market-share holder in "advanced analytics" with 36.2% of the market as of 2012.* Spark • Launched in U.C. Berkeley’s AMPLab in 2009, Apache Spark has begun to catch on like wildfire during the last year and a half. Spark had more than 465 contributors in 2014, making it the most active project in the Apache Software Foundation and among big data open source projects globally.**
• 5.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience OVERVIEW: SAS vs. Spark SAS • Basic Programming model consists of SAS Data Step and SAS Procedures • SAS Datasets move data between processing steps Spark • Native language is Scala—allows generic data types and flexible programming model (Java and Python also supported) • RDDs (and now DataFrames) are used to move distributed datasets between processing steps
• 6.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience OVERVIEW: SAS vs. Spark SAS Code Snippet http://support.sas.com/kb/24/595.html data old; input state $ accttot; datalines; ca 7000 ca 6500 ca 5800 nc 4800 nc 3640 sc 3520 va 4490 va 8700 va 2850 va 1111; Spark Code Snippet import sqlContext.implicits._ case class OLD(state: String, accttot: Int) val oldList = List( OLD("va",1111), OLD("ca",7000), OLD("ca",6500), OLD("ca",5800), OLD("nc",4800), OLD("nc",3640), OLD("sc",3520), OLD("va",4490), OLD("va",8700), OLD("va",2850) )
• 7.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience OVERVIEW: SAS vs. Spark SAS Code Snippet http://support.sas.com/kb/24/595.html proc sort data=old; by state; data new; set old (drop= accttot); by state; if first.state then count=0; count+1; if last.state then output; proc freq; tables state / out=new(drop=percent) Spark Code Snippet val oldRDD = sc.parallelize(oldList) var oldDataFrame = oldRDD.toDF() oldDataFrame = oldDataFrame.orderBy("state”) oldRDD.aggregateByKey(0)((buffer, value) => buffer + value, (b1,b2) => b1 + b2).foreach(println) val newDataFrame = oldDataFrame.groupBy("state").count()
• 8.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark DataFrame vs. SAS Dataset Spark DataFrame — a distributed collection of data organized into named columns. https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#rename-of-schemardd-to-dataframe SAS Dataset — a SAS file stored in a SAS library organized as a table of observations (rows) and variables (columns) that can be processed by SAS software. http://support.sas.com/documentation/cdl/en/lrcon/62955/HTML/default/viewer.htm#a001005709.htm
• 9.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience How does Spark DataFrame differ from SAS Dataset? • built from ground up to be distributed and can be processed in parallel by multiple machines, whereas SAS Dataset has non-distributed roots • logical entity that is not necessarily paired with a serialized on-disk version, whereas a SAS Dataset has an on-disk manifestation Spark DataFrame vs. SAS Dataset
• 10.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark SQL vs. SAS Proc SQL • My first reaction to Spark SQL was, “This looks like Proc SQL” • SAS Proc SQL Simple Example: libname Example 'c:SASPROJECTS'; proc sql; create table newtable as select a.*, b.unique_consumer_id from Example.transactions as a, Example.consumer as b where a.ref_id=b.ref_id; quit;
• 11.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark SQL vs. SAS Proc SQL • Spark SQL Simple Example: val sqlContext = new org.apache.spark.sql.SQLContext(sc) val newtable = sqlContext.sql(“ select a.*, b.unique_consumer_id from transactions as a, consumer as b where a.ref_id=b.ref_id”)
• 12.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark MLlib vs. SAS Stats • Spark Mllib • https://spark.apache.org/docs/latest/mllib-guide.html • Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction • SAS Stats • Traditional Add-on package to SAS for Statistics
• 13.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark MLlib Example Data Prep case class Meetup(mdatehr: String, mdate: String, mhour: String) val meetup5 = meetup4.map(p => Meetup(p._1, p._2, p._3)) meetup5.registerTempTable("meetup5") val meetup6 = sqlContext.sql("select mdate,mhour,count(*) as rsvp_cnt from meetup5 where mdatehr >= '2015-02- 15 02' group by mdatehr,mdate,mhour") meetup6.registerTempTable("meetup6") sqlContext.sql("cache table meetup6”) val trainingData = meetup7.map { row => val features = Array[Double](row(24).toString().toDouble,row(0).toString().toDouble, … LabeledPoint(row(27).toString().toDouble, Vectors.dense(features))}
• 14.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark MLlib Example Regression Model val trainingData = meetup7.map { row => val features = Array[Double](1.0,row(0).toString().toDouble,row(1).toString().toDouble, … LabeledPoint(row(27).toString().toDouble, Vectors.dense(features))} val model = new RidgeRegressionWithSGD().run(trainingData) val scores = meetup7.map { row => val features = Vectors.dense(Array[Double](1.0,row(0).toString().toDouble, … row(23).toString().toDouble)) (row(25),row(26),row(27), model.predict(features))} scores.foreach(println)
• 15.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Spark Streaming val ssc = new StreamingContext(sc, Seconds(10)) val lines = KafkaUtils.createStream(ssc, ”localhost:2181", "meetupstream", Map("meetupstream" -> 10)).map(_._2) val sqlContext = new org.apache.spark.sql.SQLContext(sc) lines.foreachRDD(rdd => { val lines2 = sqlContext.jsonRDD(rdd) lines2.registerTempTable("lines2") val lines3 = sqlContext.sql("select event.event_id,event.event_name,event.event_url, event.time,guests, member.member_id, member.member_name, member.other_services.facebook.identifier as facebook_identifier, member.other_services.linkedin.identifier as linkedin_identifier, member.other_services.twitter.identifier as twitter_identifier,member.photo, mtime,response, rsvp_id,venue.lat,venue.lon,venue.venue_id,venue.venue_name,visibility from lines2") //PERFORM LOGIC HERE LIKE STREAMING REGRESSION }) ssc.start()
• 16.
  © 2015 SILICONVALLEY DATA SCIENCE LLC. ALL RIGHTS RESERVED. @SVDataScience Key Takeaways • If you work with large data or compute intensive advanced analytics and want a platform built from the ground up to run faster on distributed servers then - try out Spark • If you would like to have more control over your code than just an added macro language - try out Spark • If you want to better leverage data stored in Hadoop - try out Spark • If you prefer the open source licensing model over a subscriptions model - try out Spark
• 17.
  @SVDataScience Richard Williamson richard@svds.com @superhadooper Yes, we’rehiring! info@svds.com