Download to read offline
![/linkedin/scalding/status • Started >1 year ago • Thousands of production LOC written in Scalding by our team – Pretty happy with readability, maintainability and tooling support • Dozens of flows are currently in production, and counting • Created Scalding user group • Growing interest • Learning: – Scala[Scalding] < Scala[ _ ]](https://image.slidesharecdn.com/scalabythebay2-140809090501-phpapp02/75/Using-Scalding-for-Data-Driven-Product-Development-at-LinkedIn-20-2048.jpg)
Uploaded bySasha Ovsankin
Using Scalding for Data Driven Product Development at LinkedIn
This document discusses using Scalding, which combines Scala and Hadoop, for data-driven product development. Scalding provides a domain-specific language for writing MapReduce jobs in Scala that allows processing large datasets in Hadoop. The document describes how LinkedIn uses Scalding for various tasks including processing web and application data at large scales. It highlights benefits like succinct code, abstraction, and running thousands of Scalding jobs successfully in LinkedIn's production environment.
Using Scalding for Data Driven Product Development at LinkedIn
- 1. Using Scalding forData-Driven Product Development Sasha Ovsankin LinkedIn Presented to Scala By The Bay Aug 9, 2014
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11. “Online” World /data-driven/linkedin Web Applications NoSQLData Stores “Offline” World (Hadoop) HDFS Hadoop Jobs Tracking/l ogging Analytics Data Products Messaging Message delivery Databases
- 12. /linkedin/big-data/links • “LinkedIn BigData Ecosystem” – http://lnkd.in/big-data-ecosystem • Grid Operations – http://lnkd.in/gridops2013
- 13. /scalding http://github.com/twitter/scalding • Scala-based DSLfor Map/Reduce jobs • Built on Cascading, stable and mature Hadoop framework • Uses API similar to Scala collections: class WordCountJob(args : Args) extends Job(args) { TextLine( args("input") ) .flatMap('line -> 'word) { line : String => line.split("""s+""") } .groupBy('word) { _.size } .write( Tsv( args("output") ) ) } • Succinct and powerful • High level of abstraction
- 14. /data-driven/problem/scaling • Problem: Scaling •Solution – Distributed processing – High-level description of algorithms – Functional programming
- 15.
- 16. ../problem/complexity • Problem: Complexity •Solution – Consistent way of organizing data • Self-describing data formats (Avro) • File organization – Type safety – Modularization
- 17.
- 18. /linkedin/hadoop/practices • All onlinedata end up in HDFS – Avro encoding is standard • Production Process – CI/Automatic Build • More info forthcoming – Production Review – Operations and Monitoring • More info at http://lnkd.in/gridops2013 • Result: Thousands of jobs running in production • More info at http://lnkd.in/big-data-ecosystem
- 19. ../solution/scala/killer-argument • Map &reduce -- primitives scala> (1 to 1000) map { pow(_,2) } reduce { _ + _ } res20: Int = 333833500
- 20. /linkedin/scalding/status • Started >1year ago • Thousands of production LOC written in Scalding by our team – Pretty happy with readability, maintainability and tooling support • Dozens of flows are currently in production, and counting • Created Scalding user group • Growing interest • Learning: – Scala[Scalding] < Scala[ _ ]
- 21.
- 22. /linkedin/join-us • Work onunique and interesting problems • Be part of great engineering community • Use latest tools and technologies • Help connect the world’s professionals to help them become more productive and successful • We are looking for amazing people interested in Software Engineering and Data Science – http://linkedin.com/careers Questions?