Downloaded 66 times
![Questions? [email_address] @derekg http://open.nytimes.com/](https://image.slidesharecdn.com/hw09-countingandclusteringandotherdatatricks-091025150759-phpapp01/75/Hw09-Counting-And-Clustering-And-Other-Data-Tricks-28-2048.jpg)
Uploaded byCloudera, Inc.
Hw09 Counting And Clustering And Other Data Tricks
This document summarizes the early use of Hadoop at The New York Times to generate PDFs of archived newspaper articles and analyze web traffic data. It describes how over 100 Amazon EC2 instances were used with Hadoop to pre-generate over 11 million PDFs from 4.3TB of source data in under 24 hours for a total cost of $240. The document then discusses how the Times began using Hadoop to perform web analytics, counting page views and unique users, and merging this data with demographic and article metadata to better understand user traffic and behavior.
Hw09 Counting And Clustering And Other Data Tricks
- 1. Cheap Parlor Tricks, Counting, and Clustering Derek Gottfrid The New York Times October 2009
- 2. Evolution of Hadoop@ NYTimes.com
- 3. Early Days -2007 Solution looking for a problem
- 4. Solution Wouldn’t itbe cool to use lots of EC2 instances (it’s cheap; nobody will notice) Wouldn’t it be cool to use Hadoop (MapReduce Google style is awesome)
- 5. Found a ProblemFreeing up historical archives of NYTimes.com 1851-1922
- 6. Problem Bits Articlesare served as PDFs Really need PDFs from 1851-1981 PDFs are dynamically generated Free = more traffic Real deadline
- 7. Background What goesinto making a PDF of a NYTimes.com article? Each article is made up of many different pieces - multiple columns, different sized headings, multiple pages, photos.
- 8. Simple Answer Pre-generateall 11 million PDFs and serve them statically.
- 9. Solution Copy allthe source data to S3 Use a bunch of EC2 instances and some Hadoop code to generate all the PDFs Store the output PDFs in S3 Serve the PDFs out of S3 w/ a signed query string
- 10. A Few DetailsLimited HDFS - everything loaded in and out of S3 Reduce = 0 - only used for some stats and error reporting
- 11. Breakdown 4.3 TBof source data into S3 11M PDFS - 1.5 TB output $240 for EC2 - 24hrs x 100 machines
- 12.
- 13. Currently - 2009All that darn data - Web Analytics
- 14. Data Registration /Demographic Articles 1851 - today Usage Data / Web Logs
- 15. Counting Classic cookietracking - let’s add it up Total PV Total unique users PV per user
- 16. A Few DetailsUsing EC2 - 20 Machines Hadoop 0.20.0 12+TB of data Straight MR in Java
- 17. Usage Data July 2009 ???M Page Views ??M Unique Users
- 18. Merging Data Usagedata combined with demographic data.
- 19. Twitter Click BacksBy Age Group July 2009
- 20. Merging Data Usagedata with article meta data
- 21. Usage Data combinedwith Article Data July 2009 40 Articles
- 22. Usage Data combinedwith Article Data July 2009 40 Articles
- 23.
- 24. Clustering Moving beyondsimple counting and joining Join usage data, demographic information, and article meta data Apply simple k-means clustering
- 25.
- 26.
- 27. Conclusion Large scalecomputing is transformative for NYTimes.com.
- 28. Questions? [email_address] @derekghttp://open.nytimes.com/
Editor's Notes
- #3 The ability to compute across large data is a key to success. Accumulating data is easy. Computing is hard Personify, Omniture, Webtrends - specialized not general computing tools.
- #14 Fixed reports are great and we can do that but the really interesting part is asking questions after the fact in an adhoc exploratory manner.
- #15 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #16 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #17 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #18 Distribution of page views per users for the July 2009. Most users view 1 Page. Not new but shows we have some mastery over the data.This is based of just user data - 380G of compressed data covering over 700million page views.
- #19 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #21 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #22 mapping, collbrative filtering, segementation,
- #25 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #26 We have always had data. Collecting data is not a issue for us. We are okay at that part.
- #27 We have always had data. Collecting data is not a issue for us. We are okay at that part.