Download as PDF, PPTX
![BigDebug Project Overview Titian: Data Provenance for Fine- Grained Tracing [PVLDB 2016] Vega: Incremental Computation for Interactive Debugging [SoCC 2016] Collaboration with Tyson Condie, Miryung Kim, and Todd Millstein BigDebug: Debugging Primitives for Interactive Big Data Processing in Spark [ICSE 2016] Automated Debugging in Data Intensive Scalable Computing Systems [Under Submission]](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-3-2048.jpg)
![We apply data provenance and delta debugging [Zeller et al. ] in tandem Feature 5: Automated Fault Localization](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-18-2048.jpg)
![We apply data provenance and delta debugging [Zeller et al. ] in tandem Feature 5: Automated Fault Localization Test](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-19-2048.jpg)
![We apply data provenance and delta debugging [Zeller et al. ] in tandem Feature 5: Automated Fault Localization Test Split](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-20-2048.jpg)
![We apply data provenance and delta debugging [Zeller et al. ] in tandem Feature 5: Automated Fault Localization Test Split In average BigDebug is able to localize faults within 63% of the original job running time](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-21-2048.jpg)
![We apply data provenance and delta debugging [Zeller et al. ] in tandem Feature 5: Automated Fault Localization …. Test Split In average BigDebug is able to localize faults within 63% of the original job running time In average BigDebug is able to localize faults within 63% of the original job running time](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-22-2048.jpg)
![Running Example val log = "s3n://xcr:wJY@ws/logs/enroll.log" val text_file = sc.textFile(log) text_file .map{line=>(line.split()[2],line.split()[3])} .map{t => (t._1 , getYears(t._2))} .groupByKey() .map(v => (v._1 , average(v._2))) .collect() 1 Michael Sophomore 03/12/1996 2 Justin Freshman 05/01/1998 .. .. ..](https://image.slidesharecdn.com/021gulzarinterlandi-170613024818/75/Debugging-Big-Data-Analytics-in-Apache-Spark-with-BigDebug-with-Muhammad-Gulzar-and-Matteo-Interlandi-24-2048.jpg)
Uploaded byDatabricks
Debugging Big Data Analytics in Apache Spark with BigDebug with Muhammad Gulzar and Matteo Interlandi
The document discusses the BigDebug project, which aims to improve debugging support for big data analytics in Apache Spark due to the limitations of existing frameworks like Hadoop and Spark. It elaborates on various debugging features such as simulated breakpoints, guarded watchpoints, crash culprit identification, and backward tracing to enhance user interaction during the debugging process. The average performance of BigDebug is noted to localize faults within 63% of the original job's runtime.
Debugging Big Data Analytics in Apache Spark with BigDebug with Muhammad Gulzar and Matteo Interlandi
- 1.
- 2. Open Source Data-Intensive Scalable Computing (DISC) Platforms: Hadoop MapReduce and Spark ◦ Severe lack of debugging support in these systems ◦Programs (i.e., queries, jobs) are batch executed / black boxes So what to do? ◦ Trial and error debugging on subsample ◦ Post-mortem analysis of error logs ◦ Analyze physical view of the execution (a job id, failed node, etc). Debugging Cloud Computing Programs
- 3.
- 4.
- 5.
- 6.
- 7. Simulated Breakpoint1 Guarded Watchpoint2 Crash Culprit Identification3 Backward Tracing4 BigDebug: Interactive Debugger Features $>Crash inducing inputrecords: 9K23 Cruz TX 1440023645 2FSD Cruz KS 1440026456 9909 Cruz KS 1440023768
- 8.
- 9. Feature 1: Simulated Breakpoint Stage 0 Stage1 map simulated breakpoint groupByKey map map
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16. Feature 5: Automated Fault Localization Goal: Given a test function and set of failing results ◦ Identify the minimum set of input records that can reproduce the failure Afterall is said and done more is said than done than done done more is said After all is said and After all is said and done more is said than done (After,1) (all,1) (is,1) (said,1) (and,1) (done,1) (more,1) (is,1) (said,1) (than,1) (done,1) (After,1) (all,1) (is,2) (and,1) (more,1) (said,2) (than,1) (done,2) After,1 all, 1 is, 2 and, 1 more, 1 said, 2 done, 2 than, 1 FlatMap Map ReduceByKey CollectTextFile Task 1 Task 3 Task 1 Task 3 Task 2Task 2 Stage 2Stage 1
- 17. Feature 5: Automated Fault Localization Goal: Given a test function and set of failing results ◦ Identify the minimum set of input records that can reproduce the failure ◦We apply data provenance and delta debugging in tandem After all is said and done more is said than done than done done more is said After all is said and After all is said and done more is said than done (After,1) (all,1) (is,1) (said,1) (and,1) (done,1) (more,1) (is,1) (said,1) (than,1) (done,1) (After,1) (all,1) (is,2) (and,1) (more,1) (said,2) (than,1) (done,2) After,1 all, 1 is, 2 and, 1 more, 1 said, 2 done, 2 than, 1 FlatMap Map ReduceByKey CollectTextFile Task 1 Task 3 Task 1 Task 3 Task 2Task 2 Stage 2Stage 1
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24. Running Example val log= "s3n://xcr:wJY@ws/logs/enroll.log" val text_file = sc.textFile(log) text_file .map{line=>(line.split()[2],line.split()[3])} .map{t => (t._1 , getYears(t._2))} .groupByKey() .map(v => (v._1 , average(v._2))) .collect() 1 Michael Sophomore 03/12/1996 2 Justin Freshman 05/01/1998 .. .. ..