Downloaded 154 times
Uploaded byDatabricks
Apache Spark Data Validation
The document presents a talk by Patrick Pisciuneri and Doug Balog from Target on a data validation tool for Apache Spark, highlighting its features, configuration options, and motivations for implementation. It emphasizes the tool's ability to detect errors early in data pipelines, promote best practices, and its ease of adoption within various data management systems. The talk also discusses the tool's open-source nature, future enhancements, and solicits community input for further development.
Apache Spark Data Validation
- 1. WIFI SSID: SparkAISummit| Password: UnifiedAnalytics
- 2. Patrick Pisciuneri &Doug Balog, Target Apache Spark Data Validation #UnifiedAnalytics #SparkAISummit
- 3. Outline • Introductions • Motivation •Data Validator • Open Source • Future Work • Conclusions 3#UnifiedAnalytics #SparkAISummit
- 4. Introduction Patrick Pisciuneri Patrick isa Data Scientist for Target. He received his PhD in Mechanical Engineering from the University of Pittsburgh in 2013. His research involved the intersection of high- performance computing and the simulation of turbulent reacting flows. In 2015 he joined Target as a data scientist where he has worked on product and ad recommendations. Doug is a Data Engineer for Target. He develops Machine Learning infrastructure for Target in Pittsburgh, PA. He joined Target in 2014 and is currently a Principal Data Engineer. He has a BS in Computer Science from University of Pittsburgh. 4#UnifiedAnalytics #SparkAISummit Doug Balog
- 5. Introduction Target • 1,845 storesin the United States • 39 distribution centers in the United States • 350,000+ team members worldwide • Online business at target.com • Global offices in China, Hong Kong and India corporate.target.com/ 5#UnifiedAnalytics #SparkAISummit
- 6. Motivation • Understand data •Catch errors/anomalies early in pipelines • Promote best practices for data management • Easy to adopt – language agnostic interface • Efficient on large datasets and distributed systems 6#UnifiedAnalytics #SparkAISummit
- 7. Motivation References 1. TFX: ATensorFlow-Based Production-Scale Machine Learning Platform, KDD 2017 2. Hidden Technical Debt in Machine Learning, NIPS 2015 3. Methodology for Data Validation 1.0, Eurostat – CROS, 2016 4. Extending Apache Spark APIs Without Going Near Spark Source or a Compiler, Spark + AI Summit, 2018 7#UnifiedAnalytics #SparkAISummit
- 8.
- 9. Features • Configuration • Validators •Reporting & Notification • Profiling 9#UnifiedAnalytics #SparkAISummit
- 10.
- 11. Configuration • User createsa configuration file (yaml) 1. Variables 2. Reporting settings 3. Notification settings 4. Validation checks 11#UnifiedAnalytics #SparkAISummit
- 12. Configuration - Variables •Set the value of a variable specified by the name value 1. Simple substitution 2. Environment variable substitution 3. Shell substitution • First line of stdout 4. SQL substitution • First column of first row 12#UnifiedAnalytics #SparkAISummit
- 13. Configuration – ReportSettings • Event report – Specify path and filename • local:// • hdfs:// • append – will overwrite existing file if false – Pipe event report to another program • ignoreError – mark data validator as failed if pipe command fails 13#UnifiedAnalytics #SparkAISummit
- 14. Configuration – NotificationSettings • Send email with status of validation checks 14#UnifiedAnalytics #SparkAISummit
- 15. Configuration – ValidationChecks • Specify the checks we want to run on a data asset • Support: – Hive tables – Orc files – Parquet files • Any number of checks on any number of columns • Specify condition to filter asset – Useful when applied to partition column 15#UnifiedAnalytics #SparkAISummit
- 16.
- 17.
- 18.
- 19. Example • Census (1994)Income Data Set • Learning task: predict whether income exceeds 50k http://archive.ics.uci.edu/ml/datasets/Census+Income 19#UnifiedAnalytics #SparkAISummit
- 20.
- 21. Explain Config 21#UnifiedAnalytics #SparkAISummit * formattedin text editor for clarity
- 22. Reporting • Event log –Logs configuration settings – Logs variable substitutions – Logs runtime environment and statistics – Logs checks run and status – It’s extensive! 22#UnifiedAnalytics #SparkAISummit
- 23.
- 24.
- 25.
- 26.
- 27.
- 28. Profiling Implementation 28#UnifiedAnalytics #SparkAISummit •Catalyst aggregate expressions • Min • Max • Average • Count • Stddev • UDAF • histogram
- 29. Profiling • Parse /pipe output from report to tool for visualization • Facets Overview 29#UnifiedAnalytics #SparkAISummit
- 30. Open Source • ApacheLicense, Version 2.0 • https://github.com/target/data-validator 30#UnifiedAnalytics #SparkAISummit
- 31. Future Work • Addmore validators • Leverage column statistics for default configuration generation • HDP 3 and Spark 2.4+ • Anomaly detection • Distribution shift • Ideas and input from community 31#UnifiedAnalytics #SparkAISummit
- 32. Conclusions • Adoption • Understanddata • Catch errors/anomalies early in pipelines • Promote best practices for data management • Easy to adopt – language agnostic interface • Efficient on large datasets and distributed systems 32#UnifiedAnalytics #SparkAISummit
- 33. Thank you! • Questions? •We are hiring: – jobs.target.com – Search “lead data scientist”, “lead data engineer” • Check out: Parallelizing with Apache Spark in Unexpected Ways @ 2:40 PM by Anna Holschuh 33#UnifiedAnalytics #SparkAISummit
- 34. DON’T FORGET TORATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT