Download as PDF, PPTX
![© 2017 Dremio Corporation, Two Sigma Investments, LP Columnar data persons = [{ nam e:’Joe', age:18, phones:[ ‘555-111-1111’, ‘555-222-2222’ ] },{ nam e:’Jack', age:37, phones:[‘555-333-3333’] }]](https://image.slidesharecdn.com/acfrogbgbmxapx82mxuhzjc-sl8fulqgdezkqz0uwtwijgvvcpzty9bqr-iiwmxex3rgws4wof4bamlcek6lujemvgaw1ufkmllh-180504154726/75/Improving-Python-and-Spark-Performance-and-Interoperability-with-Apache-Arrow-20-2048.jpg){kind=icon}
![© 2017 Dremio Corporation, Two Sigma Investments, LP Record Batch Construction Schema Negotiation Schema Negotiation Dictionary Batch Dictionary Batch Record Batch Record Batch Record Batch Record Batch Record Batch Record Batch name (offset)name (offset) name (data)name (data) age (data)age (data) phones (list offset)phones (list offset) phones (data)phones (data) data header (describes offsets into data)data header (describes offsets into data) name (bitmap)name (bitmap) age (bitmap)age (bitmap) phones (bitmap)phones (bitmap) phones (offset)phones (offset) { nam e:’Joe', age:18, phones:[ ‘555-111-1111’, ‘555-222-2222’ ] } Each box (vector) is contiguous memory The entire record batch is contiguous on wire Each box (vector) is contiguous memory The entire record batch is contiguous on wire](https://image.slidesharecdn.com/acfrogbgbmxapx82mxuhzjc-sl8fulqgdezkqz0uwtwijgvvcpzty9bqr-iiwmxex3rgws4wof4bamlcek6lujemvgaw1ufkmllh-180504154726/75/Improving-Python-and-Spark-Performance-and-Interoperability-with-Apache-Arrow-21-2048.jpg){kind=icon}
Uploaded byTwo Sigma
Improving Python and Spark Performance and Interoperability with Apache Arrow
The document discusses improving Python and Spark performance through Apache Arrow, highlighting the limitations of PySpark user-defined functions (UDFs) and their performance issues. It introduces Apache Arrow as a solution that streamlines memory formats for better interoperability and efficiency in data processing. The future roadmap includes enhancements for grouping functions and further integration with Spark and Arrow to optimize UDFs.
Improving Python and Spark Performance and Interoperability with Apache Arrow
- 1. Improving Python andSpark Performance and Interoperability with Apache Arrow Julien Le Dem Principal Architect Dremio Li Jin Software Engineer Two Sigma Investments
- 2. © 2017 Dremio Corporation, Two Sigma Investments, LP About Us • Architect at @DremioHQ • Formerly Tech Lead at Twitter on Data Platforms •Creator of Parquet • Apache member • Apache PMCs: Arrow, Kudu, Incubator, Pig, Parquet Julien Le Dem @J_ Li Jin @icexelloss • Software Engineer at Two Sigma Investments • Building a pythonbased analytics platform with PySpark • Other open source projects: – Flint: A Time Series Library on Spark – Cook: A Fair Share Scheduler on Mesos
- 3. © 2017 Dremio Corporation, Two Sigma Investments, LP Agenda • Current state and limitations of PySpark UDFs • Apache Arrow overview •Improvements realized • Future roadmap
- 4.
- 5. © 2017 Dremio Corporation, Two Sigma Investments, LP Why do we need User Defined Functions? • Some computation is more easily expressed with Python than Spark builtin functions. • Examples: –weighted mean – weighted correlation – exponential moving average
- 6. © 2017 Dremio Corporation, Two Sigma Investments, LP What is PySpark UDF • PySpark UDF is a user defined function executed in Python runtime. • Two types: –Row UDF: • lambda x: x + 1 • lambda date1, date2: (date1 - date2).years – Group UDF (subject of this presentation): • lambda values: np.mean(np.array(values))
- 7. © 2017 Dremio Corporation, Two Sigma Investments, LP Row UDF • Operates on a row by row basis – Similar to `map` operator •Example … df.withColumn( ‘v2’, udf(lambda x: x+1, DoubleType())(df.v1) ) • Performance: – 60x slower than buildin functions for simple case
- 8. © 2017 Dremio Corporation, Two Sigma Investments, LP Group UDF • UDF that operates on more than one row – Similar to `groupBy` followed by `map` operator •Example: – Compute weighted mean by month
- 9. © 2017 Dremio Corporation, Two Sigma Investments, LP Group UDF • Not supported out of box: – Need boiler plate code to pack/unpack multiple rows into a nested row •Poor performance – Groups are materialized and then converted to Python data structures
- 10.
- 11.
- 12.
- 13.
- 14.
- 15. © 2017 Dremio Corporation, Two Sigma Investments, LP Problems • Packing / unpacking nested rows • Inefficient data movement (Serialization / Deserialization) •Scalar computation model: object boxing and interpreter overhead
- 16.
- 17. © 2017 Dremio Corporation, Two Sigma Investments, LP Arrow: An open source standard • Common need for in memory columnar • Building on the success of Parquet. •Toplevel Apache project • Standard from the start – Developers from 13+ major open source projects involved • Benefits: – Share the effort – Create an ecosystem Calcite Cassandra Deeplearning4 j Drill Hadoop HBase Ibis Impala Kudu Pandas Parquet Phoenix Spark Storm R
- 18. © 2017 Dremio Corporation, Two Sigma Investments, LP Arrow goals • Welldocumented and cross language compatible • Designed to take advantage of modern CPU •Embeddable In execution engines, storage layers, etc. • Interoperable
- 19. © 2017 Dremio Corporation, Two Sigma Investments, LP High Performance Sharing & Interchange Before With Arrow •Each system has its own internal memory format • 70-80% CPU wasted on serialization and deserialization • Functionality duplication and unnecessary conversions • All systems utilize the same memory format • No overhead for cross-system communication • Projects can share functionality (eg: Parquet-to-Arrow reader)
- 20. © 2017 Dremio Corporation, Two Sigma Investments, LP Columnar data persons = [{ name:’Joe', age:18, phones:[ ‘555-111-1111’, ‘555-222-2222’ ] },{ nam e:’Jack', age:37, phones:[‘555-333-3333’] }]
- 21.
- 22. © 2017 Dremio Corporation, Two Sigma Investments, LP In memory columnar format for speed • Maximize CPU throughput Pipelining SIMD cache locality • Scatter/gather I/O
- 23.
- 24.
- 25.
- 26. © 2017 Dremio Corporation, Two Sigma Investments, LP How PySpark UDF works Execut or Python Worker UDF: scalar ->scalar Batched Rows Batched Rows
- 27.
- 28. © 2017 Dremio Corporation, Two Sigma Investments, LP Profile lambda x: x+1 Actual Runtimeis 2s without profiling. 8 Mb/s 91.8%
- 29.
- 30. © 2017 Dremio Corporation, Two Sigma Investments, LP Why pandas.DataFrame • Fast, featurerich, widely used by Python users • Already exists in PySpark (toPandas) •Compatible with popular Python libraries: NumPy, StatsModels, SciPy, scikitlearn… • Zero copy to/from Arrow
- 31.
- 32.
- 33.
- 34.
- 35. © 2017 Dremio Corporation, Two Sigma Investments, LP Support Group UDF • Splitapplycombine: Break a problem into smaller pieces Operate on each piece independently Put all pieces back together • Common pattern supported in SQL, Spark, Pandas, R …
- 36.
- 37. © 2017 Dremio Corporation, Two Sigma Investments, LP SplitApplyCombine (with Group UDF) • Split: groupBy, window, … • Apply: UDF •Combine: Inherently done by Spark
- 38. © 2017 Dremio Corporation, Two Sigma Investments, LP Introduce groupBy().apply() • UDF: pd.DataFrame > pd.DataFrame – Treat each group as a pandas DataFrame –Apply UDF on each group – Assemble as PySpark DataFrame
- 39.
- 40.
- 41.
- 42. © 2017 Dremio Corporation, Two Sigma Investments, LP Limitations • Requires Spark Row <> Arrow RecordBatch conversion – Incompatible memory layout (row vs column) •(groupBy) No local aggregation – Difficult due to how PySpark works. See https://issues.apache.org/jira/browse/SPARK10915
- 43.
- 44. © 2017 Dremio Corporation, Two Sigma Investments, LP What’s Next (Arrow) • Arrow RPC/REST • Arrow IPC •Apache {Spark, Drill, Kudu} to Arrow Integration – Faster UDFs, Storage interfaces
- 45.
- 46.
- 47. © 2017 Dremio Corporation, Two Sigma Investments, LP Get Involved • Watch SPARK20396 • Join the Arrow community –dev@arrow.apache.org – Slack: • https://apachearrowslackin.herokuapp.com/ – http://arrow.apache.org – Follow @ApacheArrow
- 48. © 2017 Dremio Corporation, Two Sigma Investments, LP Thank you • Bryan Cutler (IBM), Wes McKinney (Two Sigma Investments) for helping build this feature • Apache Arrow community •Spark Summit organizers • Two Sigma and Dremio for supporting this work
- 49. This document isbeing distributed for informational and educational purposes only and is not an offer to sell or the solicitation of an offer to buy any securities or other instruments. The information contained herein is not intended to provide, and should not be relied upon for investment advice. The views expressed herein are not necessarily the views of Two Sigma Investments, LP or any of its affiliates (collectively, “Two Sigma”). Such views reflect significant assumptions and subjective of the author(s) of the document and are subject to change without notice. The document may employ data derived from third-party sources. No representation is made as to the accuracy of such information and the use of such information in no way implies an endorsement of the source of such information or its validity. The copyrights and/or trademarks in some of the images, logos or other material used herein may be owned by entities other than Two Sigma. If so, such copyrights and/or trademarks are most likely owned by the entity that created the material and are used purely for identification and comment as fair use under international copyright and/or trademark laws. Use of such image, copyright or trademark does not imply any association with such organization (or endorsement of such organization) by Two Sigma, nor vice versa.