Download as PDF, PPTX
Uploaded byFlink Forward
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
This document discusses Apache Zeppelin and Apache Flink integration. It describes how the Flink interpreter allows users to run Flink jobs within Zeppelin notebooks, accessing features like dynamic forms, angular displays, and progress monitoring. The roadmap includes improving multi-tenancy with authentication and containers, and developing Helium as a platform for packaging and distributing analytics applications on Zeppelin.
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
- 1. Data science lifecyclewith Apache Flink and Apache Zeppelin (incubating) Flink Forward Moon moon@nflabs.com NFLabs www.nflabs.com
- 2. Content 1. Data sciencelifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 3.
- 4.
- 5.
- 6. Data Science: people EngineerData Scientist DevOps Business http://aarondavis.design/
- 7. Content 1. Data sciencelifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 8. Zeppelin for datascientist
- 9. ProjectTimeline ASF Incubation12.2014 08.2014 Startedgetting adoption http://zeppelin.incubator.apache.org 12.2012 Commercial Product for data analysis 10.2013 Open sourced a single feature
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19. Current Status 1 Release 68Contributors worldwide 722 Stars on GH 300/900 Emails at users/dev @i.a.o
- 20.
- 21.
- 22.
- 23. Zeppelin & Friends Z-Manager ZeppelinHub …⋯ Collaboration/Sharing Packaging& Deployment Zeppelin + Full stack on a cloud Packages Backend Integration
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30. Content 1. Data sciencelifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 31. Flink integration Integrated throughInterpreter Data processing system abstraction in Zeppelin
- 32.
- 33. Writing an Interpreter publicabstract void open(); public abstract void close(); public abstract InterpreterResult interpret(String st, InterpreterContext context); public abstract void cancel(InterpreterContext context); public abstract int getProgress(InterpreterContext context); public abstract List<String> completion(String buf, int cursor); public abstract FormType getFormType(); public Scheduler getScheduler(); Must have Good to have Advanced
- 34. Flink Interpreter https://github.com/apache/incubator-zeppelin/blob/master/flink/src/main/java/org/apache/zeppelin/flink/FlinkInterpreter.java Zeppelin Server Thrift FlinkInterpreter Interpreter JVM process FlinkILoop ExecutionEnvironment
- 35.
- 36. Using interpreter Use different interpretersin the same notebook
- 37. Display System Zeppelin Server FlinkInterpreter Other Interpreter Zeppelin webapp Websocket, REST Text Html Table Angular
- 38. Display System Select display systemthrough output
- 39. Built in scheduler Built-inscheduler runs your notebook with cron expression.
- 40.
- 41.
- 42. Content 1. Data sciencelifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 43. Flink Integration • ZeppelinContext:Access to Zeppelin provided features • - Dynamic form • - Angular display system • Dependency loading • Auto completion • Cancel • Get progress information
- 44. Thank you Q &A Moon moon@nflabs.com NFLabs www.nflabs.com http://zeppelin.incubator.apache.org/
- 45.
- 46. Multi-tenancy Two approaches 1. Implementauthentication,ACL inside of Zeppelin https://github.com/apache/incubator-zeppelin/pull/53 2. Run Zeppelin on top of Docker http://github.com/NFLabs/z-manager
- 47.
- 48. An Engineer engineer byhttp://aarondavis.design/
- 49.
- 50. An Organization engineer byhttp://aarondavis.design/
- 51. That’s too many! engineerby http://aarondavis.design/
- 52. What is theproblem? Too much: Install Configure Cluster resources
- 53.
- 55. Z Manager PoC httpd+ mod_php nginx Linux box engineer by http://aarondavis.design/ 2 days, bash + php :(
- 56.
- 57. Z Manager http://github.com/NFLabs/z-manager Apache 2.0Licence Containerized deployment per user Reverse proxy Single binary Simple web application Z Manager SGA to ASF coming *
- 58. Z Manager Auto-update engineer byhttp://aarondavis.design/ Linux box go + react :) Z Manager process
- 59.
- 60.
- 61. People do thesimilar work with different data New visualization Model & Algorithm Data process pipeline engineer by http://aarondavis.design/
- 62. Package and distributework New visualization Model & Algorithm Data process pipeline Pkg Repo engineer by http://aarondavis.design/
- 63. Helium https://s.apache.org/helium Platform for on topof Apache Zeppelin Data Analytics Application
- 64. Helium Application = + ViewAlgorithm Zeppelin provided Resources
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.
- 96.
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107.
- 108.
- 109.
- 110.
- 111.
- 112.
- 113.
- 114.
- 115.
- 116.
- 117.
- 118.
- 119.
- 120.
- 121.
- 122.
- 123.
- 124.
- 125.
- 126.
- 127.
- 128.
- 129.
- 130.
- 131.
- 132.
- 133.
- 134.
- 135.
- 136.
- 137.
- 138.
- 139.
- 140.
- 141.
- 142.
- 143.
- 144.
- 145.
- 146.
- 147.
- 148.
- 149.
- 150.
- 151.
- 152.
- 153.
- 154.
- 155.
- 156.
- 157.
- 158.
- 159.
- 160.
- 161.
- 162.
- 163.
- 164.
- 165.
- 166.
- 167.
- 168.
- 169.
- 170.
- 171. API class YourApplication extendsorg.apache.zeppelin.helium.Application { @Override public void run(ApplicationArgument arg, InterpreterContext context) { ….. } } Easy
- 172.
- 173.
- 174.
- 175. Application Spec { mavenArtifact :groupId:artifactId:version, className : your.helium.application.Class, icon : fa fa-cloud, name : My app name, description : “some description, consume : [ org.apache.spark.SparkContext ] } Simple Writing
- 176.
- 177.
- 178.
- 179.
- 180.
- 181.
- 182.
- 183.
- 184.
- 185.
- 186.
- 187.
- 188.
- 189.
- 190.
- 191.
- 192.
- 193.
- 194.
- 195.
- 196.
- 197.
- 198.
- 199.
- 200.
- 201. Thank you Q A Moon moon@nflabs.com NFLabs www.nflabs.com http://zeppelin.incubator.apache.org/