Download as PDF, PPTX
Towards Apache Flink 2.0 - Unified Data Processing and Beyond, Bowen Li
- 1. Towards Apache Flink2.0 - Unified Data Processing and Beyond Bowen Li Committer@Apache Flink, Senior Engineer@Alibaba Nov 14, 2019 @Scale by the bay
- 2. Agenda ● Flink usecases ● Flink in a nutshell - what makes Flink successful in stream processing ● Beyond stream processing ○ Batch ○ Hive and Data warehousing ○ AI/Machine Learning ○ Serverless
- 4. Flink at Alibaba ●Powers real time computations of all business units at Alibaba group ● Powers all search and recommendations, both online and offline ● Provided as cloud service to public on Ali Cloud
- 5. Single’s Day GlobalShopping Festival on 11/11
- 6. Single’s Day Stats- Nov 11, 2019 Alibaba ○ GMV ■ $14 million in the first 21s ■ $1.4 billion in the first 96s ■ $38 billion in 24h ○ 982 PB data generated in total ○ 544,000 transactions/sec at peak Flink @ Alibaba ○ 2 billion events/sec, 3 TB/sec - up 111% from 2018
- 7. Flink at Alibaba UseCase 1: online ML
- 8. ○ hundreds ofmillions events ○ 100+ billions features ○ e2e second latency ○ real-time training, feature and module update Flink at Alibaba Use Case 1: online ML
- 9. Flink at Alibaba UseCase 2: Real Time GMV dashboard
- 10. Flink at Alibaba UseCase 2: Real Time GMV dashboard
- 11. Flink at Scale RealTime AI / ML Real Time AnalyticsReal Time Fraud Detection/Risk Management Real Time Dynamic Pricing
- 12. Flink at Scale RealTime Compute Service on public cloud backed by Flink Kinesis Analytics
- 13. Flink in aNutshell - key differentiaters from other open source solutions ancient squirrel from Ice Age!
- 15. Flink in aNutshell Stateful Computations …...
- 16. Stateful Computations …... Why(built-in) state matters? ● computation with context, rather than single record ● no network IO, lower latency ● no external dependency, full control by framework for exactly-once semantics ● ...
- 17. Stateful Computations …... Flinkprovides built-in state backends that support rich, arbitrary data structure in a fully fault tolerant way ○ In-memory, splliable backend ○ RocksDB backend
- 18. Stateful Computations overEvent Streams…... Flink in a Nutshell
- 19. Stateful Computations overEvent Streams…... It means a few things… 1. All your data is data streams! ○ batch v.s. streaming - just execution models ○ bounded v.s. unbounded data streams - key difference ○ technically all data processing is stream processing
- 20. Stateful Computations overEvent Streams…... 2. Streaming-first, pipelined execution ○ record flowing thru the system -> extremely high throughput, ultra low latency ○ fondamentally different from batch-first, staged-execution framework ○ can’t achieved by mini-batch workaround
- 21. Stateful Computations overEvent Streams…... 3. Event comes with time! ○ event time v.s. processing time ○ windows aggregation, sessionazation, time-base joins ○ out-of-order and late data Flink support all the most comprehensive time semantics natively from the beginning
- 22. Stateful Computations overEvent Streams in an Expressive …... Flink in a Nutshell
- 23. Stateful Computations overEvent Streams in an Expressive …... Layered APIs with the most comprehensive semantics
- 24. Stateful Computations overEvent Streams in an Expressive, Scalable …... Flink in a Nutshell
- 25. Stateful Computations overEvent Streams in an Expressive, Scalable …... ● Horizontally scalable ● Battle tested ○ trillions of records per day ○ terabytes of state ○ run on thousands of cores
- 26. Stateful Computations overEvent Streams in an Expressive, Scalable, Operational-focused …... Flink in a Nutshell
- 27. Stateful Computations overEvent Streams in an Expressive, Scalable, Operational-focused …... ● Deploy anywhere ○ kubernetes, yarn, mesos, standalone ● Deploy flexibly ○ per-job mode, session mode ● Highly available ○ with HA setup
- 28. Stateful Computations overEvent Streams in an Expressive, Scalable, Operational-focused, Exactly-Once and Fault Tolerant way Flink in a Nutshell
- 29. Stateful Computations overEvent Streams in an Expressive, Scalable, Operational-focused, Exactly-Once and Fault Tolerant way ● Exactly-once state consistency ● End-to-end exactly-once consistency with connectors support transactions ● Checkpoint/Savepoint ○ on-the-fly, don’t scrafice performance
- 30. Stateful Computations overEvent Streams in an Expressive, Scalable, Operational-focused, Exactly-Once and Fault Tolerant way
- 31. Going Beyond StreamProcessing ● Batch -> Unified Data Processing ● Data Warehousing and Notebook ● Machine Learning / AI / DL ● Serverless
- 32. Recap the lambdaarchitecture ...... ○ infra: high operation cost ○ dev: costly maintenance, and hard to learn 2+ stack ○ business: hard to sync to guarantee logic consistency Why Unified Streaming and Batch Data Processing? MQ / Pub-Sub HDFS / S3 Stream Processing (online) Batch Processing (offline) Combine Results (serving)
- 33. Why Flink? Flink’s philosophy: streamingfirst, with batch is a special case of streaming
- 34. State-of-the-Art Batch Processingon a Stream Processor <= Flink 1.8 from Flink 1.9
- 35. * Blink isan internal fork of Flink at Alibaba. Blink has become a part of Flink from release 1.9 Performance of Blink versus Spark in the TPC-DS benchmark, aggregate time for all queries together. Presentation by Xiaowei Jiang at Flink Forward Beijing, 2018.
- 36. Data Warehousing Initial integrationin Flink 1.9 for Hive 2.3.4 and 1.2.1 Full integration (read, write, udf) in Flink 1.10 for all Hive 1.x, 2.x, and 3.x
- 37.
- 38. Machine Learning &AI & DL
- 39. Machine Learning &AI Recap the “lambda” architecture, again, for ML MQ / Pub-Sub HDFS / S3 Online Training Offline Training Model Validation Preprocessing Dynamic Model Static Model Static Model Preprocessing Inference
- 40. Flink is popularfor online ML now MQ / Pub-Sub HDFS / S3 Online Training Offline Training Model Validation Preprocessing Dynamic Model Static Model Static Model Preprocessing Inference
- 41. Streaming-Batch Unified ML UseFlink everywhere to reduce maintenance and operation cost MQ / Pub-Sub HDFS / S3 Online Training Offline Training Model Validation Preprocessing Dynamic Model Static Model Static Model Preprocessing Inference
- 42. Machine Learning ML Stage MLFlow Efforts & Requirements MQ / Pub-Sub HDFS / S3 Online Training Offline Training Model Validation Preprocessing Dynamic Model Static Model Static Model Preprocessing Inference Data Acquisition Preprocessing Model Training Model Validation & Serving Inference Rich Connectors Dataset Management Stream-Batch unification Strong API & SQL Support Enhanced Iteration Flink ML lib DL on Flink (TF, PyTorch) Model Serving Model Registry & Management Rollout/Rollback Online Evaluation Flink ML Pipeline Python API support
- 43. Machine Learning ML Stage MLFlow Efforts & Requirements MQ / Pub-Sub HDFS / S3 Online Training Offline Training Model Validation Preprocessing Dynamic Model Static Model Static Model Preprocessing Inference Data Acquisition Preprocessing Model Training Model Validation & Serving Inference Rich Connectors Dataset Management Stream-Batch unification Strong API & SQL Support Enhanced Iteration Flink ML lib DL on Flink (TF, PyTorch) Model Serving Model Registry & Management Rollout/Rollback Online Evaluation Flink ML Pipeline Python API support
- 44. Flink ML Libs ●Completely rewrote ● Based on ML pipeline, powered by Table API ● Battle tested algorithms ○ K-means ○ Naive Bayes ○ Linear regression ○ GBDT ○ Decision tree ○ PCA ○ Random forest ○ Correlation ○ ….
- 45. Flink ML Pipeline training inferenc e inputtable 1 ModelTransformerEstimatorTransformer input table 2 result tableModelTransformer Two type of operators ● Transformer: data -> data ● Estimator: data -> model Estimator Pipeline Model Pipeline Model Pipeline pipeline.fit(input1) pipeline.transform(input2)
- 46. Deep Learning Pipeline DataAcquisition Data Process & Tranformation Model Training Model Validation Model Serving Parameter Tuning
- 47. Deep Learning Pipeline source1 source 2 join udtf Flink Cluster External MQ / FS Tensorflow Cluster worker workerworker PS PS
- 48. Flink Deep LearningPipeline Data Acquisition Data Process & Tranformation Model Training Model Validation Model Serving Parameter Tuning
- 49. Flink + TensorflowIntegration source 1 source 2 join udtf Flink Cluster External MQ / FS Tensorflow Cluster worker workerworker PS PS source 1 source 2 join udtf A single Flink Cluster worker workerworker PS PS
- 50. DL on FlinkML Pipeline source 1 source 2 join udtf worker workerworker PS PS Transformer Estimator
- 51. Serverless Event-Driven, Function asa Service Benefits: ● elastic ● lightweight Challenges: ○ state ■ consistency ■ IO ■ capacity ○ hard to build complex applications
- 52. Event Driven StateManagement Composable isn’t that….
- 53. Event Driven StateManagement Composable isn’t that…. Stream Processing!
- 54. Check out projectState Function, announced in Oct 2019! https://statefun.io/ It officially became part of Apache Flink last week.
- 55.
- 56. Q & A Email:user/dev@flink.apache.org Twitter: @ApacheFlink @Bowen__Li