Download as PDF, PPTX
Uploaded byTill Rohrmann
Scaling stream data pipelines with Pravega and Apache Flink
The document discusses scaling stream data pipelines. It covers how streams from social networks, online shopping, server monitoring and IoT sensors are becoming more common. It discusses how workloads have daily, weekly and seasonal cycles that cause spikes. It then discusses how to scale event processing by adding more processors as the input rate increases. It introduces Pravega as a stream storage system that can store streams permanently while preserving ordering and scaling to varying workloads. Pravega uses segments that can be split or merged to scale a stream. It describes Pravega's auto-scaling policies and how it triggers scaling events based on throughput metrics. Finally, it discusses how reader groups allow Pravega to maintain ordering during scaling.
Scaling stream data pipelines with Pravega and Apache Flink
- 1. Scaling stream data pipelines Flavio Junqueira, Pravega- Dell EMC Till Rohrmann, Data Artisans
- 2.
- 3. Flink Forward - San Francisco, 2018 Social networks Online shopping Streams ahoy! Stream of user events •Status updates • Online transactions 3
- 4. Flink Forward - San Francisco, 2018 Social networks Online shopping Server monitoring Stream of user events • Status updates •Online transactions Stream of server events • CPU, memory, disk utilization Streams ahoy! 4
- 5. Flink Forward - San Francisco, 2018 Social networks Online shopping Server monitoring Sensors (IoT) Stream of user events • Status updates •Online transactions Stream of server events • CPU, memory, disk utilization Stream of sensor events • Temperature samples • Samples from radar and image sensors in cars Streams ahoy! 5
- 6. Workload cycles and seasonalspikes Flink Forward - San Francisco, 2018 6 Daily cycles NYC Yellow Taxi Trip Records, March 2015 http://www.nyc.gov/html/tlc/html/about/trip _record_data.shtml Seasonal spikes https://www.slideshare.net/iwmw/building- highly-scalable-web-applications/7- Seasonal_Spikes
- 7. Workload cycles and spikes Flink Forward -San Francisco, 2018 7 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Seasonal spikes 0:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 1:00 3:00 5:00 7:00 9:00 11:00 13:00 15:00 17:00 19:00 21:00 23:00 Daily cycles 0 2 4 6 8 10 12 14 Weekly cycles Unplanned
- 8. Overprovisioning… what ifwe don’t want to overprovision? Flink Forward - San Francisco, 2018 8
- 9. Event processing Flink Forward - San Francisco, 20189 Processor 1Source Source emits 2 events/second Processor processes 3 events/second Append-only Log Colors represent event keys
- 10. Event processing Flink Forward - San Francisco, 201810 Source Processor processes 3 events/second Processor 1 Source emits 2 events/second Append-only Log Colors represent event keys
- 11. Event processing Flink Forward - San Francisco, 201811 Source ü Source rate increases ü New rate: 4 events/second ü Processor still processes 3 events/second ü Can’t keep up with the source rate Processor 1 Append-only Log Colors represent event keys
- 12. Event processing Flink Forward - San Francisco, 201812 Source ü Source rate increases ü New rate: 4 events/second Processor 1 Append-only Log Processor 2 ü Add a second processor ü Each processor processes 3 events/second ü Can keep up with the rate Colors represent event keys
- 13. Event processing Flink Forward - San Francisco, 201813 Source ü Source rate increases ü New rate: 4 events/second Processor 1 Append-only Log Processor 2 ü Add a second processor ü Each processor processes 3 events/second ü Can keep up with the rate Problem: Per-key order
- 14. Event processing Flink Forward - San Francisco, 201814 Source Processor 1 Processor 2 ü Source rate increases ü New rate: 4 events/second ü Add a second processor ü Each processor processes 3 events/second ü Can keep up with the rate Split the input and add processors Append-only Log
- 15. Event processing Flink Forward - San Francisco, 201815 Source Processor 1 Processor 2 ü Source rate increases ü New rate: 4 events/second ü Add a second processor ü Each processor processes 3 events/second ü Can keep up with the rate Split the input and add processors Append-only Log Problem: Per-key order
- 16. Event processing Flink Forward - San Francisco, 201816 Source Processor 1 Processor 2 ü Source rate increases ü New rate: 4 events/second ü Add a second processor ü Each processor processes 3 events/second ü Can keep up with the rate Split the input and add processors Processor 2 only starts once earlier events have been processed
- 17. Flink Forward - San Francisco, 2018 17 Whatabout the order of events? What happens if the rate increases again? What if it drops?
- 18. Scaling in Pravega Flink Forward -San Francisco, 2018 18
- 19. Pravega • Storing datastreams • Young project, under active development • Open source http://pravega.io http://github.com/pravega/pravega 19Flink Forward - San Francisco, 2018
- 20.
- 21.
- 22.
- 23. Flink Forward - San Francisco, 2018 Time PresentRecent Past Distant Past Anatomy ofa stream Unbounded amount of data Ingestion rate might vary 23 Pravega
- 24. Pravega aims tobe a stream store able to: • Store stream data permanently • Preserve order • Accommodate unbounded streams • Adapt to varying workloads automatically • Low-latency from append to read Flink Forward - San Francisco, 2018 24
- 25. Pravega and Streams ….. 01110110 01100001 01101100 ….. 01001010 01101111 01101001 Pravega 01000110 01110110 Append Read 01000110 01110110 Flink Forward -San Francisco, 2018 Ingest stream data Process stream data 25
- 26. Pravega and Streams 01000110 01110110 Append Read Flink Forward -San Francisco, 2018 26 Event writer Event writer Event reader Event reader Group • Load balance • Grow and shrink Pravega Ingest stream data Process stream data
- 27. Segments in Pravega Flink Forward - San Francisco, 2018 01000111 01110110 11000110 01000111 01110110 11000110 Pravega Stream Composition of Segment: •Stream unit • Append only • Sequence of bytes 27
- 28.
- 29. Segments in Pravega Pravega 01000110 01110110 Segments Append Read 01000110 01110110 01101111 01101001 01101001 01101111 Segments •Segments are sequences of bytes • Use routing keys to determine segment Flink Forward - San Francisco, 2018 〈key, 01101001 〉 Routing key ….. 01110110 01100001 01101100 ….. 01001010 01101111 01101001 29
- 30. Segments can besealed Flink Forward - San Francisco, 2018 30
- 31. Segments in Pravega ….. 01110110 01100001 01101100 ….. 01101001 01110110 01001010 Pravega 01000110 01110110 Segments AppendRead 01000110 01110110 01101111 01101001 01101001 01101111 Segments Once sealed, a segment can’t be appended to any longer. Flink Forward - San Francisco, 2018 E.g., ad clicks 31
- 32. How is sealingsegments useful? Flink Forward - San Francisco, 2018 32
- 33.
- 34. Segments in Pravega 01000110 Segments Segments 01101111 01000110 01000110 01000110 01101111 01101111 01101111 01101111 01000110 01000110 0110111101101111 01000110 01101111 Stream Compose to form a stream • Each segment can live in a different server • Not limited to the capacity of a single server •Unbounded streams Flink Forward - San Francisco, 2018 00101111 01101001 34 Pravega
- 35.
- 36.
- 37. 01000110 Scaling a stream ….. 01110110 01100001 01101100 01000110 • Stream has one segment 1 ….. 01110110 01100001 01101100 •Seal current segment • Create new ones 2 01000110 01000110 • Say input load has increased • Need more parallelism • Auto or manual scaling Flink Forward - San Francisco, 2018 37
- 38. Routing key space 0.0 1.0 Time Split Split Merge 0.5 0.75 Segment 1Segment 2 Segment 3 Segment 4 Segment 5 Segment 6 t0 t1 t2 Flink Forward - San Francisco, 2018 38
- 39.
- 40.
- 41. Daily Cycles Peak rate is10x higher than lowest rate 4:00 AM 9:00 AM NYC Yellow Taxi Trip Records, March 2015 http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml
- 42.
- 43.
- 44.
- 45. How do Icontrol scaling? Flink Forward - San Francisco, 2018 45
- 46. Scaling policies • Configuredon a per stream basis • Specifies a policy for the stream • Policies • Fixed • Set of segments is fixed • Bytes per second • Scales up and down according to volume of data • Target data rate • Events per second • Scales up and down according to volume of events • Target event rate Flink Forward - San Francisco, 2018 46
- 47. Auto-Scaling: Triggering a scaling event •By byte and event rates • Target T per segment • Reports every 2 minutes ü 2-min rate (2M) ü 5-min rate (5M) ü 10-min rate (10M) ü 20-min rate (20M) Flink Forward - San Francisco, 2018 47 Scale up x x + 2 min x + 4 min x + 6 min time • Scaling down ∧ 2M, 5M, 10M < T ∧ 20M < T / 2 2M = 60 5M = 56 10M = 46 T = 50 2M = 60 5M = 60 10M = 48 T = 50 2M = 60 5M = 60 10M = 5 T = 50 2M = 60 5M = 60 10M = 52 T = 50 Scale down x x + 2 min x + 4 min x + 6 min time 2M = 20 5M = 20 10M = 20 20M = 27 T = 50 • Scaling up ∨ 2M > 5 x T ∨ 5M > 2 x T ∨ 10M > T 2M = 20 5M = 20 10M = 20 20M = 26 T = 50 2M = 20 5M = 20 10M = 20 20M = 25 T = 50 2M = 20 5M = 20 10M = 20 20M = 24 T = 50
- 48.
- 49.
- 50. Reader groups + Scaling Pravega Segment 2 Segment 1 Reader Reader 3 Segment 3 Segment 4 • Hit end of segment • Get successors •Update reader group state Pravega Reader Reader 4 Segment 4 Segment 2 Segment 3 Pravega Reader {3} Reader {2, 4} 5 Segment 4 Segment 2 Segment 3 Flink Forward - San Francisco, 2018 50
- 51. Building pipelines – Scalingdownstream Flink Forward - San Francisco, 2018 51
- 52. Scaling pipelines Flink Forward - San Francisco, 201852 Stage 1 Stage 2Source All stages can handle the load induced by the source
- 53. Scaling pipelines Flink Forward - San Francisco, 201853 Scaled Stage 1 Stage 2Big source Stage 2 can’t cope with the load change Load coming from source increases Stage 1 scales and adapts to the load change
- 54. Scaling signals Flink Forward - San Francisco, 201854 Pravega AppBig source • Pravega won’t scale the application
- 55. Scaling signals Flink Forward - San Francisco, 201855 Pravega AppBig source • Pravega won’t scale the application downstream • … but it can signal • E.g., more segments • E.g., number of unread bytes is growing Signals from Pravega
- 56. Reader group notifier • ListenerAPI • Register a listener to react to changes • E.g., changes to the number of segments Flink Forward - San Francisco, 2018 56 ReaderGroupManager groupManager = new ReaderGroupManagerImpl(SCOPE, controller, clientFactory, connectionFactory); ReaderGroup readerGroup = groupManager.createReaderGroup(GROUP_NAME, ReaderGroupConfig.builder().build(), Collections.singleton(STREAM)); readerGroup.getSegmentNotifier(executor).registerListener(segmentNotification -> { int numOfReaders = segmentNotification.getNumOfReaders(); int segments = segmentNotification.getNumOfSegments(); if (numOfReaders < segments) { //Scale up number of readers based on application capacity } else { //More readers available time to shut down some } });
- 57. Reader group: listener and metrics • ListenerAPI • Register a listener to react to changes • E.g., changes to the number of segments • Metrics • Reports specific values of interest • E.g., number of unread bytes in a stream Flink Forward - San Francisco, 2018 57
- 58. Consuming Pravega streams withApache Flink Flink Forward - San Francisco, 2018 58
- 59. How to read Pravega streams with Flink? Flink Forward - San Francisco, 201859 Task Manager ReaderPravega Stream • FlinkPravegaReader • ReaderGroup • Assignment of segments • Rebalance • Key to automatic scaling Task Manager Task Manager Task Manager Reader https://github.com/pravega/flink-connectors
- 60. How to react to segment changes? Flink Forward - San Francisco, 201860 Pravega Stream Task Manager Job Manager Reader Rescaling Policy Task Manager Task Manager (2) Segment change notification (1) Register segment listener (3) Rescale job (4) Take savepoint (5) Redeploy & resume tasks Reader
- 61. Scaling signals 61 • Latency • Throughput •Resource utilization • Connector signals Flink Forward - San Francisco, 2018
- 62. Rescaling Flink applications Flink Forward -San Francisco, 2018 62
- 63. Scaling stateless jobs 63 Scale Up Scale Down Source Mapper Sink • Scale up: Deploy new tasks •Scale down: Cancel running tasks Flink Forward - San Francisco, 2018
- 64. Scaling stateful jobs 64 ? • Problem: Which state to assign to new task? FlinkForward - San Francisco, 2018
- 65. Different state typesin Flink Flink Forward - San Francisco, 2018 65
- 66. Keyed vs. operator state 66 • State bound to a key • E.g. Keyed UDF and window state •State bound to a subtask • E.g. Source state Keyed Operator Flink Forward - San Francisco, 2018
- 67. Repartitioning keyed state • Similar to consistent hashing • Split key space into key groups •Assign key groups to tasks 67 Key space Key group #1 Key group #2 Key group #3Key group #4 Flink Forward - San Francisco, 2018
- 68.
- 69. Repartitioning operator state • Breaking operator state up into finer granularity • State has to contain multiple entries •Automatic repartitioning wrt granularity 69 #1 #2 #3 Flink Forward - San Francisco, 2018
- 70. Acquiring New Resources– Resource Elasticity Flink Forward - San Francisco, 2018 70
- 71. Flink’s Revamped Distributed Architecture Flink Forward -San Francisco, 2018 71 • Motivation • Resource elasticity • Support for different deployments • REST interface for client-cluster communication • Introduce generic building blocks • Compose blocks for different scenarios
- 72. The Building Blocks 72 • ClusterManager-specific • May live across jobs •Manages available Containers/TaskManagers • Used to acquire / release resources ResourceManager TaskManagerJobManager • Registers at ResourceManager • Gets tasks from one or more JobManagers • Single job only, started per job • Thinks in terms of "task slots" • Deploys and monitors job/task execution Dispatcher • Lives across jobs • Touch-point for job submissions • Spawns JobManagers Flink Forward - San Francisco, 2018
- 73.
- 74. Building Flink-on-YARN 74 YARN ResourceManager YARN Cluster YARN Cluster Client (1)Submit YARN App. (JobGraph / JARs) Application Master Flink-YARN ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn Application Master (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
- 75. Does It ActuallyWork? Flink Forward - San Francisco, 2018 75 Flink Forward - San Francisco, 2018
- 76.
- 77. Wrap Up Flink Forward - San Francisco, 201877 Flink Forward - San Francisco, 2018
- 78. Wrap up • Pravega •Stream store • Scalable ingestion of continuously generated data • Stream scaling • Apache Flink • Stateful job scaling • Full resource elasticity • Operator rescaling policies work in progress • Pravega + Apache Flink • End-to-end scalable data pipelines Flink Forward - San Francisco, 2018 78
- 79. Flink Forward - San Francisco, 2018 79 Questions? http://pravega.io http://github.com/pravega/pravega http://flink.apache.org http://github.com/pravega/flink-connectors https://github.com/tillrohrmann/flink/tree/rescalingPolicy E-mail:fpj@apache.org, trohrmann@apache.org Twitter: @fpjunqueira, @stsffap