Downloaded 25 times
Uploaded byBahadir Cambel
Onyx data processing the clojure way
Onyx is a data processing framework for Clojure that allows users to define workflows, functions, and windows to process streaming and batch data across distributed clusters. It uses concepts like peers, virtual peers, and Zookeeper for scheduling and Aeron for messaging. Users can write Onyx jobs in Clojure to perform ETL, analytics, and other data processing tasks in a declarative way.
Onyx data processing the clojure way
- 1. Onyx - Data Processing TheClojure Way By Bahadir Cambel @bahadircambel
- 2. Raise your handsif you have used - Cascalog - Hadoop - Spark - Flink - Samza - Storm - Sqoop
- 3. What’s good for? Realtime event stream processing Continuous computation Extract, transform, load (ETL) Data transformation à la map-reduce Data ingestion and storage medium transfer Data cleaning
- 4. Data Structures +Simple Functions - Sounds familiar ?
- 5.
- 7.
- 9.
- 11.
- 13.
- 15. Internet Scale MongoDBcould save youu?!
- 17.
- 18. Workflow - articulate thepaths that data flows through the cluster at runtime - DAG
- 19. Catalog - Describe andconfigure workflow items
- 20.
- 21. Flow Conditions From ->To ( if predicate correct) Flow conditions are used for isolating logic about whether or not segments should pass through different tasks in a workflow, exception handling and support a rich degree of composition with runtime parameterization.
- 22. Windows / Triggers partitionsa possible unbounded sequence of data into finite pieces, allowing aggregations to be specified - Timer - Segment - Punctuation - Watermark
- 23. Life Cycles allows youto hook in and execute arbitrary code at critical points during a task (kinda middleware) :lifecycle/start-task? :lifecycle/before-task-start :lifecycle/before-batch :lifecycle/after-read-batch :lifecycle/after-batch :lifecycle/after-task-stop :lifecycle/after-ack-segment :lifecycle/after-retry-segment
- 24. Job A job willbe translated into multiple tasks. Peers will take care of these tasks. If your number of tasks > available peers A job won’t be complete ( Buy me a beer or 10)
- 25. Bulk functions perform afn more efficiently over a batch of segments rather than processing one segment at a time. - Write to DB Onyx will ignore the output of your function and pass the same segments that you received downstream
- 26. Group by “like” valuesare always routed to the same virtual peer - Group by key - Group by a fn Specify in the catalog!
- 27. Fixed Windows a datapoint will fall into exactly one instance of a window (often called an extent in the literature) Between t1=0 and t2=4 how many events have happened? t1=5 t2=9, t1=10 t2=14 And so on..
- 28. Sliding Window a slidevalue for how long to wait between spawning a new window extent Between t1=0 and t2=14 how many events have happened? t1=5 t2=19 ? t1=10 t2=24 ?
- 29.
- 30. Session Window dynamically resizetheir upper and lower bounds in reaction to incoming data Sessions capture a time span of activity for a specific key, such as a user ID. If no activity occurs within a timeout gap, the session closes. If an event occurs within the bounds of a session, the window size is fused with the new event, and the session is extended by its timeout gap either in the forward or backward direction
- 31.
- 32.
- 33. Peer is a nodein the cluster responsible for processing data
- 34. Virtual Peer A VirtualPeer refers to a single peer process running on a single physical machine. A single Virtual Peer executes at most one task at a time.
- 35.
- 36. Aeron Efficient reliable UDPunicast, UDP multicast, and IPC message transport Messaging layer takes care of the direct peer to peer transfer of segment batches, acks, segment completion and segment retries to the relevant virtual peers.
- 37.
- 38. Scheduling If there isno master, how does scheduling work ? Peers contend to work on tasks.
- 39. Types of JobSchedulers - Greedy ( I need ALL!!!! Gimme all!!) - Balanced Robin ( Fair play) - Percentage ( Not so fair play)
- 40. Types of TaskSchedulers - Balanced - Percentage - Colocation (assigns them to the peers on a single physical machine, low latency, min network)
- 41. Tags a set ofmachines in your cluster are privileged Run some tasks at some specific machines Declare a peer with capabilities - Datomic - Special Hardware (GPU, Memory) - Network
- 42.
- 43. Example - Let’sprocess some logs 49556677821280438558577372995495836672945903576549425154
- 44. Check out therepository https://github.com/bcambel/onyx- test
- 45. End users configuringwhat - workflows should look like. - Language agnostic - Location agnostic - Tolerant to machine generation - Temporally agnostic ( should wait for a time to be realized)
- 46. If you arenot enjoying your experience There is something fundamentally wrong with the tool Think about Apple’s smooth product experience. Pain detected, thought through. (Pain -> Pleasure)
- 47. Tools - Onyx ETLhttps://github.com/onyx-platform/onyx-etl - Dashboard https://github.com/onyx-platform/onyx-dashboard - Replica Cons https://github.com/onyx-platform/onyx-console-dashboard - Ansible Playbook https://github.com/onyx-platform/ansible-onyx - Metrics Suite https://github.com/onyx-platform/onyx-metrics - Benchmark Suite https://github.com/onyx-platform/onyx-benchmark - Jepsen https://github.com/onyx-platform/onyx-jepsen
- 48.
- 49. Questions - How doesOnyx distributes reads (input tasks) ? Parallelization?? - evenly break up a database table into chunks which can be read by multiple peers - Segments realized. Tasks created. Peers get into action
- 50.
- 51.
- 52. Thanks Michael Drogalis -https://twitter.com/michaeldrogalis Lucas Bradstreet - https://twitter.com/ghaz
- 53. Thanks for usingClojure!