Download as PDF, PPTX
![Job = [[:input :processing-1] [:input :processing-2] [:processing-1 :output-1] [:processing-2 :output-2]] [{:flow/from :input-stream :flow/to [:process-adults] :flow/predicate :my.ns/adult? :flow/doc "Emits segment if an adult.”}] workflow + flow conditions + catalogue[{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}]](https://image.slidesharecdn.com/adatalayerinclojure-170427162815/75/A-data-layer-in-clojure-10-2048.jpg)
![Catalogue [{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}] Vanilla Clojure function
(defn adder [n {:keys [x] :as segment}] (assoc segment :x (+ n x)))) Plugins (I/O) seq, async, Kafka, Datomic, SQL, S3, SQS, … parameter self-documenting](https://image.slidesharecdn.com/adatalayerinclojure-170427162815/75/A-data-layer-in-clojure-11-2048.jpg)
Uploaded bySimon Belak
A data layer in clojure
The document discusses a data layer in Clojure, focusing on the Onyx framework, which enables near real-time data processing and supports both batch and streaming machine learning workflows. It highlights the functionality, architecture, and self-service infrastructure for data scientists, as well as the use of Clojure for data manipulation and the automatic generation of materialized views based on event data. Finally, it emphasizes the importance of clean architecture and the concept of 'data is code' in structuring data processing.
A data layer in clojure
- 1. A data layerin Clojure @sbelak simon@goopti.com
- 2. • Started inmachine learning • Turned to data science and helped 20+ companies become data-driven • Now leading data science department at GoOpti
- 3. Self-service infrastructure for datascientists
- 4. The analytics chasm Ideal.Almost real-time, can be done during brainstorming without disrupting flow < 2min < 20min project squeeze in somewhere in the day fail roadmap ahoy!
- 5. My goto architecture KafkaDBEvents Onyx Onyx Onyx Persist all events to S3 • time travel • query with AWS Athena
- 6. Onyxa masterless, cloudscale, fault tolerant, high performance distributed computation system … written entirely in Clojure
- 7. Clojure at aglance • Lisp running on JVM • Functional, dynamic, immutable • Excellent concurrency and state management support • Unparalleled data manipulation • Good Java interoperability
- 8. Onyx at • Inproduction for almost a year • ETL • online machine learning • offline (batch) machine learning • ad-hoc analysis
- 9. Onyx at aglance
- 10. Job = [[:input :processing-1] [:input:processing-2] [:processing-1 :output-1] [:processing-2 :output-2]] [{:flow/from :input-stream :flow/to [:process-adults] :flow/predicate :my.ns/adult? :flow/doc "Emits segment if an adult.”}] workflow + flow conditions + catalogue[{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type :function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}]
- 11. Catalogue [{:onyx/name :add-5 :onyx/fn :my/adder :onyx/type:function :my/n 5 :onyx/params [:my/n]} {:onyx/name :in :onyx/plugin :onyx.plugin.core-async/input :onyx/type :input :onyx/medium :core.async :onyx/batch-size batch-size :onyx/max-peers 1 :onyx/doc "Reads segments from a core.async channel"} {:onyx/name :out :onyx/plugin :onyx.plugin.core-async/output :onyx/type :output :onyx/medium :core.async :onyx/doc "Writes segments to a core.async channel"}] Vanilla Clojure function (defn adder [n {:keys [x] :as segment}] (assoc segment :x (+ n x)))) Plugins (I/O) seq, async, Kafka, Datomic, SQL, S3, SQS, … parameter self-documenting
- 12. Computation entirely described withdata data is code!
- 13. Everything can berun locally!
- 14.
- 15. Resilience and handling state •Activity log • Window and trigger states checkpointed • Resume points • Configurable flux policies
- 16. How Onyx rewiredmy brain
- 17. It’s not aboutscaling, but clean architecture
- 18.
- 19.
- 20. Machine learning withOnyx • Hyperparameter server build on top of Onyx parameters • Batch & streaming mode • Monitoring: performance metrics, side channels for partial results/introspection into computiation • Everything is data so easy to build tools around
- 21.
- 22. Putting “data iscode” to work
- 23. Describing data with clojure.spec composingsmaller parts into the whole} code is data!
- 24. Queryable data descriptions Turn specinto a graph A fully interactive and open type system! order promo code user account age country always always alwaysmaybe
- 25. “Composition is about decomposing.” —E. Normand
- 26. Case study: autogeneratingmaterialised views Kafka Materialised views Events External data Automatic view generation • Event & attribute ontology • Manual (via spec) • Inferred • Statistical analysis (seasonality detection, outlier removal, …) Onyx Onyx Onyx
- 27. Automatic view generation 1.Walk spec registry 2. Apply rules 1. Define new view (spec) 2. Trigger Onyx job that creates the view ⤾
- 28.
- 29. Everything should be liveand interactive
- 30. Computation graphs are agreat way to structure data processing code
- 31. Queryable data and computationdescriptions supercharge interactive development and are a great building block for automation
- 32.
- 33.