Flow Based Programming and Reactive Programming with node.js Sven Beauprez
2 Who am I? ● Independent Enterprise Solution Architect – Specialized in SOA and EAI – Strong background in Java web development since '90s ● Steering member of IoT BE ● Steering member of JavaPolis/Devoxx 2002 – 2009 ● Keep in touch – sven.beauprez@prorabel.com – https://www.linkedin.com/in/svenbeauprez
3 Node.js main characterisitcs ● … runs as a single process – Does not start a new process for each new request, which would mean a lot of overhead ● … runs applications in a single thread – Does not start a new thread for each new request, which means less complex code and avoiding race conditions ● … is asynchronous – Can handle multiple requests at one time without blocking new requests by using I/O eventing, callbacks and an event loop ● … is ideal for I/O intensive tasks, but less for CPU intensive – CPU intensive tasks within the application would block the process and must be offloaded in worker threads/processes In other words a perfect fit for IoT where the main tasks on the edges and in the fog is collecting sensor data
4 Node.js event loop* waiting waiting Event Loop (single thread) Incoming request Query DB (offloaded) Process result Write log (offloaded) Respond Thread Pool & Async I/O Filesystem Network Other Event Queue *http://kunkle.org/nodejs-explained-pres
5 Node.js callback hell callback 1 callback 2 callback 3 ● Naive example of a node.js application Confusing, this gives even wrong result
6 Node.js promises ● Also called futures, deffereds, eventuals,... ● A promise is an object (proxy) that represents the result of an asynchronous function call var promise = doSomethingAync() promise.then(onFulfilled, onRejected) ● Biggest advantage of promises is that they can be chained to avoid callback hell readFile().then(readAnotherFile).then(doSomethingElse).then(...) ● Promises can return any other value and the next 'onFulfilled' will be passed the value as argument readFile().then(function (buf) {return JSON.parse(buf.toString())}) .then(function (data) {// do something with `data`}) Promises are a simple form of dataflow programming
7 What is dataflow? ● Traditionally a program is modeled as a series of operations happening in a specific order, in other words how to do computations. Source: http://www.autobild.de/
8 What is dataflow? ● Dataflow programming emphasizes the movement of data and models programs as a series of connection. It describes what computation should be performed. Source: http://commons.wikimedia.org/wiki/File:Hyundai_car_assembly_line.jpg
9 Flow Based Programming (FBP) ● Describes a graph of nodes, which exchange messages containing data via the edges. The edges are defined outside the nodes, in others words nodes have no control on where the data comes from and where it goes to
10 FBP: Unix Pipes ● Unix pipes are an example of linear flow based programming while a stream of data follows the pipes and the different 'smaller' dedictated applications run in parallel ps aux | grep myApp | grep -v grep | awk '{print $2}' | xargs kill
11 FBP: Yahoo Pipes Source: https://pipes.yahoo.com/pipes/pipe.info?_id=ZKJobpaj3BGZOew9G8evXg
12 FBP: Enterprise Application Integration http://www.eaipatterns.com/Chapter1.html
13 FBP: Node-RED
14 FBP: Node-RED ● According to the founders: – A light-weight, edge of network, application builder – Easy to use, drag and drop interface – A tool that allows the developer to focus on the task at hand ● My definition: – A network of connected light-weight nodes to process stream(s) of edge generated events via message passing
15 FBP: Node-RED ● Messages are JSON objects “payload : “Hello world!” “topic” : “foo” ……
16 FBP: Node-RED ● Different categories of nodes are defined: – Input: to process incoming events ● Example: Inject trigger, HTTP, TCP, UDP, websocket, MQTT, file, Datastore, social (Twitter), … and custom nodes – Output: to serve or send outgoing events ● Example: Debug, HTTP, TCP, UDP, websocket, MQTT, file, Datastore, social (Twitter), … and custom nodes – Functions: to manipulate messages and message payloads ● Transform, filter, map, analyse, …
17 FBP: Node-RED – Function node ● Generic Function node: – Runs user-defined javascript code in a sandbox – Returns (forward) custom build messages with payloads ● return msg; ● return [msg1, msg2, msg3]; ● return [msg1, [msgA, msgB]];
18 FBP: Node-RED – Function Context ● Each node has its own context to save state between calls – Since node.js is single threaded, no race conditions can occur and the state is always correct and up to date
19 FBP: Node-RED – Custom Nodes ● Custom nodes are defined via a pair of files: – .js : ● defines what the node does ● runs on the server – .html : ● defines the node's properties, edit dialog and help text ● runs in a browser ● When configuration needs to be shared between nodes, eg. IN node and the related OUT node share the same IP address, a special config node can be defined – A dependeny on the config node is added in the custom node
20 FBP: Node-RED Demo
21 (Functional) Reactive Programming (FRP) ● What if C = A + B meant C always equals A plus B, at any time?
22 FRP: spreadsheets are reactive ● The functions in a cell/chart react on changes in another cell
23 FRP: dynamic values ● In reactive programming, datatypes represent a value “over time”* – An example is getting coördinates of the mouse x = <mouse-x> y = <mouse-y> in reactive programming, the assignment needs to be made only once and the variables/properties will always be up to date with the latest values – All following computations based on these variables will also be values that change over time minX = x - 16; minY = y - 16; maxX = x + 16; maxY = y + 16; //draw 32x32 box around mouse pointer rectangle(minX, minY, maxX, maxY) *http://stackoverflow.com/a/1028642
24 FRP: bacon.js how does it look like? http://jsfiddle.net/njs3szqe/
25 FRP: bacon.js 'c = a + b', always http://jsfiddle.net/Lpe77wjk/
26 FRP: 'functional' reactive programming ● 'Functional' means that building blocks of functional programming are used – Ideas are borrowed from languages such as Haskell, Lisp, Scheme,... ● Higher order functions are used extensively all over the place – Higher order = takes one or more functions as input and/or outputs a function – Examples: ● Map, filter, scan, combine, takeWhile,... ● See https://github.com/baconjs/bacon.js/ for an exhaustive list supported by bacon ● See https://github.com/ReactiveX/RxJava/wiki/Observable for visual representation of many of these functions
27 FRP: higher order function map() ● Transform items emitted by a stream/property by applying a function to each of them https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables#map
28 FRP: higher order function filter() ● Filter items emitted by a stream https://github.com/ReactiveX/RxJava/wiki/Filtering-Observables#filter
29 FRP: higher order function scan() ● Apply a function to each item emitted by a stream and emit each successive value https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables#scan
30 FRP: higher order function takeWhile() ● Emit items emitted by a stream as long as a specified condition is true, then skip the remainder https://github.com/ReactiveX/RxJava/wiki/Conditional-and-Boolean-Operators#takewhile-and-takewhilewithindex
FRP: Applying FRP in IoT – Bacon.js & Tessel 31
32 Q&A

Flow Base Programming with Node-RED and Functional Reactive Programming with Baconjs

  • 1.
    Flow Based Programmingand Reactive Programming with node.js Sven Beauprez
  • 2.
    2 Who amI? ● Independent Enterprise Solution Architect – Specialized in SOA and EAI – Strong background in Java web development since '90s ● Steering member of IoT BE ● Steering member of JavaPolis/Devoxx 2002 – 2009 ● Keep in touch – sven.beauprez@prorabel.com – https://www.linkedin.com/in/svenbeauprez
  • 3.
    3 Node.js maincharacterisitcs ● … runs as a single process – Does not start a new process for each new request, which would mean a lot of overhead ● … runs applications in a single thread – Does not start a new thread for each new request, which means less complex code and avoiding race conditions ● … is asynchronous – Can handle multiple requests at one time without blocking new requests by using I/O eventing, callbacks and an event loop ● … is ideal for I/O intensive tasks, but less for CPU intensive – CPU intensive tasks within the application would block the process and must be offloaded in worker threads/processes In other words a perfect fit for IoT where the main tasks on the edges and in the fog is collecting sensor data
  • 4.
    4 Node.js eventloop* waiting waiting Event Loop (single thread) Incoming request Query DB (offloaded) Process result Write log (offloaded) Respond Thread Pool & Async I/O Filesystem Network Other Event Queue *http://kunkle.org/nodejs-explained-pres
  • 5.
    5 Node.js callbackhell callback 1 callback 2 callback 3 ● Naive example of a node.js application Confusing, this gives even wrong result
  • 6.
    6 Node.js promises ● Also called futures, deffereds, eventuals,... ● A promise is an object (proxy) that represents the result of an asynchronous function call var promise = doSomethingAync() promise.then(onFulfilled, onRejected) ● Biggest advantage of promises is that they can be chained to avoid callback hell readFile().then(readAnotherFile).then(doSomethingElse).then(...) ● Promises can return any other value and the next 'onFulfilled' will be passed the value as argument readFile().then(function (buf) {return JSON.parse(buf.toString())}) .then(function (data) {// do something with `data`}) Promises are a simple form of dataflow programming
  • 7.
    7 What isdataflow? ● Traditionally a program is modeled as a series of operations happening in a specific order, in other words how to do computations. Source: http://www.autobild.de/
  • 8.
    8 What isdataflow? ● Dataflow programming emphasizes the movement of data and models programs as a series of connection. It describes what computation should be performed. Source: http://commons.wikimedia.org/wiki/File:Hyundai_car_assembly_line.jpg
  • 9.
    9 Flow BasedProgramming (FBP) ● Describes a graph of nodes, which exchange messages containing data via the edges. The edges are defined outside the nodes, in others words nodes have no control on where the data comes from and where it goes to
  • 10.
    10 FBP: UnixPipes ● Unix pipes are an example of linear flow based programming while a stream of data follows the pipes and the different 'smaller' dedictated applications run in parallel ps aux | grep myApp | grep -v grep | awk '{print $2}' | xargs kill
  • 11.
    11 FBP: YahooPipes Source: https://pipes.yahoo.com/pipes/pipe.info?_id=ZKJobpaj3BGZOew9G8evXg
  • 12.
    12 FBP: EnterpriseApplication Integration http://www.eaipatterns.com/Chapter1.html
  • 13.
  • 14.
    14 FBP: Node-RED ● According to the founders: – A light-weight, edge of network, application builder – Easy to use, drag and drop interface – A tool that allows the developer to focus on the task at hand ● My definition: – A network of connected light-weight nodes to process stream(s) of edge generated events via message passing
  • 15.
    15 FBP: Node-RED ● Messages are JSON objects “payload : “Hello world!” “topic” : “foo” ……
  • 16.
    16 FBP: Node-RED ● Different categories of nodes are defined: – Input: to process incoming events ● Example: Inject trigger, HTTP, TCP, UDP, websocket, MQTT, file, Datastore, social (Twitter), … and custom nodes – Output: to serve or send outgoing events ● Example: Debug, HTTP, TCP, UDP, websocket, MQTT, file, Datastore, social (Twitter), … and custom nodes – Functions: to manipulate messages and message payloads ● Transform, filter, map, analyse, …
  • 17.
    17 FBP: Node-RED– Function node ● Generic Function node: – Runs user-defined javascript code in a sandbox – Returns (forward) custom build messages with payloads ● return msg; ● return [msg1, msg2, msg3]; ● return [msg1, [msgA, msgB]];
  • 18.
    18 FBP: Node-RED– Function Context ● Each node has its own context to save state between calls – Since node.js is single threaded, no race conditions can occur and the state is always correct and up to date
  • 19.
    19 FBP: Node-RED– Custom Nodes ● Custom nodes are defined via a pair of files: – .js : ● defines what the node does ● runs on the server – .html : ● defines the node's properties, edit dialog and help text ● runs in a browser ● When configuration needs to be shared between nodes, eg. IN node and the related OUT node share the same IP address, a special config node can be defined – A dependeny on the config node is added in the custom node
  • 20.
  • 21.
    21 (Functional) ReactiveProgramming (FRP) ● What if C = A + B meant C always equals A plus B, at any time?
  • 22.
    22 FRP: spreadsheetsare reactive ● The functions in a cell/chart react on changes in another cell
  • 23.
    23 FRP: dynamicvalues ● In reactive programming, datatypes represent a value “over time”* – An example is getting coördinates of the mouse x = <mouse-x> y = <mouse-y> in reactive programming, the assignment needs to be made only once and the variables/properties will always be up to date with the latest values – All following computations based on these variables will also be values that change over time minX = x - 16; minY = y - 16; maxX = x + 16; maxY = y + 16; //draw 32x32 box around mouse pointer rectangle(minX, minY, maxX, maxY) *http://stackoverflow.com/a/1028642
  • 24.
    24 FRP: bacon.jshow does it look like? http://jsfiddle.net/njs3szqe/
  • 25.
    25 FRP: bacon.js'c = a + b', always http://jsfiddle.net/Lpe77wjk/
  • 26.
    26 FRP: 'functional'reactive programming ● 'Functional' means that building blocks of functional programming are used – Ideas are borrowed from languages such as Haskell, Lisp, Scheme,... ● Higher order functions are used extensively all over the place – Higher order = takes one or more functions as input and/or outputs a function – Examples: ● Map, filter, scan, combine, takeWhile,... ● See https://github.com/baconjs/bacon.js/ for an exhaustive list supported by bacon ● See https://github.com/ReactiveX/RxJava/wiki/Observable for visual representation of many of these functions
  • 27.
    27 FRP: higherorder function map() ● Transform items emitted by a stream/property by applying a function to each of them https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables#map
  • 28.
    28 FRP: higherorder function filter() ● Filter items emitted by a stream https://github.com/ReactiveX/RxJava/wiki/Filtering-Observables#filter
  • 29.
    29 FRP: higherorder function scan() ● Apply a function to each item emitted by a stream and emit each successive value https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables#scan
  • 30.
    30 FRP: higherorder function takeWhile() ● Emit items emitted by a stream as long as a specified condition is true, then skip the remainder https://github.com/ReactiveX/RxJava/wiki/Conditional-and-Boolean-Operators#takewhile-and-takewhilewithindex
  • 31.
    FRP: Applying FRPin IoT – Bacon.js & Tessel 31
  • 32.