IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets)

Kai Wähner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de November 2016 Comparison of Open Source Frameworks for Integrating the Internet of Things

© Copyright 2000-2016 TIBCO Software Inc. • used to develop lightweight IoT edge apps and microservices • part of a hybrid integration architecture • complementary to IoT cloud platforms and streaming analytics Key Takeaways IoT integration frameworks are...

© Copyright 2000-2016 TIBCO Software Inc. 1) IoT Trends and Challenges 2) Hybrid Integration Architecture 3) IoT Integration o Dataflow Pipeline o Stream Processing o Process Engine 4) Open Source IoT Process Engines o Eclipse Kura (+ Apache Camel) o Node-RED o Flogo 5) Cloud IoT Platforms Agenda

© Copyright 2000-2016 TIBCO Software Inc. Internet of Things (IoT) … refers to the ever-growing network of physical objects that feature an IP address for internet connectivity, and the communication that occurs between these objects and other Internet-enabled devices and systems.

© Copyright 2000-2016 TIBCO Software Inc. Integration is Key for Success of IoT Projects Without Integration there is no Internet of Things! Gartner Market Guide for IoT Integration “Through 2018, half the cost of implementing IoT solutions will be spent on integration. “ Strategic Planning Assumption, Benoit J. Lheureux, Massimo Pezzini, Alfonso Velosa !

© Copyright 2000-2016 TIBCO Software Inc. IoT raises several new challenges • Devices are not connected to the cloud • Devices have low bandwidth to connect • Latency of connectivity is significant • Connectivity is not reliable • Connectivity is not cost-effective Great IoT standards available • MQTT, CoaP, OPC US, many more • Different abstraction levels • Not the one single standard Challenges

© Copyright 2000-2016 TIBCO Software Inc. Integration Frameworks Decision Data Conversion Transitions Process Call Process Design Tools Integration Processes Connectors to vendor applications Connectors to technologies Connectors (e.g. JMS, MQTT, FTP) (e.g. SF.com, SAP)

© Copyright 2000-2016 TIBCO Software Inc. Enterprise Integration Patterns “The goal of EIPs is to document technology- independent design guidance that helps developers and architects describe and develop robust integration solutions.” http://www.eaipatterns.com

© Copyright 2000-2016 TIBCO Software Inc. Pervasive Integration with different Technologies and User Roles https://www.linkedin.com/pulse/how-implement-hybrid-integration-platform-tackle-massimo-pezzini There is no “one-size-fits-all” IoT integration!

© Copyright 2000-2016 TIBCO Software Inc. Hybrid Integration Platform (HIP) Application Integration Application Integration (on a PaaS) iPaaS Integration Platform as a Service iSaaS Integration Software as a Service Cloud-Ready Cloud-Native API Management On Premise Public Cloud Public Cloud Public Cloud On Premise Public Cloud Process Integration Business Process Management Platform Streaming Analytics Fog Computing Edge Devices Edge Integration Analysts are sure: HIP is “the new default” in most enterprises!

Typical IoT Integration Scenario What’s wrong with Cloud-only IoT Integration? • Connectivity is unreliable • Non-trivial latency in hops • Networks costs add up quickly

A better IoT Integration Blueprint Benefits: • Local control = more reliable • Less traffic = Lower TCO • Edge integration & event processing

© Copyright 2000-2016 TIBCO Software Inc. • Flow-based Programming • Ingest data from various sources • Extract (E) – Transform (T) – Load (L) • Transform / Filter / Route / Aggregate / Enrich … • High-throughput straight-through data flows • Data lineage • Synchronous and asynchronous communication • Batch or stream processing • Visual coding with flow editor • Not to be mixed-up with stream processing for event correlation Dataflow Pipeline - Characteristics

© Copyright 2000-2016 TIBCO Software Inc. Stream Processing time 1 2 3 4 5 6 7 8 9 Event Streams • Continuous Queries • Sliding Windows • Filter • Aggregation • Correlation • … Stream Processing H20.ai Open Source R TERR Spark ML MATLAB SAS PMML

© Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics à Predictive Maintenance Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert.”

© Copyright 2000-2016 TIBCO Software Inc. • Integrate and orchestrate various data sources • Wire together hardware devices, APIs and online services • Transform / Filter / Route / Aggregate / Enrich … • Error handling, Re-Try, Re-Routing, Wait, Resume, ... • Not just Extract-Transform-Load • Synchronous and asynchronous communication • Typically real time processing • Visual coding with flow editor • Deployed at the edge Process Engine

© Copyright 2000-2016 TIBCO Software Inc. Commonalities: • Open source frameworks • Connectivity to IoT technologies (MQTT, CoaP, REST, …) • Web UI for visual coding, testing, debugging • Deployable “at the edge” • For developers / integration specialists / citizen integrators • Extendable SDKs / APIs Open Source IoT Process Engines

© Copyright 2000-2016 TIBCO Software Inc. • Focus on IoT Gateway • Set of Java and OSGi services • including I/O services, Data Services, Cloud Services, Networking, etc. • Web UI for Configuration (Devices, Network, Protocols, etc.) • Eclipse Public License 1.0 • Mature framework (~3 years old, code based committed in December 2013) • Best of breed with focus on coding instead of a visual designer • Eclipse IoT Projects • IoT Gateway --> Eclipse Kura • Services like Eclipse SmartHome • Standard implementations like Mosquitto (MQTT Server) • Connectivity via Eclipse Paho (MQTT Client) • Apache Camel Connector • Integration Framework (connectivity, enterprise integration patterns) • Visual coding via JBoss or Talend (both available as open source edition) • For Developers / Integration Specialists only • More complex to install and build a first IoT integration flow • Write source code • Can be run on a variety of platforms (on premise, cloud, edge devices, container) Eclipse Kura (+ Apache Camel) à Facts

© Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway • Built on Node.js and Javascript • Very easy to install and build a first IoT integration flow • For developers / integration specialists and ciziten integrators • Visual coding via web designer • Apache 2.0 License • Mature framework (~3 years old) with many examples and docs • Native integration into IBM Bluemix cloud platform • Leverage other Bluemix services • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • No binaries to share and deploy on devices Node-RED à Facts

© Copyright 2000-2016 TIBCO Software Inc. • Focus on Integration on an IoT Gateway and very lightweight edge applications • Powered by Golang • Very easy to install and build a first IoT integration flow • Ultra-lightweight footprint with zero dependency model • For developers / integration specialists and ciziten integrators • Visual coding via web designer • BSD-style License • Developer Preview (released in October 2016) • Can be run on a variety of platforms (on premise, cloud, edge devices, container) • Flows can be shared as JSON strings / files • Lightweight binaries with zero dependencies can be shared and deployed on devices Flogo à Facts

Web-Native Step-Back Debugger • Step back in a flow • Interactively design/debug your process • Simulate sensor events • Change data or configuration and run from Trigger or any Activity • Underlying technology could be used to support remote ops debugging of failed flows

What technology to build edge apps? Java • Heavy (>200MB) • Not agile • Robust • Licensing risk Node.js • Lighter (~70MB) • Agile • Not IoT robust • Clear licensing C/C++ • Lightest • Not Agile • Robust • Clear licensing

© Copyright 2000-2016 TIBCO Software Inc. Why Go? Modern programming language Getting a lot of traction due to usage in frameworks like Docker or Kubernetes Concurrency Built into the language (Goroutines, channels, no explicit thread programming) Memory management Modern Garbage collector Statically typed Type-safe development with no “surprises” during runtime “Partly” object-oriented Simple and flexible type system, composition instead of inheritance Zero-dependency programming model All-included binary file, no dependent “external” libraries Speed Ultra fast compilation and startup time, very lightweight footprint

Infrastructure Layers Hardware OS VM - JVM Framework - OSGi App Eclipse Kura Hardware OS VM – V8 Framework – Node.js App Node-RED Hardware OS App Flogo Typically 100s of dependencies – 100MB+ “All included” zero- dependency model enabled by Go

Hardware OS Security Logging Metrics Hardware OS V8 Protocol Bridges Diagnostics Sensor Processing V8 Node.js Logging Node.js Security Whoa! Overhead for every app means you run out of room for your apps Node-RED Flogo

Comparison – Resource Requirements of IoT Frameworks Eclipse Kura Node-RED Flogo VM JVM V8 Golang Base Disk Space 59 MB 56 MB ~ 0 MB Base Runtime Memory ~ 170 MB (with Open JDK) ~ 20 MB (with Oracle Embedded Java) > 50 MB ~ 5 MB Startup time Slow (~8 sec) Slow (~5 sec) Fast (~1 sec) Application Build Slow Not Applicable Fast (~2 secs)

Outlook: Flogo Nanoservices on Microcontrollers Flogo nanoservices that target microcontrollers like ARM M0 Offload I/O centric compute farther off to the edge.

Intel IoT Reference Architecture Potential use cases • Event-driven flows & rules • Device Lifecycle Mgmt • User Experience Core Integration Agile Microservices API Management

Amazon IoT Reference Architecture Potential use cases • Event-driven Flows • Device Lifecycle Mgmt • Orchestration Core Integration Agile Microservices API Management

Google IoT Reference Architecture Potential edge apps

© Copyright 2000-2016 TIBCO Software Inc. Serverless Architecture with IBM’s OpenWhisk (Open Source) https://apiguy.tokyo/2016/03/01/openwhisk-a-world-first-in-open-serverless-architecture/ Edge App Edge Gateway Ultra- Lightweight Serverless App

Questions? Please contact me! Kai Wähner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de LinkedIn

IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets)

More Related Content

What's hot

Viewers also liked

Similar to IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets)

More from Kai Wähner

Recently uploaded

IoT Open Source Integration Comparison (Kura, Node-RED, Flogo, Apache Nifi, StreamSets)