Introduction to AIoT & TinyML - with Arduino

Intro to TinyML Technical, implementation with Arduino Arduino Day 2020 Online Event Bandung, Indonesia Mar 21, 2020 Andri Yadi CEO, DycodeX Microsoft MVP, Azure

Andri Yadi Co-founder & CEO of DycodeX Vice Chairman, Indonesia IoT Association (ASIOTI) Microsoft MVP, Azure a (at) dycodex.com | andriyadi.com | github.com/andriyadi Physicist, Developer, Maker, Community Guy, Entrepreneur About Me MicrosoJ Most Valuable Professional (MVP) for 12 years Code for food & passion for 20 years Break & make electronic stuOs for 22 years Trying to change the world through entrepreneurship, 15 years now

I’VE BEEN PROMOTING AIOT & TINYML 20+ TALKS LAST YEAR

Communication Networks IoT High Level Architecture Gateways / Base Station Things (lots of them) Internet Apps Cloud User Internet Ingestion Infrastructure Logics API

GNSS 176 Gyroscope 130 Accelerometer 21 Bluetooth 100 Radio 800 Display 400 in mW Clearly Radio consumes almost 50% of power. 65% if no display! Typical IoT Device’s Energy Consumption

GNSS 176 Gyroscope 130 Accelerometer 21 Bluetooth 100 Radio 800 Display 400 in mW Clearly Radio consumes almost 50% of power. 65% if no display! Typical IoT Device’s Energy Consumption LPWA for IoT wireless connectivity is the option LPWA = Low Power Wide Area Network

You oKen see… about LPWA connectivity

You oKen see… 10 years baOery life is GIMMICK! about LPWA connectivity

You oKen see… Achievable by 2-3 of these combinations: Using big capacity ba_ery (e.g. 19Ah) Transmi_ing data few times a day Payload size 10s-100s bytes ➙ shob tx 10 years baOery life is GIMMICK! Notes: NB-IoT (or other LPWA connectivities) does consume much lower power than WiFi or 4G/LTE, due to its low data rate. Only tens to hundreds mA per-tx about LPWA connectivity

So, if you have… Sensors: GPS / GNSS Accelerometer, Gyroscope Body Temperature Ambient Temperature & Humidity Barometric Pressure Ambient Light MEMS Microphone Device Removal Detector SMARTernak electronics board

So, if you have… Sensors: GPS / GNSS Accelerometer, Gyroscope Body Temperature Ambient Temperature & Humidity Barometric Pressure Ambient Light MEMS Microphone Device Removal Detector You may only transmit: Current lat, long, speed, direction Current values or average motion Current, peak, or lowest body temperature Current, peak, or avg. temperature & humidity Current, peak, or avg. Barometric Pressure Current, peak, or lowest ambient Light Current, peak, or lowest sound level in db Detached or not SMARTernak electronics board 99% of sensor data has to be discarded due to connectivity’s bandwidth or power constraints

Motor failure detection Dashboard For that, usually we collect raw data from vibration sensor and transmit to the Cloud. In the cloud, raw data are processed to classify whether motor is working normal or fault. On the other hand, consider a use case… MicrocontrollerMotor

Motor failure detection Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Dashboard For that, usually we collect raw data from vibration sensor and transmit to the Cloud. In the cloud, raw data are processed to classify whether motor is working normal or fault. On the other hand, consider a use case… MicrocontrollerMotor

Motor failure detection Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Dashboard For that, usually we collect raw data from vibration sensor and transmit to the Cloud. In the cloud, raw data are processed to classify whether motor is working normal or fault. On the other hand, consider a use case… MicrocontrollerMotor Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes)

Motor failure detection Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Fault detected Dashboard For that, usually we collect raw data from vibration sensor and transmit to the Cloud. In the cloud, raw data are processed to classify whether motor is working normal or fault. On the other hand, consider a use case… MicrocontrollerMotor Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes)

Motor failure detection Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Fault detected Dashboard If we try to send 1KB of raw data over LPWA (e.g. LoRa), it may take “forever”, and consume more power For that, usually we collect raw data from vibration sensor and transmit to the Cloud. In the cloud, raw data are processed to classify whether motor is working normal or fault. On the other hand, consider a use case… MicrocontrollerMotor Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) …it will beat the purpose of low power connectivity LPWA

Motor failure detection Dashboard Due to bandwidth and power constraints, we only send sampled and calculated data So, what we may do, instead MicrocontrollerMotor Peak Peak value FFT Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Sampled/Calculated LPWA

Motor failure detection Dashboard Due to bandwidth and power constraints, we only send sampled and calculated data So, what we may do, instead MicrocontrollerMotor But we may miss lot of interesting events! Peak Peak value FFT Fault detected Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) Sampled/Calculated LPWA

Motor failure detection Dashboard There’s on-device “intelligence” to do complete analysis, right on the device. Then, only transmit the conclusion (just few bytes of data) to the cloud. Somehow… MicrocontrollerMotor On-device processing Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) LPWA

Motor failure detection Dashboard There’s on-device “intelligence” to do complete analysis, right on the device. Then, only transmit the conclusion (just few bytes of data) to the cloud. Somehow… MicrocontrollerMotor Processed data 0A 0B 01 (➙ Normal) 0A 0B 02 (➙ Fault) On-device processing Fault detected Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) LPWA

Motor failure detection Dashboard There’s on-device “intelligence” to do complete analysis, right on the device. Then, only transmit the conclusion (just few bytes of data) to the cloud. Somehow… MicrocontrollerMotor That may save power, bandwidth, and deliver more complete analysis Processed data 0A 0B 01 (➙ Normal) 0A 0B 02 (➙ Fault) On-device processing Fault detected Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) LPWA

Motor failure detection Dashboard There’s on-device “intelligence” to do complete analysis, right on the device. Then, only transmit the conclusion (just few bytes of data) to the cloud. Back to… MicrocontrollerMotor Processed data 0A 0B 01 (➙ Normal) 0A 0B 02 (➙ Fault) On-device processing What is this? Machine Learning? Fault detected That may save power, bandwidth, and deliver more complete analysis Vibration raw data 5D 1B 16 56 01 41 … (1K Bytes) LPWA

Machine Learning is oKen associated with…

Machine Learning ClassiYcation What’s happening right now? Anomaly Detection Is it “normal”? Forecasting What will happen in the future? Some ML problems Credit: h_ps://www.slideshare.net/janjongboom/adding-intelligence-to-lorawan-devices-the-things-conference-2020

Machine Learning ClassiYcation What’s happening right now? Anomaly Detection Is it “normal”? Forecasting What will happen in the future? Some ML problems For those problems, ML Inference is possible on ultra-low power, low cost, tiny Microcontroller

High pelormance: Less latency as no raw data need to go to cloud/backend Still works omine: Machine Learning inference happens right on-device. No internet connectivity needed to get inference result from cloud. Encient power and bandwidth consumption: Only transmit necessary inference result, which are small in data size. Possible to use Low Power Wide Area (LPWA) connectivity, e.g: LoRa or NB-IoT Using optimised Edge hardware, even possible to be powered by ba_ery AI at the Edge Why the hassle?

So, how can we achieve TinyML?

ML-accelerated processor a class of microprocessor designed as hardware acceleration for AI applications, e.g. for neural networks, machine vision and machine learning Example: GPU FPGA ASIC DSP Development Board /Accessories a standalone computing system with ML- accelerated processor (main or co-) as a main or additional development unit Example: Coral DevBoard or USB Accelerator Intel Neural Compute Stick Sipeed Maix Arduino Nano 33 BLE Sense ESPectro32 SoKware Tools ML compilers ML Model design tool ML Model conveber SDK & libraries OS Example: TensorFlow & TensorFlow Lite (TFLite) TFLite for Microcontroller OpenVino Kendryte’s nncase EdgeImpulse Suppos Documentation Model Zoo Examples Datasheets Hardware SoKware What do we need?

Optimize & Conve (compress, remove, replace, conve) Train ML Model (on your awesome machine or Cloud) .h5 .pb .cabemodel ONNX ML Model Intermediate Representation (IR) File Transfer IR qle to Edge device Inference (Edge-optimized Inference library) Inference Result AI at the Edge General Pipeline

Optimize (TOCO) Train ML Model (on your awesome machine or Cloud) .h5 .pb TinyML Pipeline TFLite Flat Buber Conve (xxd) Integrate into MCU Firmware C Byte Array

Optimize (TOCO) Train ML Model (on your awesome machine or Cloud) .h5 .pb TinyML Pipeline TFLite Flat Buber Conve (xxd) Integrate into MCU Firmware C Byte Array Custom Logics ML Model Microcontroller (MCU) Firmware TensorFlow Lite for Microcontroller

TinyML Pipeline e.g. leveraging Azure Machine Learning Azure Machine Learning Pipeline TensorFlow model Yle Compress (toco)TFLite Flat Buber Conve (xxd) C Byte ArrayIntegrate into MCU Firmware TinyML Pipeline

Typical TinyML Model Training Labelled Raw Data Label1 Label2 Label3 Signal Processing (Low/High Pass Filter, FFT) Neural Network Output Features Credit: EdgeImpulse.com. Check it out! Processing Blocks To extract features Learning Blocks To classify new data. NN size can be signiqcantly small due to the extracted features from previous blocks AKer Filter Frequency Domain

using Arduino framework, TFLite, PlahormIO, ESPectro32 board Gesture Recognition DEMO

NB-IoT + GPS/GNSS Kit LoRa/LoRaWAN + GNSS Kit Sensors & Actuators Kit (20+ Peripherals) 3G + GNSS + Sensors ESPectro32 v2 Optimized for learning STEM, coding & electronics, also for fast prototyping. An AIoT Development Board: Low-power, dual-core 240 MHz Microcontroller Built-in WiFi & Bluetooth connectivity 7x7 LED matrix for user intelace Sensors: Accelerometer & Gyro using LSM6DS3, Digital temperature sensor, phototransistor microSD card slot Extensible via: 40 RPi-compatible pins, Grove connectors, Micro:bit connector Extensible with Backpacks: NB-IoT/Cellular + GPS/GNSS, LoRa, Sensor Kits, Motor Driver, and more ESPectro32 Backpacks hOps://shop.makestro.com/product/espectro32-v2/

git clone hOps://github.com/andriyadi/MagicWand-TFLite-ESP32

Train Model If necessary. Then compress and conveb the model…. Or use pre-trained model, instead Reference: h_ps://github.com/tensollow/tensollow/tree/master/tensollow/lite/micro/examples/magic_wand/train

Enjoy! Open Serial Monitor to see the inference result Recognised gestured also displayed on LED matrix* * On latest source code, display is already rotated :)

Other Demo Similar demo, but using Arduino Nano 33 BLE Sense board hOps://www.youtube.com/watch?v=Lfv3WJnYhX0 hOps://github.com/andriyadi/MagicWand-TFLite-Arduino

TinyML: TensorFlow Lite for Microcontrollers: h_ps://www.tensollow.org/lite/ microcontrollers Great book: TinyML, by Daniel Situnayake, Pete Warden Azure: Azure Machine Learning: h_ps://azure.microsoJ.com/en-in/services/ machine-learning-service/ DycodeX: ESPectro32 dev board: h_ps://shop.makestro.com/product/espectro32- v2/ SMARTernak: h_ps://smabernak.com Other IoT + AI products & solutions: h_ps://dycodex.com Contact me: andri (at) dycodex.com h_p://github.com/andriyadi | hOps://slideshare.net/andri_yadi/ Call to Action

Want to put “AI” in “BrAIns”? … a.k.a. adopting AI + IoT -> AIoT?

AI + IoT enabler Keep in touch hi (at) dycodex.com | https://dycodex.com Bandung, Indonesia

Introduction to AIoT & TinyML - with Arduino

More Related Content

What's hot

Similar to Introduction to AIoT & TinyML - with Arduino

More from Andri Yadi

Recently uploaded

In this document

Introduction to AIoT & TinyML - with Arduino