Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo

Session 7 - Creating a Context-Aware Solution: Complex Event Processing with Perseo Fernando López, Cloud & Platform Senior Expert fernando.lopez@fiware.org @flopezaguilar FIWARE Foundation, e.V.

Learning Goals 1 ● What is a Complex Event Processing? ● How to connect Perseo with Orion Context Broker? ● How can I deﬁne an event with EPL? ● What are the available acBons?

Overview 2 ▪ Context-aware CEP GE • Listen to events from context information to identify patterns described by rules, and immediately react by triggering actions. • Designed to be NGSI-compliant, and thus, to implement systems that works seamless and jointly with Orion CB. □ Uses NGSI as communication protocol for events □ Follows a PUSH model • Based on Esper CEP ▪ It is focused on Complex Event Processing, not on Stream Processing. • It defines a rule (query) engine, not a graph engine. • Rules are set using SQL-like queries (EPL: Event Processing Language)

Introduction to Complex Event Processing 3 ▪ Requirement to process events (or messages) in real-time or near real-time. ▪ Key considerations for these types of applications are throughput, latency and the complexity of the logic required: • High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second) • Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds) • Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event series, trigger based on absence of events etc.

Introduction to the Esper architecture ▪ Esper is a language, a language compiler and a runtime environment. ▪ Language • The Esper language is the Event Processing Language (EPL). • It is a declarative, data-oriented language for dealing with high frequency time-based event data. • EPL is compliant to the SQL-92 standard and extended for analyzing series of events and in respect to time. ▪ Language Compiler • Compiles EPL source code into Java Virtual Machine (JVM) bytecode. • The resulting executable code runs on a JVM within the Esper runtime environment. 4

Introduction to the Esper architecture ▪ Runtime • The Esper runtime runs on top of a JVM. • You can run byte code produced by the Esper compiler using the Esper runtime. ▪ The Esper architecture is similar to Scala, Clojure and Kotlin. 5

Overview: CEP in FIWARE with Perseo 6 PERSEO Context-Aware CEP ACTIONS RULES ORION Context Broker EVENTS NGSI subscription NGSI Context notification Pattern matching REST API ACTION: NGSI Update Context ACTION: HTTP ACTION: SMS ACTION: EMAIL ACTION: TWEET Perseo-fe Perseo-core Events Alarms Actuate Sensors SMART IoT PLATFORM

▪ Repositories • Perseo Core (https://github.com/telefonicaid/perseo-core) • Perseo Front-End (https://github.com/telefonicaid/perseo-fe) ▪ Docker images • Perseo Core (https://hub.docker.com/r/fiware/perseo-core/) • Perseo Front-End (https://hub.docker.com/r/fiware/perseo/) Overview: CEP in FIWARE with Perseo 7

Notice vs. Event ▪ Notifications or notices from Context Broker are processed by Perseo (perseo-fe) before being sent to its core (perseo-core + Esper) as events. ▪ They correspond to changes in the values of an attribute of an entity to which Perseo is subscribed to. { "subscriptionId":"5bfaf6ab27124e3d9b6c0b20", "data":[{ "id":"streetlight:gothenburg:3", "type":"Streetlight", "status":{ "type":"Text", "value":"ok", "metadata":{} }, "dateLastLampChange":{ "type":"DateTime", "value":"2018-11-20T11:00:00.00Z", "metadata":{} }, "address":{ "type":"Address", "value":{ "addressLocality":"Gothenburg", "addressCountry":"Sweden", "postalCode":"425 30", "streetAddress":"Norra Ligården 1” }, "metadata":{} }, "areaServed":{ "type":"Text", "value":"Karra-Sodra", "metadata":{} }, "illuminanceLevel":{ "type":”Number", "value":”95", "metadata":{} }, "powerState":{ "type":"Text", "value":"off", "metadata":{} } }], "subservice":"/demo/fiwaresummit19", "service":”carthage” } Notice 8

Notice vs. Event ▪ Notifications or notices from Context Broker are processed by Perseo (perseo-fe) before being sent to its core (perseo-core + Esper) as events. ▪ They correspond to changes in the values of an attribute of an entity to which Perseo is subscribed to. { "noticeId":"710e8dd0-f15a-11e8-90d5-7fa905f73247", "noticeTS":1543223138094, "id":"streetlight:gothenburg:3", "type":"Streetlight", "isPattern":false, "subservice":"/demo/fiwaresummit19", "service":"carthage", "status__type":"Text", "status":"ok", "address__type":"Address", "address__addressLocality":"Gothenburg", "address__addressCountry":"Sweden", "address__postalCode":"425 30", "address__streetAddress":"Norra Ligården 1", "areaServed__type":"Text", "areaServed":"Karra-Sodra", "illuminanceLevel __type":"Number", "illuminanceLevel": 95 "powerState__type":"Text", "powerState":"off” } Event Restriction: ▪ An attribute cannot be named id or type 9

Events, datatypes and NGSIv2 ▪ Perseo flattens the data structures to access the internal values of each data type, and add the next parsed values in the event. This attributes can be used in the rules individually. "location__type":"geo:point", "location__lat":12.0011546, "location__lon":57.7934324, "location__x":586374.5004314554, "location__y":1326805.6699024632, Location (geo:point, geo:line, geo:box, geo:polygon and geo:json): "location": { "type": "geo:point", "value": "12.0011546, 57.7934324", "metadata": {} } "dateLastLampChange__type": "DateTime", "dateLastLampChange__iso": 2017-09-10T11:00:00.00Z, "dateLastLampChange__ts": 1505041200000, "dateLastLampChange__day": 10, "dateLastLampChange__month": 9, "dateLastLampChange__year": 2017, "dateLastLampChange__hour": 13, "dateLastLampChange__minute": 0, "dateLastLampChange__second": 0, "dateLastLampChange__millisecond": 0, "dateLastLampChange__dayUTC": 10, "dateLastLampChange__monthUTC": 9, "dateLastLampChange__yearUTC": 2017, "dateLastLampChange__hourUTC": 11, "dateLastLampChange__minuteUTC": 0, "dateLastLampChange__secondUTC": 0, "dateLastLampChange__millisecondUTC": 0, DateTime: "dateLastLampChange": { "type": "DateTime", "value": "2017-09-10T11:00:00.00Z", "metadata": {} } 10

Rules { "name":“streetlight_column_alarm", "text":"select ev.id? as id, ev.status? as laststatus, ev.dateLastLampChange__iso? as lastchange pattern [every ev=iotEvent(type="Streetlight" and ev.status? = ‘columnIssue’)]", "action”:{ "type":"update", "parameters":{ "id":"${id}_alarm_column", "type”:"StreetlightAlarm", "version":"2", "attributes": [ { "name":"status", "type":"Text", "value":"${laststatus}" }, { "name":”lastChange", "type":"DateTime", "value":"${lastchange}" } ] } } Rule example. (create/update a StreetlightAlarm entity when a problem is detected in the column/pole) ▪ Perseo rules have three mandatory sections: name, text and action 11

Rules: names ▪ Simple description of the rule. 12 "name":“streetlight_column_alarm” Name

Rules: text ▪ Based on Esper EPL SQL-like rules ▪ Pattern matching: • from pattern • match_recognize ▪ Event stream called iotEvent and instanciated as ev ▪ We can use predefined functions, such as cast(), avg(), etc… ▪ It is no longer necessary to add the RuleName in the "text" field SELECT ev.id? as id, ev.status? as laststatus, ev.areaServed? as areaServed FROM pattern [every ev=iotEvent(type=‘Streetlight’ and ev.status? != ‘ok’)] EPL using paGern clause SELECT * FROM iotEvent(type=‘Streetlight’).win:time(1 minute) MATCH_RECOGNIZE ( measures A.status? as status pattern (A B) define( A as A.status? != ‘ok’, B as B.status? != ‘ok’ and B.id? != A.id) EPL using match_recognize 13

Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action":”:{ "type":"update", "parameters":{ "id":"${id}_alarm_column", "type”:"StreetlightAlarm", "version":"2", "attributes": [ { "name":"status", "type":"Text", "value":"${laststatus}" }, { "name":”lastChange", "type":"DateTime", "value":"${lastchange}" } ] } Update CB entity action "action": { "type": "sms", "template": "${id} status has change to ${laststatus}", "parameters": { "to": "123456789" }} SMS action 14

Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action": { "type": "email", "template": "${id} status has change to ${laststatus}”, "parameters": { "to": "someone@fiware.org", "from": "cep@system.org", "subject": ”Problems with StreetLamp ${id}. The status has change to ${laststatus}" } } Email action 15

Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action":{ "type": "post", "template": "${id} status has change to ${laststatus}", "parameters":{ "url": "http://localhost:9182/${type}/${id}", "method": "PUT", "headers": { "Content-type": "text/plain", "X-${type}-status": "${laststatus}" }, "qs": { ”status": "${laststatus}", } } } HTTP acLon 16

Rules: actions Rule actions ▪ Options: • Update context broker entity • SMS • Email • HTTP invocation • Send a tweet ▪ We can use placeholders, ${X}, to insert data from the events "action": { "type": "twitter", "template": "${id} status has change to ${laststatus}" "parameters": { "consumer_key": "xvz1evFS4wEEPTGEFPHBog", "consumer_secret": "L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg", "access_token_key": "xvz1evFS4wEEPTGEFPHBog", "access_token_secret": "L8qq9PZyRg6ieKGEKhZolGC0vJWLw8iEJ88DRdyOg" } } Tweet acLon 17

Timed Rules { "name":”10_sec_timer_rule", "text":"select * from pattern [every timer:interval(10 sec)]", "action":{...} } Rule example: fires each 10 seconds { "name":”2_week_timer_rule", "text":"select * from pattern [every timer:interval(2 weeks)]", "action":{...} } ▪ timer:interval(<number|variable_name> <time-period>) Rule example: fires each 2 weeks Time Periods: • " years" | "year" • " months" | "month “ • " weeks" | "week“ • "days" | "day“ • "hours" | "hour“ • "minutes" | "minute" | "min“ • "seconds" | "second" | "sec“ • "milliseconds" | "millisecond" | "msec" 18

Timed Rules ▪ timer:at (minutes, hours, days of month, months, days of week [, seconds [, time zone]]). { "name":”5_min_past_hour", "text":"select * from pattern [every timer:at(5, *, *, *, *)]", "action":{...} } Rule example: fires once an hour, at minute five { "name":”17_PST", "text":"select * from pattern [timer:at (0, 17, *, *, *, *, 'PST')]", "action":{...} } Rule example: only fires 1 time at 5:00 pm Pacific Standard Time 19

Timed Rules ▪ timer:schedule(repetitions, date, period) { "name":"specific_date", "text":"select * from pattern[every timer:schedule(repetitions: 2, date: '2018-03-01T13:00:00Z', period: 1 year 2 month 10 days 2 hours 30 minutes)]", "action":{...} } Rule example: fires 2 times, first callback (2018-03-01 at 13:00:00 UTC) second callback (2019-05-11 at 15:30:00 UTC) { "name":"next_minute", "text":"select * from pattern[every timer:schedule(date: '2019-10-01T05:52:00Z')]", "action":{...} } Rule example: fires on 2019-10-01T05:52:00Z Note: There are many other kind of timed Esper rules, such as time_batch , ext_timed_batch, match_recognize intervals, etc. 20

Summary: Terms 21 ● CEP, Complex Event processing is event processing that combines data from mulBple sources to infer events or paYerns that suggest more complicated circumstances. (source: Wikipedia). ● Esper is an event stream processor that aims to enable a short development cycle from incepBon to producBon for these types of applicaBons. (source: EsperTech). ● EPL, Event Processing Language (EPL) deﬁned to be the Esper language and designed for Complex Event Processing and Streaming AnalyBcs. (source: EsperTech).

References 22 ● FIWARE Catalogue o https://www.fiware.org/developers/catalogue ● FIWARE Academy o https://fiware-academy.readthedocs.io/en/latest/index.html ● Perseo documentation o https://perseo.readthedocs.io/en/latest/

References 23 ● GitHub repos o Perseo Core (https://github.com/telefonicaid/perseo-core) o Perseo Front-End (https://github.com/telefonicaid/perseo-fe) ● Docker images o Perseo Core (https://hub.docker.com/r/fiware/perseo-core) o Perseo Front-End (https://hub.docker.com/r/fiware/perseo) ● Esper Doc o http://esper.espertech.com/release-7.1.0/esper-reference/html/index.html

Question & Answer 24 fiware-tech-help@lists.fiware.org

Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo

More Related Content

What's hot

Similar to Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo

More from Fernando Lopez Aguilar

Recently uploaded

Creating a Context-Aware solution, Complex Event Processing with FIWARE Perseo