![Event Streaming with Confluent and IBM Cloud Pak “… rescue data off of the mainframe, in a cloud native, microservice- based fashion … [to] … significantly reduce the reads on the mainframe, saving RBC fixed infrastructure costs (OPEX). RBC stayed compliant with bank regulations and business logic, and is now able to create new applications using the same event-based architecture.” Mainframe Offloading for massive cost-savings https://www.confluent.io/customers/rbc/](https://image.slidesharecdn.com/202106ibmconfluentwebinar-210617123010/75/IBM-Cloud-Pak-for-Integration-with-Confluent-Platform-powered-by-Apache-Kafka-19-2048.jpg)
Uploaded byKai Wähner
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
The document discusses the collaboration between IBM and Confluent in advancing event streaming technology, emphasizing its importance for real-time data processing across various industries. It highlights the capabilities of the IBM Cloud Pak for Integration, featuring extensive integration options and event streaming functionality powered by Apache Kafka. Several real-world use cases are presented to illustrate the transformative potential of event streaming in enhancing operational efficiency and enabling real-time decision-making.
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
- 1. The Rise ofData in Motion powered by Event Streaming Use Cases and Architecture for IBM Cloud Pak with Confluent Platform Kai Waehner Field CTO contact@kai-waehner.de linkedin.com/in/kaiwaehner @KaiWaehner www.confluent.io www.kai-waehner.de Gianluca Natali Senior Partner Solutions Engineer EMEA gnatali@confluent.io linkedin.com/in/gianlucanatali/
- 2. Event Streaming withConfluent and IBM Cloud Pak Agenda • IBM & Confluent – Strategic Partnership • Data in Motion powered by Event Streaming • Real World Deployments across Industries • Architectures for Edge, On-Premise, Hybrid Cloud • Live Demo
- 3. Event Streaming withConfluent and IBM Cloud Pak Agenda • IBM & Confluent – Strategic Partnership • Data in Motion powered by Event Streaming • Real World Deployments across Industries • Architectures for Edge, On-Premise, Hybrid Cloud • Live Demo
- 4. Event Streaming withConfluent and IBM Cloud Pak
- 5. Event Streaming withConfluent and IBM Cloud Pak IBM Cloud Pak for Integration with Confluent IBM Integration / © 2020 IBM Corporation 5 Cloud Pak for Integration Broadest integration capabilities Unified experience, operational efficiency & reuse – Deploy where needed Container-based architecture with common enterprise services – Enterprise-grade Secure, scalable, resilient Market-leading event streaming capabilities Access to over 100 connectors, ksqlDB, Tiered Storage, and more – Original Kafka Creators Confluent team wrote 80% of Kafka commits and has 1M hours of technical experience with Kafka – Enterprise-grade Deploy at scale with enterprise-grade security The market’s broadest set of integration capabilities, now with the industry’s leading event streaming platform
- 6. Event Streaming withConfluent and IBM Cloud Pak IBM Cloud Pak for Integration with Confluent API Lifecycle Management Application & Data Integration Enterprise Messaging Event Streaming High Speed Data Transfer Secure Gateway Governance Asset Sharing Unified User Experience IBM containerized software Container platform and operational services 6 Cloud Hybrid On-premises Cloud Pak for Integration IBM Integration / © 2020 IBM Corporation
- 7. Event Streaming withConfluent and IBM Cloud Pak Agenda • IBM & Confluent – Strategic Partnership • Data in Motion powered by Event Streaming • Real World Deployments across Industries • Architectures for Edge, On-Premise, Hybrid Cloud • Live Demo
- 8. Event Streaming withConfluent and IBM Cloud Pak Real-time Data beats Slow Data. Transportation Real-time sensor diagnostics Driver-rider match ETA updates Banking Fraud detection Trading, risk systems Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-app purchases
- 9. Event Streaming withConfluent and IBM Cloud Pak This is a fundamental paradigm shift... 10 Infrastructure as code Data as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
- 10. Event Streaming withConfluent and IBM Cloud Pak Apache Kafka is a Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 11
- 11. Event Streaming withConfluent and IBM Cloud Pak Databases Messaging ETL / Data Integration Data Warehouse Why can’t I do this with my existing data platforms?
- 12. Event Streaming withConfluent and IBM Cloud Pak Enterprise Data Platform Requirements Are Shifting 1 3 4 2 Scalable for Transactional Data Transient Raw data Built for Historical Data Built for Real- Time Events Scalable for ALL data Persistent + Durable Enriched data ● Value: Trigger real- time workflows (i.e. real-time order management) ● Value: Scale across the enterprise (i.e. customer 360) ● Value: Build mission-critical apps with zero data loss (i.e. instant payments) ● Value: Add context & situational awareness (i.e. ride sharing ETA) 13
- 13. Event Streaming withConfluent and IBM Cloud Pak Only Event Streaming Has All 4 Requirements 14
- 14. Event Streaming withConfluent and IBM Cloud Pak Messaging Databases Event Streaming Data Warehouse BUILT FOR REAL- TIME EVENTS SCALABLE FOR ALL DATA PERSISTENT & DURABLE CAPABLE OF ENRICHMENT 15 Good for transactional applications Good for ultra low-latency, fire-and-forget use cases Good for batch data integration Good for historical analytics and reporting Platform for Event-Driven Transformation (Scalable Messaging + Real-Time Data Integration + Stream Processing) ETL/Data Integration Only Event Streaming Has All 4 Requirements
- 15. Event Streaming withConfluent and IBM Cloud Pak Event Topics Storage Partitions Events / sec Kafka Servers 10,000,000 25,000 1,000,000 1,500 Event Topics Storage Partitions Events / sec Kafka Servers 250,000 500 25,000 25 Event Topics Storage Partitions Events / sec Kafka Servers 100 5 300 3 Kafka Scales with Your Business.
- 16. Event Streaming withConfluent and IBM Cloud Pak Kafka Connect + IBM MQ Connector Kafka Cluster MQ Integration Domain-Driven Design for your Integration Layer Mainframe Integration Custom Application IBM IIDR Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform Customer Domain Payment Domain Fraud Domain
- 17. Event Streaming withConfluent and IBM Cloud Pak Agenda • IBM & Confluent – Strategic Partnership • Data in Motion powered by Event Streaming • Real World Deployments across Industries • Architectures for Edge, On-Premise, Hybrid Cloud • Live Demo
- 18. Event Streaming withConfluent and IBM Cloud Pak Every Industry is Moving from Batch/Manual to Software-Defined 19 Auto / Transport Software-using Software-defined Spreadsheet-driven driver schedule Real-time ETA Banking Nightly credit-card fraud checks Real-time credit card fraud prevention Retail Batch inventory updates Real-time inventory management Healthcare Batch claims processing Real-time claims processing Oil and Gas Batch analytics Real-time analytics Manufacturing Scheduled equipment maintenance Automated, predictive maintenance Defense Reactive cyber-security forensics Automated SIEM and Anomaly Detection U.S. Defense Agencies
- 19. Event Streaming withConfluent and IBM Cloud Pak “… rescue data off of the mainframe, in a cloud native, microservice- based fashion … [to] … significantly reduce the reads on the mainframe, saving RBC fixed infrastructure costs (OPEX). RBC stayed compliant with bank regulations and business logic, and is now able to create new applications using the same event-based architecture.” Mainframe Offloading for massive cost-savings https://www.confluent.io/customers/rbc/
- 20. Event Streaming withConfluent and IBM Cloud Pak “We look at events as running our business. Business people within our organization want to be able to react to events—and oftentimes it's a combination of events.” VP of Streaming Data Engineering
- 21. Event Streaming withConfluent and IBM Cloud Pak ● Capital One “Second Look” ● Customers do not check statements regularly ● Duplicate charges, high tips, increased recurring charges go unnoticed ● The right level of signal vs noise for the consumer ● Preventing $150 of fraud on average a year/customer Use Cases: -Customer 360 -Customer Notifications & Alerts -Fraud Detection
- 22. Event Streaming withConfluent and IBM Cloud Pak 10X Banking Cloud-native Core Banking Platform https://www.confluent.io/customers/rbc/ https://videos.confluent.io/watch/ip69LsbDX8JpERSP1soNXo
- 23. Event Streaming withConfluent and IBM Cloud Pak Smart City Digital Buildings, Connected Vehicles and Things, Citizens Real-time Operations, Logistics, Predictive Maintenance, Security Traffic Data Vehicles, Accidents, Traffic Lights, etc. Surveillance Data Buildings, People, Distance Measuring, etc. Mobility Services Multi-modal integration, E-car Charging Stations, Parking Lots, etc. Drone Data Deliveries, Survey/Inspection of Traffic, People, Construction Areas etc. Edge Analytics Bidirectional Edge to Cloud Integration Data Ingestion Stream Processing Data Integration Logistics Track&Trace Routing Monitoring Alerting Command&Control Batch Analytics Reporting Machine Learning Backend Systems Oracle, SAP, OSIsoft PI, etc. X = Event Streaming X = Other Technologies Bi-Directional Hybrid Cloud Replication
- 24. DB Musterfirma |Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile") 25 Customer timetable Operational timetable Assignments Railway station knowledge Dispositions Train positions Matching Aggregation Consolidation Apache Kafka Analysis Railway station Trains Mobile Apps Employees Deutsche Bahn AG | Reisendeninformation Deutsche Bahn - Reisendeninformation
- 25. Event Streaming withConfluent and IBM Cloud Pak NAV (Norwegian Work and Welfare Department): Life is a Stream of Events https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/ Assist people through all phases of life within the domains of work, family, health, retirement and social security 26
- 26. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Severstal Predictive Maintenance and Quality Assurance at the Shop Floor Real Time Streaming Machine Learning with Kafka https://www.confluent.io/customers/severstal/
- 27. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
- 28. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Devon Energy Oil & Gas Industry Improve drilling and well completion operations Edge stream processing/analytics + closed-loop control ready Vendor agnostic (pumping, wireline, coil, offset wells, drilling operations, producing wells) Replication to the cloud in real-time at scale Cloud agnostic (AWS, GCP, Azure) Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
- 29. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Agenda • IBM & Confluent – Strategic Partnership • Data in Motion powered by Event Streaming • Real World Deployments across Industries • Architectures for Edge, On-Premise, Hybrid Cloud • Live Demo
- 30. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Kafka is Complementary to other Middleware in the Enterprise Architecture Orders Customers Payments Stock REST JMS ESB REST CRM Mainframe SOAP … Kafka Kafka Kafka Kafka SOAP API Management HTTP MQ
- 31. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Global Event Streaming Streaming Replication between Kafka Clusters Bridge to Databases, Data Lakes, Apps, APIs, SaaS Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters with RPO=0 and RTO~0 Stream Data Globally with Replication and Cluster Linking 32
- 32. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de CRM 3rd party payment provider Real-Time Asset Management Customer data Payment processing and fraud detection as a service Manager Outage Management API Customer Customer Customer data Truck schedule Payment data Route details Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Energy Production and Distribution with a Hybrid Architecture Cloud Edge Data Center Smart Meters Smart Buildings
- 33. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Smart Grid Upstream Operations Management Well Completion Operations Customer data Truck schedule Payment data Route details Streams of real time events Real-Time Supply Chain Management Event Streaming for Energy Production (Upstream + Midstream) at the Edge with a 5G Campus Network Cloud Edge Data Center Smart Home
- 34. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Energy Production at the Disconnected Edge (Upstream) Time P C1 C2 C3 Predictive Analytics Human Machine Interface Predictive Maintenance Sensors Always on (even “offline”) Replayability Reduced traffic cost Better latency
- 35. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Edge Processing at the Intelligent Gas Station (Downstream) Time P C3 C1 C2 Upsell in the store Context-specific coupon (e.g. carwash not in use à 30% off)) Real-time inventory and intelligent pricing
- 36. Event Streaming withConfluent and IBM Cloud Pak Event Streaming Is The Future Of Data 37 Infrastructure as code Data as continuous streams of events Future of the datacenter Future of data Cloud Event Streaming
- 37. Event Streaming withConfluent and IBM Cloud Pak Live Demo
- 38. Event Streaming withConfluent and IBM Cloud Pak An example 200+ connectors Streaming ETL
- 39. Event Streaming withConfluent and IBM Cloud Pak IBM MQ Source 40
- 40. Event Streaming withConfluent and IBM Cloud Pak IBM MQ Source (continued) 41
- 41. Event Streaming withConfluent and IBM Cloud Pak Create new Stream from Select (Join) 42
- 42. Event Streaming withConfluent and IBM Cloud Pak ksqlDB Flow 43
- 43. Event Streaming withConfluent and IBM Cloud Pak Joined Data streaming in Real Time 44
- 44. Event Streaming withConfluent and IBM Cloud Pak IBM MQ Sink 45
- 45. Event Streaming withConfluent and IBM Cloud Pak IBM MQ Sink (continued) 46
- 46. Event Streaming withConfluent and IBM Cloud Pak Enriched Data arriving in IBM MQ 47
- 47. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Why Confluent?
- 48. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de I N V E S T M E N T & T I M E V A L U E 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management... 49
- 49. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de The Rise of Data in Motion 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka 50
- 50. IoT and EventStreaming with Apache Kafka in the Energy Industry – @KaiWaehner - www.kai-waehner.de Confluent... Complete. Cloud-native. Everywhere. Freedom of Choice Committer-driven Expertise Open Source | Community licensed Fully Managed Cloud Service Self-managed Software Training Partners Enterprise Support Professional Services ARCHITECT OPERATOR DEVELOPER EXECUTIVE Apache Kafka Dynamic Performance & Elasticity Self-Balancing Clusters | Tiered Storage Flexible DevOps Automation Operator | Ansible GUI-driven Mgmt & Monitoring Control Center | Proactive Support Event Streaming Database ksqlDB Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development Non-Java Clients | REST Proxy Admin REST APIs Global Resilience Multi-Region Clusters | Replicator Cluster Linking Data Compatibility Schema Registry | Schema Validation Enterprise-grade Security RBAC | Secrets | Audit Logs TCO / ROI Revenue / Cost / Risk Impact Complete Engagement Model Efficient Operations at Scale Unrestricted Developer Productivity Production-stage Prerequisites Partnership for Business Success
- 51. Kai Waehner Field CTO contact@kai-waehner.de @KaiWaehner www.kai-waehner.de www.confluent.io linkedin.com/in/kaiwaehner Questions?Feedback? Let’s connect! Gianluca Natali Senior Partner Solutions Engineer EMEA gnatali@confluent.io linkedin.com/in/gianlucanatali/