Downloaded 15 times
Uploaded byconfluent
Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka - Introduction to AiB's KafkaBlockchain
The document discusses the development of a secure, tamper-proof, and scalable blockchain solution built on Apache Kafka, focusing on various use cases, including healthcare, finance, and supply chain management. It emphasizes the integration of artificial intelligence to enhance data security and immutability, alongside the benefits of event streaming technology. Additionally, it showcases the potential of aib's patented private-permissioned blockchain and its applications within enterprise infrastructures.
Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka - Introduction to AiB's KafkaBlockchain
- 1. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Kai Waehner Technology Evangelist Confluent kai.waehner@confluent.io LinkedIn @KaiWaehner confluent.io kai-waehner.de Stephen Reed CEO, Co-founder AiB stephen@ai-blockchain.com LinkedIn ai-blockchain.com
- 2. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka 6. Live Demo Agenda
- 3. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka PRODUCTS Built FIRST DocuSign Blockchain product. Allows for the writing of contracts on the blockchain with proof of trust Software agents keep the blockchain tamper proof and immutable Current POC’s across eCommerce, Healthcare and Supply Chain SOLUTIONS • Artificial Intelligence, Blockchain, Ecommerce, Healthcare, Media, Payments and Supply Chain • Securely store commercial and non-commercial transactions • Energy efficient and ESG GROWTH OPPORTUNITIES How do we expand globally with new customers using AiB products? AiB offers services for scalability to meet all of the demands of global businesses offering architecture assessment diagrams to clients. TECHNOLOGY We employ one of a kind Artificial Agents on the Blockchain to give world class security and to ensure all data is immutable. PLATFORM Its most visible success is in the unique approach to financial transactions through its implementation of its Artificial Intelligent Blockchain Cloud Payment Platform. Enterprise Grade Platform COMPANY Since its founding in 2014, Ai-Blockchain has built a combination of innovative IP and blockchain products that have been recognized in the marketplace and USPTO. Potential Customer Use Cases and Problems Ai-Blockchain Products Ai-Blockchain Solutions and advantages Industry Expansion Ai-Blockchain – Patented Private-Permissioned Blockchain for Smart Contracts
- 4. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 4 CUSTOMER CLIENT EXPERIENCE WITH AiB: ● Immediate go-to-market product execution using blockchain ● Artificial Intelligence and Machine Learning recommendation systems Ai-Blockchain – Patented Private-Permissioned Blockchain for Smart Contracts
- 5. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Ai-Blockchain – Co-Founders 6
- 6. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain for Supply Chain Management (Manufacturing, Logistics, …) 7 https://coletiv.com/blog/blockchain-supply-chain-management-system/ https://www.disruptordaily.com/blockchain-use-cases-supply-chain-management/
- 7. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain in Healthcare 8 • Use Cases • Management of patient data, especially access control • RVU Agreements with Ai-Blockchain • Management and use of academic research data • Logistics / Supply chain for drugs, medical technology, and other products • Often no need for Blockchain / DLT • Only for Identity Management • Processing and (big) data storage outside blockchain https://digiforest.io/en/blog/blockchain-examples-in-healthcare
- 8. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain for Financial Services 9 https://www.accenture.com/t20170120T074124Z__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/Consulting/Accenture-Banking-on-Blockchain.pdf%23zoom=50
- 9. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka by AiB 6. Live Demo Agenda
- 10. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Bitcoin – The Tip of the Iceberg! 11 ‘Bitcoin’ compared to ‘blockchain’, ‘distributed ledger’, etc.
- 11. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain vs. Distributed Ledger Technology (DLT) 12 https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology
- 12. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain 14 https://www.investopedia.com/terms/b/blockchain.asp
- 13. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Blockchain Concepts • Permissioned or permissionless (or consortium) • Consensus algorithm (Proof of work, Proof of Stake, or Voting systems) • Blockchain: Global Consensus across all nodes • DLT: Consensus without having to validate across the entire Blockchain • Blockchain • Growing list of records, called blocks, linked using cryptography • Each block contains a cryptographic hash of the previous block, a timestamp, and transaction data 16
- 14. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Challenges and Concerns of Blockchains • Technical and organizational Complexity • Transaction speed • Energy consumption • Security • Data tenancy • Lifecycle costs • Ease of integration
- 15. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Purpose-built solutions • Ripple: Payment settling, currency exchange and remittance system • Libra: Global payment system and financial infrastructure • IOTA: IoT transactions Non-Blockchain DLTs to solve the challenges to provide better performance and scalability 18 Open and flexible solution for many use cases • Apache Kafka: Open, scalable, real time event streaming Technology Partners • Ai-Blockchain: “AiB” USA based
- 16. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka 6. Live Demo Agenda
- 17. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Event Streaming in the Finance Industry Check past Kafka Summit videos for details about the use cases: https://kafka-summit.org/past-events/
- 18. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Tamper-Proof Event Streaming for (some of the) Use Cases in the Finance Industry Real Time Processing Digital Transformation Strategic Goals Short-Sale Risk Calculation / Trade Approval Mainframe Offloading and Replacement Instant Payment Next-Best Offer Robot Process Automation (e.g. Know Your Customer, KYC) Customer Service (e.g. Chat Bots) IT ModernizationRegulatory Reporting Fraud Detection Anomaly Detection Across Assets and Locations Derivatives Pricing Compliance Trading Post-Processin g Strategic Planning and Simulations
- 19. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC Event Streaming
- 20. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Event Streaming Platform – The Immutable Commit Log Time P C1 C2 C3
- 21. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Event Streaming Platform – A Distributed System for 24/7 and Zero Data Loss Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 22. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka An Event Streaming Platform is the Underpinning of an Event-driven Architecture Microservices Mainframes SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experiences Streams of real time events Stream processing apps Connectors Connectors Stream processing apps
- 23. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Kafka Connect with Ethereum Connector Kafka Cluster Blockchain Integration Domain-Driven Design and Decoupled Applications Legacy Integration Custom Application ETL Tool with Mainframe Connector Java / C++ / Go / Python / KSQL Schema Registry Event Streaming Platform Blockchain Domain Payment Domain Fraud Domain Audit Logs, RBAC, etc.
- 24. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Reasons for Event Streaming with Apache Kafka Real Time Scalable High Throughput Cost Reduction 24/7 – Zero downtime, zero data loss Decoupling – Storage, Domain-driven Design Data (re-)processing and stateful client applications Integration – Connectivity to IoT, legacy, big data, everything Hybrid Architecture – On Premises, multi cloud, edge computing Fully managed cloud No vendor locking 29
- 25. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Kafka Clusters can stretch over Regions Zero Downtime + Zero Data loss (RPO=0 and RTO=0) e.g. Stretched over US East + Mid + West Automate Disaster Recovery Sync or Async Replication per Topic Offset Preserving Automated Client Failover without Custom Code Multi-Region Cluster (Only available in Confluent Platform) 31
- 26. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Example of a Multi-Region Cluster in a Bank Large FinServ Customer Payment Log Payment Log Location Location synchronous asynchronous ● ‘Payment’ transactions enter from us-east and us-west with fully synchronous replication ● ‘Log’ and ‘Location’ information in the same cluster use async - optimized for latency ● Automated disaster recovery (zero downtime, zero data loss) Result: Clearing time from ‘deposit’ to ‘available’ goes from 5 days to 5 seconds (including security checks) (Only available in Confluent Platform)
- 27. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka 6. Live Demo Agenda
- 28. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Real Time High Throughput Decentralized database Distributed log of records Immutable log Replication High availability Decoupling of applications / clients Role-based access control to data Tamper-Proof Encrypted payloads is not a Blockchain! X
- 29. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Kafka AND Blockchain 37 Kafka Real Time Instant Payment App (Java, C++, Python, etc.) Batch Analytics Platform (Spark, Splunk, etc.) Blockchain (Bitcoin) Blockchain (Ethereum) Kafka-native Blockchain Kafka AS Blockchain Real Time Instant Payment App (Java, C++, Python, etc.) Batch Analytics Platform (Spark, Splunk, etc.)
- 30. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka 6. Live Demo Agenda
- 31. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
- 32. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka AiB’s BRICK Messaging Platform For security, privacy and scalability, the AiB’s BRICK platform is a platform agnostic layer that is built for enterprise and micro payments in the cloud. Third-Party Audited AiB’s technology stack has been security audited and stress tested by third-party global cloud providers. Full-Stack Capability Deploying nodes and writing smart contracts up to front end development and deployment. Thought Leaders Consistently asked to speak at international events to share our Blockchain expertise. Blockchain Experts Established in 2014 Ai Blockchain’s team of over 100 Blockchain developers & experts have been building solutions for over six years and are ready today for any and all market opportunities. Ai-Blockchain – Proprietary Blockchain Technology Stack
- 33. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 41 KafkaBlockchain Architecture Producers reuse the KafkaBlockchain library and Tamper-Evident Records. Consumers reuse the KafkaBlockchain library and Tamper-Evident Records.
- 34. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka KafkaBlockchain – Producer, Consumer & Verifier Open Source Apache Licensed on GitHub
- 35. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 43
- 36. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka 1. Use Cases for Secure and Tamper-Proof Data Processing 2. Distributed Ledger and Blockchain Technologies 3. Event Streaming with Apache Kafka 4. Relation between Kafka and Blockchain 5. A Tamper-Proof Blockchain implemented with Kafka 6. Live Demo Agenda
- 37. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Demonstration: run-kafka-blockchain.demo.sh • Demonstrate putting messages into a Kafka blockchain with one partition, one producer and one consumer. • Describe the single blockchain partition • Demonstrate verifying messages with one consumer Demonstration: run-kafka-blockchain-multiple-partition.demo.sh • Demonstrate putting messages into a Kafka blockchain with five partitions, one producer and one consumer. • Describe the five blockchain partitions • Demonstrate verifying messages with five cooperating consumers - one for each partition. 45 KafkaBlockchain – Live Demo
- 38. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka When to use Kafka vs. a “real blockchain framework”? 46 Use Apache Kafka with AIBlockchain for • Enterprise infrastructure • Open, scalable, real-time requirements • Flexible architectures for many use cases Use Hyperledger, Ethereum, et al. for • Deployment over various independent organizations • Participants verify the distributed ledger contents themselves. • Specific use cases • Server-side managed and controlled by multiple organizations • Scenarios where the business value overturns the added complexity and project risk Use Kafka and Blockchain together to combine the benefits of both (only where this makes sense!)
- 39. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka What about long-term storage in Kafka? 48
- 40. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Today, Kafka works well for recent events, short horizon storage, and manual data balancing Kafka’s present-day design offers extraordinarily low messaging latency by storing topic data on fast disks that are collocated with brokers. This is usually good. But sometimes, you need to store a huge amount of data for a long time. Blockchain is such a use case! Kafka Processing App Storage Transactions, auth, quota enforcement, compaction, ...
- 41. Building a Secure,Tamper-Proof & Scalable Blockchain on Top of Apache Kafka Confluent Tiered Storage for Kafka Object Store Processing Storage Transactions, auth, quota enforcement, compaction, ... Local Remote Kafka Apps Store Forever – AiB’s Tamper-Proof Blockchain with KafkaBlockchain Older data is offloaded to inexpensive object storage, permitting it to be consumed at any time. Using AiB storage can be made tamper-proof and immutable Save $$$ Storage limitations, like capacity and duration, are effectively uncapped. Instantaneously scale up and down Your Kafka clusters will be able to automatically self-balance load and hence elastically scale (Only available in Confluent Platform)
- 42. Questions? Feedback? Let’s connect! KaiWaehner Technology Evangelist kai.waehner@confluent.io LinkedIn @KaiWaehner confluent.io kai-waehner.de Stephen Reed CEO, Co-founder AiB stephen@ai-blockchain.com LinkedIn ai-blockchain.com