Downloaded 12 times
Uploaded byScyllaDB
Building an Event Streaming Architecture with Apache Pulsar
The document introduces Apache Pulsar, a unified messaging and streaming platform, detailing its features, architecture, and connectivity options. It covers the integration of various data processing libraries and protocols, along with real-world adoption cases in companies like Tencent and Applied Materials. Additionally, it highlights the educational resources and events like the Pulsar Summit for those interested in learning more about the platform.
Building an Event Streaming Architecture with Apache Pulsar
- 1. Building an Event StreamingArchitecture with Apache Pulsar Developer Advocate at StreamNative Tim Spann
- 2.
- 3. Agenda ● Apache PulsarBasics ○ Features ○ Architecture ○ Connectivity ○ Protocols ● Stream Storage ○ In-Memory ○ Apache BookKeeper ○ ScyllaDB ● Demo
- 4.
- 5.
- 6. Streaming Consumer Consumer Consumer Subscription Shared Failover Consumer Consumer Subscription In case offailure in Consumer B-0 Consumer Consumer Subscription Exclusive X Consumer Consumer Key-Shared Subscription Pulsar Topic/Partition Messaging Unified Messaging Model
- 7. Connectivity • Libraries -(Java, Python, Go, NodeJS, WebSockets, C++, C#, Scala, Kotlin,...) • Functions - Lightweight Stream Processing (Java, Python, Go) • Connectors - Sources & Sinks (Cassandra, Kafka, …) • Protocol Handlers - AoP (AMQP), KoP (Kafka), MoP (MQTT), RoP (RocketMQ) • Processing Engines - Flink, Spark, Presto/Trino via Pulsar SQL • Data Offloaders - Tiered Storage - (S3) hub.streamnative.io 7
- 8. ScyllaDB Compatible SinkConnector https://pulsar.apache.org/docs/en/io-quickstart/ ScyllaDB Compatible Sink 8
- 9. Kafka On Pulsar (KoP) 9 Pulsar Producer Pulsar Lib Pulsar Producer Pulsar Lib Kafka Producer Kafka Lib Kafka Consumer Kafka Lib Pulsar ProtocolHandler Kafka Protocol Handler Managed Ledger Pulsar Topic BK Client Geo Replicator Load Balance Pulsar Broker Zookeeper Bookie Pulsar Cluster
- 10. MQTT On Pulsar (MoP) 10 Pulsar Producer Pulsar Lib Pulsar Producer Pulsar Lib MQTT Producer MQTT Lib MQTT Consumer MQTT Lib Pulsar ProxyMQTT Proxy Pulsar Protocol Handler MQTT Protocol Handler Managed Ledger Pulsar Topic BK Client Geo Replicator Load Balance Pulsar Broker Zookeeper Bookie
- 11. AMQP On Pulsar (AoP) 11 Pulsar Producer Pulsar Lib Pulsar Producer Pulsar Lib RabbitMQ Producer RabbitMQ Lib RabbitMQ Consumer RabbitMQ Lib Pulsar ProxyAMQP Proxy Pulsar Protocol Handler AMQP Protocol Handler Managed Ledger Pulsar Topic BK Client Geo Replicator Load Balance Pulsar Broker Zookeeper Bookie
- 12. Schema Registry Schema Registry schema-1 (value=Avro/Protobuf/JSON) schema-2 (value=Avro/Protobuf/JSON) schema-3 (value=Avro/Protobuf/JSON) Schema Data ID LocalCache for Schemas + Schema Data ID + Local Cache for Schemas Send schema-1 (value=Avro/Protobuf/JSON) data serialized per schema ID Send (register) schema (if not in local cache) Read schema-1 (value=Avro/Protobuf/JSON) data deserialized per schema ID Get schema by ID (if not in local cache) Producers Consumers 12
- 13. ● Consume messagesfrom one or more Pulsar topics. ● Apply user-supplied processing logic to each message. ● Publish the results of the computation to another topic. ● Support multiple programming languages (Java, Python, Go) ● Can leverage 3rd-party libraries to support the execution of ML models on the edge. 13 Pulsar Functions
- 14. Function Mesh Pulsar Functions,along with Pulsar IO/Connectors, provide a powerful API for ingesting, transforming, and outputting data. Function Mesh, another StreamNative project, makes it easier for developers to create entire applications built from sources, functions, and sinks all through a declarative API. 14
- 15. 15 ● Buffer ● Batch ●Route ● Filter ● Aggregate ● Enrich ● Replicate ● Dedupe ● Decouple ● Distribute Multi-Cloud Deployment
- 16.
- 17.
- 18. Passionate and dedicatedteam. Founded by the original developers of Apache Pulsar. StreamNative helps teams to capture, manage, and leverage data using Pulsar’s unified messaging and streaming platform. 18
- 19. Powered by ApachePulsar, StreamNative provides a cloud-native, real-time messaging and streaming platform to support multi-cloud and hybrid cloud strategies. Built for Containers Cloud Native StreamNative Cloud Flink SQL 19
- 20. Keep in touch! TimSpann Developer Advocate StreamNative tim@streamnative.io @PaasDev
- 21.
- 22. Executive Team ✓ Dataveterans with extensive industry experience ✓ Original creators of Apache Pulsar & BookKeeper ✓ Operated the largest Pulsar/BookKeeper cluster Sijie Guo Founder and CEO Apache Software Foundation Member Pulsar / BookKeeper PMC Matteo Merli CTO Co-creator and PMC Chair of Pulsar, BookKeeper PMC Jia Zhai Co-Founder Pulsar / BooKKeeper PMC 22
- 23.
- 24. Adopted Pulsar toreplace Kafka in their DSP (Data Streaming Platform). ● 1.5-2x lower in capex cost ● 5-50x improvement in latency ● 2-3x lower in opex due Adopted Pulsar to power their billing platform, Midas, which processing hundreds of billions of financial transactions daily. Adoption then expanded to Tencent’s Federated Learning Platform and Tencent Gaming. Applied Materials is one of the biggest semiconductor hardware and software supplier in the industry. They adopted Pulsar to enable them to build a message bus to tie all of their data together. They previously used Tibco. Pulsar Adoption Use Cases 24
- 25. StreamNative Academy ➔ Pulsarexpert instructor-led courses ➔ On-demand learning with labs ➔ 300+ engineers, admins and architects trained! Academy.StreamNative.io LEARN MORE ABOUT APACHE PULSAR WITH: 25
- 26. Pulsar Summit SanFrancisco ➔ August 18th, 2022 ➔ Hotel Nikko San Francisco ➔ Hosted by Get early bird tickets now! Pulsar-Summit.org
- 27. Pulsar Summit SanFrancisco Sponsorship Prospectus Help engage and connect the Apache Pulsar community by becoming an official sponsor for Pulsar Summit San Francisco 2022! Learn more about the requirements and benefits of becoming a community sponsor. COMMUNITY SPONSORSHIPS AVAILABLE: LEARN MORE https://hubs.ly/Q01dJ3ly0
- 28. Welcome to Pulsar! Scanthe QR code to sign up for the StreamNative Newsletter for Apache Pulsar. You will get the latest Pulsar news and have the chance to win a Pulsar Giveaway Bundle valued at $500! $500 Pulsar Giveaway Bundle: ■ 1 pair of Apple AirPods Pro ■ StreamNative and We <3 Pulsar swag ■ 2 free registration tickets to Pulsar Summit San Francisco 2022!
Editor's Notes
- #6 The compute is done in Pulsar, while the state is stored in BookKeeper, leveraging an external metadata store (e.g. Zookeeper). This allows for scaling to support (use case) (e.g. commodity storage, Flink, Spark, etc). To show it in a diagram…
- #17 ScyllaDB Cloud T3 cluster for testing (Free for 30 days!)
- #18 Pulsar is a unified Messaging and Event Streaming Platform
- #23 Instructor Notes Data veterans with extensive industry experience Original creators of Apache Pulsar & BookKeeper Operated the largest Pulsar/BookKeeper cluster More Apache Pulsar PMC/Committers than any other company Lead the day-to-day development of Apache Pulsar Provide services and support to help drive Pulsar adoption
- #29 LEARN MORE ABOUT PULSAR AND ENTER TO WIN! https://share.hsforms.com/1dAyP4l3jQey3KBWH8Q9EGQ3x5r4