Superpower Your Apache Kafka Applications Development with Complementary Open Source Technologies

© 2024 NetApp, Inc. All rights reserved. © 2024 NetApp, Inc. All rights reserved. Kafka Summit, Bangalore 2024 Superpower your Apache Kafka® applications development with complementary open source technologies Paul Brebner Instaclustr Technology Evangelist

© 2024 NetApp, Inc. All rights reserved. Complementary Kafka Technologies Cassandra PostgreSQL Superset Camel Cadence OpenTelemetry TensorFlow RisingWave LLMs Guava EventBus Kubernetes Prometheus Grafana Parallel Consumer OpenSearch + Dashboard Matisse, Goldfish - Red/Green complementary colors (Source: Wikimedia)

© 2024 NetApp, Inc. All rights reserved. C.f. analogous Kafka technologies • Apache Pulsar, Flink, Storm, Spark Streaming, Beam, ActiveMQ, RocketMQ, StreamPark, RisingWave etc. Van Gogh, Sunflowers on Yellow Background, (Source: Wikimedia) But we will look at RisingWave

© 2024 NetApp, Inc. All rights reserved. Design 1: Pure Kafka, many topics 1000s of locations (warehouses, trucks) and millions of goods Each location has a topic and multiple consumer groups (all goods at that location) 7,000 TPS → SLOW! Many topics/partitions (without increasing cluster resources) reduced throughput on older versions of Kafka

© 2024 NetApp, Inc. All rights reserved. Design 2: One topic + Guava EventBus for notifications Single topic, one consumer group Kafka supplemented with Guava Event Bus to handle high fan-out notifications 1.2M TPS → FAST! Uber’s Cadence can be/has been used for scalable notifications

© 2024 NetApp, Inc. All rights reserved. Streaming anomaly detection Incoming Event Stream Run Anomaly Check – Quickly! Persist new event Get previous 50 events for key Run algorithm Fast writes → Cassandra Application scaling → Kubernetes Initially single threaded consumers

© 2024 NetApp, Inc. All rights reserved. Apache Cassandra® What? • NoSQL horizontally scalable key-value database Superpowers • Fast writes (lots of typewriters) • Wide column store • Good for ML feature stores • Clustering columns • Good for hierarchical data modeling (eg. Geospatial) • In-built multi-DC replication

© 2024 NetApp, Inc. All rights reserved. Kubernetes What? • Automation of containerized applications Superpowers • Available on public clouds (E.g. AWS EKS) • Ephemeral Pods are the unit of concurrency • Easy to scale applications with more or less Pods

© 2024 NetApp, Inc. All rights reserved. Prometheus + Grafana What? • Prometheus: Monitoring and alerting • Grafana: Graphing Superpowers • Instrumentation or agents (exporters) to expose application metrics • Time series data with counter, gauge, histogram, and summary metrics • Instaclustr monitoring API supports Prometheus metrics for Apache Kafka clusters • Integration of Kafka Cluster metrics and Kafka application (e.g. producers and consumers) is powerful à Metrics suggested optimizations

© 2024 NetApp, Inc. All rights reserved. Slow Kafka consumers problem Slow consumers require more partitions/consumers (Source: Getty Images) Little’s Law: Concurrency (Partitions=Consumers) = Time x Throughput

© 2024 NetApp, Inc. All rights reserved. Optimize consumer speed/concurrency using 2 stage pipeline Less consumers (around 100) gives higher throughput— a surprise! Hint: Less partitions 1. Minimize polling time (thread pool 1) 2. Maximize anomaly detector concurrency (thread pool 2) 1 2

© 2024 NetApp, Inc. All rights reserved. Kafka Parallel Consumer: Multi-threaded consumer • Multiple ordering options—c.f. default Kafka only guarantees order within partitions! PARTITION → KEY → UNORDERED Increasing concurrency → • Concurrency from 1 to lots—depends on client resources, and partitions/key space sizes • KEY has higher concurrency than partition and is ordered by KEY— reasonable compromise • Higher concurrency for less partitions/consumers

© 2024 NetApp, Inc. All rights reserved. OpenSearch + Dashboard What? • Open source version of Elasticsearch • Based on Lucene—powerful and scalable text searching Superpowers • Ingestion, indexing, and searching of JSON documents • Complex linguistic and geospatial queries • Integrated dashboard for visualization

© 2024 NetApp, Inc. All rights reserved. Apache Camel™ What? • Apache Camel – integration framework • Apache Camel Kafka Connectors Superpowers • Large number of open source Kafka Connectors—179 sources and sinks • Auto-generated from Camel components

© 2024 NetApp, Inc. All rights reserved. Uber’s Cadence® What? • Scalable code-as-workflows engine Superpowers • Sequenced, stateful, long-running, scheduled steps • Scalable and reliable using event-sourcing o Workflows are failproof, history is replayed until the point of failure and resumed

© 2024 NetApp, Inc. All rights reserved. Drone way point flight calculations Returning to base leg • Drone flight path is computed in an activity • Using location, distance, bearing, speed, and charge • Every 10 seconds • On failure, the drone won’t crash and will continue flying from the last location

© 2024 NetApp, Inc. All rights reserved. Uber’s Cadence + Apache Kafka = similarities Cadence (Workflows) Kafka (Streaming Events) Scalable (event sourcing) Scalable (partitions, cluster) Persistent (event sourcing) Persistent (event replaying) Reliable workflow execution (event sourcing) Reliable event delivery Asynchronous signals Asynchronous events Open source Open source Available as a managed service Available as a managed service

© 2024 NetApp, Inc. All rights reserved. Cadence + Kafka = Complementary timescales Cadence (Slow Workflows) Kafka (Fast Streaming Events) Synchronous events Asynchronous events Stateful flows Stateless events Sequences One-off events Slow/long running flows Fast/instantaneous events Sleep/schedule events Real-time processing of events Complex flow logic Complex stream processing (Kafka Streams)

© 2024 NetApp, Inc. All rights reserved. Drone learning problem Kafka Streams Kafka Streams computes aggregated hourly shop and order details → Busy/NotBusy categorization Sent to TensorFlow Train model to predict shop busy/not busy an hour ahead Simulation produces streaming spatiotemporal data (drone and order state and locations)

© 2024 NetApp, Inc. All rights reserved. TensorFlow Watch out for • ML over streaming spatiotemporal data with concept drifts is tricky o Time/space bias - Wild model accuracy oscillation o Concept shift can result in very low-accuracy models initially - Train/use multiple models

© 2024 NetApp, Inc. All rights reserved. Use case 6: Santa’s elves' toy and box packing KafkaStreams, ChatGPT, RisingWave, and OpenTelemetry Streaming joins to match toys and boxes (Source: Adobe Stock)

© 2024 NetApp, Inc. All rights reserved. OpenTelemetry • OpenTelemetry is the new standard for distributed tracing • Combines tracing (OpenTracing), metrics, and logs • Automatic instrumentation • Lots of open source visualization tools - Jager, SigNoz, Uptrace, etc. • Used in new client monitoring KIP-714 - Kafka 3.7.0

© 2024 NetApp, Inc. All rights reserved. RisingWave What? • Stream processing database—also as a service Superpowers • Stateful stream processing o SQL syntax o Using cloud native storage o Potential replacement for Kafka Streams • PostgreSQL compatible o Works with Apache Superset for visualization

© 2024 NetApp, Inc. All rights reserved. LLMs/GenAI • E.g. ChatGPT - not open source + there may be suitable open source alternatives for code generation • Worked well to generate + Kafka clients + Kafka Streams DSL + and test-cases • Not as accurate for RisingWave - lack of examples?

© 2024 NetApp, Inc. All rights reserved. Bonus Technologies from my Instaclustr colleagues ● Kafka benchmarking ○ Apache JMeter for Kafka benchmarking (Thanks to Anup Shirolkar) ○ OpenMessaging (Thanks to Alastair Daivis) ● Strimzi – a Kafka Operator for Kubernetes, and Debezium (CDC using Kafka Connect) (Thanks to Felix Alipaz-Dicke) ● Kafka GUIs (Thanks to Ana-Maria Minda) ○ Kafdrop ○ AKHQ ○ UI for Apache Kafka ○ These all work with Kafka + Instaclustr console and provide complementary features

© 2024 NetApp, Inc. All rights reserved. Try us out • We offer Apache Kafka and these open source technologies as a managed service • You can use the others with our managed services • FREE 30-day trial of developer- sized clusters

Superpower Your Apache Kafka Applications Development with Complementary Open Source Technologies

More Related Content

Similar to Superpower Your Apache Kafka Applications Development with Complementary Open Source Technologies

More from Paul Brebner

Recently uploaded

Superpower Your Apache Kafka Applications Development with Complementary Open Source Technologies