Elastically Scaling Kafka Using Confluent

Elastically Scaling Kafka Using Confluent

• 1.
  Elastically Scaling Kafka UsingConfluent
• 2.
  Confluent Platform Fully ManagedCloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer
• 3.
  Frictionless operations Kafka is areliable, real-time, distributed architecture but takes tons of resources to manage, especially at scale and across multiple environments Better elasticity Scaling Kafka to meet demand is a complex, manual process - and ensuring granular data security as use cases and teams grow is not easy Lower operational cost Kafka is the best-in-class data architecture, but managing it requires an upfront investment in hardware, people to support it and expertise to keep it running efficiently Apache Kafka is powerful. Contemporary digital businesses require more. 3
• 4.
  Step 1: Elasticscaling for Kafka Get started quickly Spin up Kafka with a few clicks in cloud or bring cloud-like experience to self-managed Kafka deployment Scale elastically Effortlessly scale workloads from 0 MBps to GBps as needed without complex sizing and provisioning Pay for use Pay only for data streamed with scale-to-zero pricing, mix and match across 3 cluster types to fit any scale, budget, or use case Confluent Cloud is the only Kafka service offering serverless properties like elastic scaling and scale-to-zero pricing, enabling customers to go from 0 to production without ever speaking with sales or worrying about future needs 1 2 3
• 5.
  5 Start streaming ina matter of minutes Quickly deploy fully-managed Kafka in any cloud with just a few clicks Automate the deployment of Kafka on Kubernetes using a standardized architecture for self-managed clusters. Effortlessly connect your critical data sources and sinks to Kafka with 100+ pre-built connectors Process streaming data with ksqlDB’s simple SQL interface without writing source code 1
• 6.
  6 Let your appgrow and shrink with customer demand Instantly scale your apps from 0 to 100 MB/s and back down without ever sizing a cluster in Confluent Cloud Basic and Standard tiers Grow to Gbps scale with self-serve provisioning and expansion in Dedicated tier Dynamically scale self-managed Kafka clusters on Kubernetes - using lighter- weight brokers that rebalance automatically Throughput/DataTraffic demand capacity 2
• 7.
  Confluent Cloud Milliseconds Minutes Basic,Standard [0-100Mbps] Do Nothing Elastic Scaling w fully-managed Confluent Cloud *Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already. Dedicated [Mbps - Gbps] 1 Click Select CKU from drop down in cluster management UI and click Apply Changes Other Kafka Services Days - Weeks Determine how much capacity is needed Procure capacity* Configure new brokers a. Disks b. OS c. Network d. Kafka (application) Identify partitions on specific brokers to rebalance & topics they are part of For each Topic: migrate partitions a. Increase ISR +1 b. Wait for new replica to sync c. Failover master d. Reduce ISR -1 e. Delete old replica 2
• 8.
  Dynamic Scaling withConfluent Platform 2 T3, P0 T1, P0 T2, P1 T3, P0 T1, P1 T2, P0 T3, P1 Broker 1 Broker 2 New: Broker 3 T1, P0 T2, P1T1, P1 T2, P0T3, P1 Broker 1 Broker 2 Broker 3 Scaling with OSS Kafka T3, P0T1, P0 T2, P1 Broker 1 Broker 2 New: Broker 3 Object Store Broker 1 Broker 2 Broker 3 Object Store T3, P1T1, P1 T2, P0 T3, P0T1, P0 T2, P1T3, P1 T1, P1 T2, P0 Scaling with Tiered Storage + Self- balancing Clusters on Kubernetes Rebalance in Hours... Rebalance in Seconds...
• 9.
  9 Accessible for any budgetor use case with scale-to-0 pricing Get started for free Pay only for data streamed with a PAYG model - easily spin resources up and down as needed As your Kafka scales, leverage usage- based commitment for discounts and access to Dedicated clusters Mix and match resources across our three cloud tiers and everything else available in the cloud service like connectors, ksqlDB, and support tiers $22.00 ($22.00) 200 GB $28.00/$50.00 USD remaining 0 GB 0 GB $0.00 $22.00 $0.00 200 GB $22.00 Try Free $ $ $ $ PAY-AS-YOU-GO $$ COMMITMENT $$ x % OFF 3
• 10.
  Usage-based pricing helpslower your costs 10 1 Standard Cluster Rate Card Unit Price ($/GB) Usage Spend ($/month) Compute $1.5 1 $1,080 Read $0.06 253 $15 Write $0.13 253 $33 Storage $0.10 177 $18 Total $1,146 Running clusters with 253GB read/write, 177GB stored in Confluent Cloud 5 Basic Clusters Rate Card Unit Price ($/GB) Usage Spend ($/month) Compute $0.0 - $0 Read $0.13 253 $33 Write $0.13 253 $33 Storage $0.10 177 $18 Total $84 Total monthly spend of $1,229 for 6 clusters that includes 1 Standard cluster, or just $84 per month for 5 Basic clusters
• 11.
  Try FREE $50 Free eachmonth 3 Months from signup confluent.io/cloud
• 12.
  Live Demo -Cloud Pipeline 12 AWS RDS - Postgres DB GCP Storage - Buckets Confluent Cloud - ksqlDB Confluent Cloud - Kafka
• 13.
  Live Demo -Data Flow 13 SQL table: eventlogs Kafka topic: eventlogs ksqlDB stream: eventlogs ksqlDB tables: COUNT_PER_SOURCE, SUM_PER_SOURCE Kafka topics: COUNT_PER_SOURCE, SUM_PER_SOURCE Storage Bucket files: *.json, *.avro
• 14.
  Live Demo
• 15.
  Any Questions?