![Improve customer experience (CX) Increase revenue (make money) Business value Decrease costs (save money) Core business platform Increase operational efficiency Migrate to cloud Mitigate risk (protect money) Key drivers Strategic objectives (sample) Fraud detection IoT sensor ingestion Digital replatforming / mainframe offload Connected car – Navigation and improved in-car experience: Audi Customer 360 Simplifying omni-channel retail at scale: Target Faster transactional processing / analysis including machine learning / AI Mainframe offload: RBC Microservices architecture Online fraud detection Online security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital transformation Application modernization: Multiple examples Website / core operations (central nervous System) The [Silicon Valley] digital natives: LinkedIn, Netflix, Uber, Yelp . . . Predictive maintenance: Audi Streaming platform in a regulated environment (e.g., electronic medical records): Celmatix Real-time app updates Real-time streaming platform for communications and beyond: Capital One Developer velocity – Building stateful financial applications with Kafka streams: Funding Circle Detect fraud and prevent fraud in real time: PayPal Kafka as a service – A tale of security and multi-tenancy: Apple Example use cases $↑ $↓ $↔ Example case studies (of many)](https://image.slidesharecdn.com/reivopn202appmodernization20201028finalkai-201201154902/75/App-modernization-on-AWS-with-Apache-Kafka-and-Confluent-Cloud-8-2048.jpg)
Uploaded byKai Wähner
App modernization on AWS with Apache Kafka and Confluent Cloud
The document discusses the evolution and significance of Apache Kafka and Confluent Cloud in modern event streaming and application modernization, emphasizing their roles in enhancing customer experience, revenue generation, and operational efficiency for various industries. It highlights key use cases, hybrid cloud architectures, and the advantages of a fully managed service for enterprises migrating from legacy systems to cloud-based solutions. The text also outlines strategic objectives for leveraging Kafka in today's data-driven environment.
App modernization on AWS with Apache Kafka and Confluent Cloud
- 1.
- 2. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. App modernization on AWS with Apache Kafka and Confluent Cloud Kai Waehner Technology Evangelist O P N 2 0 2
- 3. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Event streaming with Apache Kafka and Confluent
- 4. The rise ofevent streaming 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 companies trust and use Apache Kafka
- 5. An event streamingplatform is the underpinning of an event-driven architecture Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experiences Streams of real time events Streaming appsStreaming apps Streaming apps
- 6. Kafka Connect andKafka Streams 6 SinkSource Kafka Streams Kafka Connect Kafka Connect Your app
- 7. Confluent pioneered event streaming EnterpriseTechnology Innovation Hall of Innovation CTO Innovation Award Winner 2019 AWARDS Confluent founders are original creators of Kafka Confluent team wrote 80% of Kafka software commits, has over one million hours technical experience with Kafka, and operates 5,000+ clusters Confluent Cloud is the only multi-cloud, fully managed, pay-as-you-go event streaming service in the world Confluent Platform completes Apache Kafka and turns it into a secure, enterprise-ready platform
- 8. Improve customer experience (CX) Increase revenue (make money) Business value Decrease costs (save money) Core business platform Increase operational efficiency Migrate to cloud Mitigate risk(protect money) Key drivers Strategic objectives (sample) Fraud detection IoT sensor ingestion Digital replatforming / mainframe offload Connected car – Navigation and improved in-car experience: Audi Customer 360 Simplifying omni-channel retail at scale: Target Faster transactional processing / analysis including machine learning / AI Mainframe offload: RBC Microservices architecture Online fraud detection Online security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital transformation Application modernization: Multiple examples Website / core operations (central nervous System) The [Silicon Valley] digital natives: LinkedIn, Netflix, Uber, Yelp . . . Predictive maintenance: Audi Streaming platform in a regulated environment (e.g., electronic medical records): Celmatix Real-time app updates Real-time streaming platform for communications and beyond: Capital One Developer velocity – Building stateful financial applications with Kafka streams: Funding Circle Detect fraud and prevent fraud in real time: PayPal Kafka as a service – A tale of security and multi-tenancy: Apple Example use cases $↑ $↓ $↔ Example case studies (of many)
- 9. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Modernization with hybrid architectures using Kafka, Confluent, and AWS
- 10. Redshift Sink Lambda Sink AWSDirect Connect Replicator Legacy EDW Mainframe Legacy DB JDBC / CDC connectors On-premises AWS cloud Amazon Athena AWS Glue SageMakerLake Formation Amazon Dynamo DB Amazon Aurora S3 Sink Data streamsApps ksqlDB Hybrid event streaming
- 11. Data integration withKafka Connect 100+ prebuilt connectors 80+ Confluent supported 20+ Partner supported, Confluent verified Amazon Redshift Amazon Kinesis Amazon S3
- 12. A common streamprocessing architecture DB Connector Connector App App DB Stream processing Connector AppDB 2 3 4 1
- 13. A simplified streamprocessing architecture with ksqlDB DB App App DB Pull Push Connectors Stream processing State stores ksqlDB 1 2 App
- 14. Connectivity + On premises and cloud + Hybridreal time replication at scale https://www.confluent.io/kafka-summit-sf18/bringing-streaming-data-to-the-masses Bayer AG C L O U D - F I R S T S T R A T E G Y A N D S T A R T E D A M U L T I - Y E A R T R A N S I T I O N T O T H E C L O U D W I T H A K A F K A - B A S E D C R O S S - D A T A C E N T E R D A T A H U B Real-life use cases and architectures for event streaming with Apache Kafka and Confluent
- 15. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Modernization and cloud migration
- 16. Cloud adoption Journey from mainframeto hybrid and cloud Phase 3 Hybrid cloud Cloud-first development Phase 2 Phase 1 Case study – Bank CEO “This is the last 5-year $20M IBM contract. Get rid of the mainframe!”
- 17. Redshift Sink Lambda Sink AWSDirect Connect Replicator Legacy EDW Mainframe Legacy DB JDBC / CDC connectors On-premises AWS cloud Amazon Athena AWS Glue SageMakerLake Formation Amazon Dynamo DB Amazon Aurora S3 Sink Data streamsApps ksqlDB Hybrid event streaming
- 18. Strangler design pattern https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/ https://martinfowler.com/bliki/StranglerFigApplication.html “Themost important reason to consider a strangler fig application over a cut-over rewrite is reduced risk.” Martin Fowler
- 19. Year 0: Directcommunication between mainframe and app 1) Direct legacy mainframe communication to app Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core banking ‘1970’ (mainframe) On-premises only
- 20. Year 1: Kafkafor decoupling between mainframe and app 1) Direct legacy mainframe communication to app 2) Kafka for decoupling between mainframe and app Mainframe integration - Change data capture (IIDR) - Kafka Connect (JMS, MQ, JDBC) - REST Proxy - Kafka client - Third-party CDC tool CloudOn-premises Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core banking ‘1970’ (mainframe)
- 21. Year 2 to4: New projects and applications Microservices Agile, lightweight (but scalable, robust) applications Big data project (Elastic, Spark, AWS Services . . .) 1) Direct legacy mainframe communication to app 2) Kafka for decoupling between mainframe and app 3) New projects and applications External solution On-premises Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Core banking ‘1970’ (mainframe) Mainframe integration - Change data capture (IIDR) - Kafka Connect (JMS, MQ, JDBC) - REST Proxy - Kafka client - Third-party CDC tool Cloud
- 22. Year 5: Mainframereplacement 1) Direct legacy mainframe communication to app 2) Kafka for decoupling between mainframe and app 3) New projects and applications 4) Mainframe replacement Core banking ‘2020’ (modern technology) Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Big data project (Elastic, Spark, AWS Services . . .) External solution Microservices Agile, lightweight (but scalable, robust) applications Cloud only
- 23. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved.
- 25. © 2020, AmazonWeb Services, Inc. or its affiliates. All rights reserved. Why Confluent?
- 26. Infrastructure management (commodity) Scaling ● Upgrades (lateststable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than 60-minute response time Infra as a service Harness full power of Kafka Kafka-specific management Platform as a service Evolve as you need Future proof Mission-critical reliability An event streaming platform needs more than Kafka Core à Fully managed data integration, stream processing, data governance, security . . . Confluent Cloud: What does fully managed mean? Most Kafka as a service offerings are partially managed
- 27.
- 28. Tiered storage andself-balancing clusters for elastically scalable and cost-efficient Kafka Broker 1 Broker 2 Lengthy, manual process for reassigning large topic partitions Broker 3 New: Broker 4 Broker 1 Broker 2 Broker 3 Broker 4 Inelastic scaling process Broker 1 Broker 2 Scaling with tiered storage and self-balancing clusters Fast, automated process for reassigning small topic partitions Broker 3 New: Broker 4 Broker 1 Broker 2 Broker 3 Broker 4 Object store Object store Dynamic performance and elasticity
- 29. Cluster linking usesthe Kafka protocol • Uses the Kafka protocol • Requires no additional infrastructure (such as MirrorMaker) • Preserves offsets Migrate Kafka clusters to Confluent Cloud Global resilience 29
- 30. Thank you! © 2020,Amazon Web Services, Inc. or its affiliates. All rights reserved. Email: contact@kai-waehner.de LinkedIn: @KaiWaehner www.confluent.io www.kai-waehner.de