Downloaded 13 times
Uploaded byAndrea Gioia
How to integrate legacy systems within a modern polyglot and event driven architecture using an evolutionary strategy
This document summarizes a presentation about evolving legacy systems to a modern, event-driven architecture using Apache Kafka and change data capture. It discusses moving from a tangled "spaghetti architecture" to a message-driven system using microservices and APIs. It also covers offloading legacy data and processes from mainframes and other systems by reading change logs with Attunity Replicate to minimize disruption. The talk presents an evolutionary approach and maturity model for offloading from the data level to fully migrating business processes and applications.
How to integrate legacy systems within a modern polyglot and event driven architecture using an evolutionary strategy
- 1. Milano Apache KafkaMeetup
- 2. Milano Apache Kafka®Meetup / quantyca.it www.quantyca.it www.blindata.io
- 3. How to integratelegacy systems within a modern polyglot and event driven architecture using an evolutionary strategy Milano Apache Kafka® Meetup / quantyca.it Milano, 4.07.2018
- 4. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 First steps in the evolutionary journey 1 Why we need to evolve 3 The destination: message driven architecture 5 Legacy offloading 6 Attunity replicate and the CDC maturity model 2 The starting point: spaghetti architecture
- 5. We really needto evolve? Milano Apache Kafka® Meetup / quantyca.it Martec’s Law “The world is changing very fast. Big will not beat small anymore. It will be the fast beating the slow.” Rupert Murdoch
- 6. Evolve or die MilanoApache Kafka® Meetup / quantyca.it
- 7. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 First steps in the evolutionary journey 3 The destination: message driven architecture 5 Legacy offloading 6 Attunity replicate and the CDC maturity model 1 Why we need to evolve 2 The starting point: spaghetti architecture
- 8. You are here:spaghetti architecture Milano Apache Kafka® Meetup / quantyca.it @see Big Ball of Mud
- 9. Driving force: Polyglotism 9 PolyglotProgramming Applications should be written in a mix of languages to take advantage of the fact that different languages are suitable for tackling different problems. Complex applications combine different types of problems, so picking the right language for the job may be more productive than trying to fit all aspects into a single language 2006 Polyglot Persistence A complex enterprise application uses different kinds of data, and already usually integrates information from different sources. Increasingly we'll see such applications manage their own data using different technologies depending on how the data is used. This trend will be complementary to the trend of breaking up application code into separate components integrating through web services 2011 Milano Apache Kafka® Meetup / quantyca.it Polyglotism without a clear IT strategy has increased the architectural technical debt @see Polyglot Persistence by Martin Fowler
- 10. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 First steps in the evolutionary journey 3 The destination: message driven architecture 1 Why we need to evolve 5 Legacy offloading 6 Attunity replicate and the CDC maturity model 2 The starting point: spaghetti architecture
- 11. Message driven systemsto the rescue Reactive Systems rely on asynchronous message- passing to establish a boundary between components that ensures loose coupling, isolation and location transparency. This boundary also provides the means to delegate failures as messages. Employing explicit message-passing enables load management, elasticity, and flow control by shaping and monitoring the message queues in the system and applying back-pressure when necessary. Location transparent messaging as a means of communication makes it possible for the management of failure to work with the same constructs and semantics across a cluster or within a single host. Non-blocking communication allows recipients to only consume resources while active, leading to less system overhead. The Reactive Manifesto Milano Apache Kafka® Meetup / quantyca.it @see https://www.reactivemanifesto.org/
- 12. Message driven architectures Event-drivenarchitectures (EDA) usually integrate several disparate systems together using message queues. There are two common implementations of this type of architecture: Milano Apache Kafka® Meetup / quantyca.it broker pattern mediator pattern @see Variations in event-driven architecture: Find out if mediator or broker topology is right for you.
- 13. Traditional SOA (theevil) is an example of mediator pattern In building communication structures between different processes, SOA approach put significant smarts into the communication mechanism itself (smart pipe, dumb endpoints) Milano Apache Kafka® Meetup / quantyca.it = Spaghetti architecture Spaghetti in a box architecture @see Does My Bus Look Big in This?
- 14. The real problemwith traditional SOA The problem with traditional SOA is not anyway the use of the mediator pattern but the degree to which developers have deviated from bounded context. This is the key element that largely determines the quantum size and how much damaging coupling appears in the overall architecture. Enterprise services Share nothing domain services > coupling continuum Milano Apache Kafka® Meetup / quantyca.it Traditional SOA Microservices @see Microservices as an Evolutionary Architecture
- 15. Software architectures aren’tcreated in a vacuum If there are clear benefits, then why haven’t developers embraced this style before? A decade ago, automatic provisioning of machines wasn’t possible. Operating systems were commercial and licensed, with little support for automation. Real-world constraints like budgets impact architectures, which is one of the reasons developers build more and more elaborate shared resources architectures, segregated at the technical layers. If operations is expensive and cumbersome, architects build around it, as they did in ESB-SOAs. Milano Apache Kafka® Meetup / quantyca.it
- 16. Benefit and tradeoffs Developersunderstand the benefits of everything and the tradeoffs of nothing! Rich Hickey, creator of Clojure Milano Apache Kafka® Meetup / quantyca.it Architects must understand both benefits and tradeoffs of microservices and build engineering practices accordingly. Despite their popularity microservices are not the right solution for any kind of problems
- 17. Alternative to microservices Relaxingshare nothing principle A more commonly used architectural style for migration is a service-based architecture, which is similar to but could differ from microservices in three important ways: 1. service granularity, 2. database scope, 3. integration middleware. Service-based architectures are still domain-centric but address some challenges developers face when restructuring existing applications toward more evolutionary architectures. Microservices and service based architectures can cohesist. It’s a responsibility of the system architects to decide analyzing the different use cases when to go with one solution and when to go with the other. Milano Apache Kafka® Meetup / quantyca.it @see Service-Based Architecture as an Alternative to Microservice Architecture
- 18. Example of anhybrid service architecture Milano Apache Kafka® Meetup / quantyca.it Service based architecture and the service bus are used when 1. control is more important than scalability 2. there are transaction that span bounded contexts 3. there are not enough skilled developers to implement all with microservices 4. there is a monolith that requires an anti-corruption layer 5. there is a legacy database that cannot be splitted between microservices Microservices architecture and the data bus are used in all other cases An API gateway hides to the callers if it are calling a service through the service bus or the data bus @see Quantyca’s Reference Implementation
- 19. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 The starting point: spaghetti architecture 3 The destination: message driven architecture 5 Legacy offloading 6 Attunity replicate and the CDC maturity model 1 Why we need to evolve 2 First steps in the evolutionary journey
- 20. How to evolvefrom spaghetti architecture? Milano Apache Kafka® Meetup / quantyca.it
- 21. Step 1: Workingat the edge Milano Apache Kafka® Meetup / quantyca.it @see Microservice Prerequisites
- 22. Step 2: Refactoringmonoliths Milano Apache Kafka® Meetup / quantyca.it @see How to break a Monolith into Microservices
- 23. Refactoring monoliths Milano ApacheKafka® Meetup / quantyca.it
- 24. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 First steps in the evolutionary journey 5 Legacy offloading 1 Why we need to evolve 3 The destination: message driven architecture 6 Attunity replicate and the CDC maturity model 2 The starting point: spaghetti architecture
- 25. Vendor king antipattern Theviral lock-in Many organizations become overambitious with some legacy software (i.e. ERP, Mainframe, ecc ..), leading to the vendor king antipattern, an architecture built entirely around a vendor product that pathologically couples the organization to a tool. By placing an external tool or framework at the heart of the architecture, developers severely restrict their ability to evolve in two key ways, ■ TECHNICAL PROCESS From a business process standpoint the tool whole architecture depends by choices the vendor makes in terms of persistence, supported infrastructure, and a host of other constraints ■ BUSINESS PROCESS From a business process standpoint, the tool simply can’t support the optimal workflow; this is a side effect of the Last 10% Trap. Most companies end up knuckling under the framework, modifying their processes rather than trying to customize the tool. The more companies do that, the less differentiators exist between companies, which is fine as long as that differentiation isn’t a competitive advantage. Milano Apache Kafka® Meetup / quantyca.it
- 26. To big toremove Milano Apache Kafka® Meetup / quantyca.it Having developed and optimized Legacy King Systems for decades, enterprises are looking for ways to capitalize on these investments with the ability to reuse, repurpose, and integrate them in support of new business and IT initiatives.
- 27. Legacy offloading Milano ApacheKafka® Meetup / quantyca.it DB2/MF IMS VSAM SAP Legacy King Data Warehouse Straem Analytics Hybrid Cloud Microservices C-Level / Menagement Customer LoB Analyst WHERE DATA NEED TO BE KAFKA OFFLOADING
- 28. Offloading via batchloading ● Queries based on timestamp orchestrated with custom ETL Jobs ● Consume significant processing power on the source system ● Usually executed during “batch windows” (not responsive) ● Unable to manage easily changes in the schema structure ● Custom error handling and failover management Milano Apache Kafka® Meetup / quantyca.it @see Streaming change data capture
- 29. Offloading via CDC MilanoApache Kafka® Meetup / quantyca.it ● Read incremental changes directly from the transaction log ● Minimize disruption to production workloads ● Fast updates (near real-time) ● Manage easily changes in the schema structure ● Mature market @see Streaming change data capture
- 30. Agenda Milano Apache Kafka®Meetup / quantyca.it 4 First steps in the evolutionary journey 3 The destination: message driven architecture 5 Legacy offloading 2 The starting point: spaghetti architecture 1 Why we need to evolve 6 Attunity replicate and the CDC maturity model
- 31. Attunity Replicate Milano ApacheKafka® Meetup / quantyca.it @see Attunity Replicate Home Page
- 32. Attunity: workflow Milano ApacheKafka® Meetup / quantyca.it STEP 1: Create connections STEP 2: Create task STEP 3: Define task
- 33. Attunity: workflow Milano ApacheKafka® Meetup / quantyca.it STEP 4: Run task
- 34. Attunity: Source andtargets Milano Apache Kafka® Meetup / quantyca.it
- 35. Attunity: Kafka target MilanoApache Kafka® Meetup / quantyca.it
- 36. Attunity: business cases MilanoApache Kafka® Meetup / quantyca.it
- 37. Offloading maturity model MilanoApache Kafka® Meetup / quantyca.it Data offloading Process offloading Business offloading
- 38. Thanks! Questions? Milano Apache Kafka®Meetup / quantyca.it Milano, 4.07.2018
- 39. Corso Milano, 45/ 20900 Monza (MB) T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it www.quantyca.it
Editor's Notes
- #12 https://www.reactivemanifesto.org/
- #14 In a microservices architecture, each bounded context represents a business process or workflow.
- #15 In a microservices architecture, each bounded context represents a business process or workflow.
- #16 The Continuous Delivery and DevOps movements added a new factor into the dynamic equilibrium. Now, machine definitions live in version control and support extreme automation. Deployment pipelines spin up multiple test environments in parallel to support safe continuous deployment. Because much of the software stack is open source, licensing and other concerns no longer impact architectures. The community reacted to the new capabilities emergent in the software development ecosystem to build more domain-centric architectural styles. Make sure your architecture matches the problem domain. Don’t try to force fit an unsuitable architecture.
- #18 Larger service granularity The services in this architecture tend to be larger, more “portion of a monolith” granularity than purely around domain concepts. While they are still domain-centric, the larger size makes the unit of change (development, deployment, coupling, and a host of other factors) larger, diminishing the ability to make change easily. When architects evaluate a monolithic application, they often see coarse-grained divisions around common domain concepts such as CatalogCheckout or Shippping, which form a good first-pass at partitioning the architecture. The goals of operational isolation are the same in service-based architectures as in microservices but are more difficult to achieve. Because the service size is bigger, developers must consider more coupling points and the complications inherent in larger chunks of code. Ideally, the architecture should support the same kind of deployment pipeline and small unit of change as microservices: when a developer changes a service, it should trigger the deployment pipeline to rebuild the dependent services, including the application. Database scope Service-based architectures tend towards a monolithic database, regardless of how well-factored the services are. In many applications, it isn’t feasible or possible to restructure years (or decades) of intractable database schemas into atomic-sized chunks for microservices. While the inability to atomize the data may be inconvenient in some situations, it is impossible in some problem domains. Heavily transactional systems are a poor match for microservices because coordination between services, transactional behavior is too costly. Systems with complex transactional requirements map more cleanly to service-based architectures because of less stringent database requirements. While the database remains unpartitioned, the components that rely on the database will likely change, becoming more granular. Thus, while the mapping between the services and the underlying data may change, it requires less restructuring. We cover evolutionary database design in Chapter 5. Integration middleware The third difference between microservices and service-based architectures concerns externalized coordination via a mediator like a service bus. Building greenfield microservices applications allows developers to not worry about old integration points, but those horrors describe many environments rife with legacy systems that still perform useful work. Integration hubs, like enterprise service buses, excel at forming glue between disparate services with different protocols and message formats. If architects find themselves in environments where integration architecture is the top priority, using an integration hub makes adding and changing dependent services easier.
- #26 Rather than fall victim to the vendor king antipattern, treat vendor products as just another integration point. Developers can insulate vendor tool changes from impacting their architecture by building anticorruption layers between integration points.