Developing, Testing and Scaling with Apache Camel - UberConf 2015

Photos by Developing, Testing and Scaling with Apache Camel Matt Raible • http://raibledesigns.com

Blogger on raibledesigns.com Founder of AppFuse Father, Skier, Mountain Biker, Whitewater Rafter Web Framework Connoisseur Who is Matt Raible? Bus Lover

What about YOU? Are you familiar with Enterprise Integration Patterns? Have you used Apache Camel? What about Spring Integration? XML or JavaConﬁg? How do you feel about testing?

Do you want to take a Camel ride?

What is Apache Camel? Apache Camel is a versatile open-source integration framework based on known Enterprise Integration Patterns.

Enterprise Integration Why is it necessary? Businesses need to integration diﬀerent systems Why a framework? To allow you to focus on the business logic Others have done it before, don’t reinvent the wheel Simpliﬁed testing

Camel EIPs http://camel.apache.org/eip

Content Based Router from newOrder choice when isWidget to widget otherwise to gadget

Content Based Router from(newOrder) choice() when(isWidget) to(widget) otherwise to(gadget)

Content Based Router from(newOrder) .choice() .when(isWidget).to(widget) .otherwise().to(gadget);

Content Based Router Endpoint newOrder = endpoint(“activemq:queue:newOrder"); from(newOrder) .choice() .when(isWidget).to(widget) .otherwise().to(gadget);

Content Based Router Endpoint newOrder = endpoint("activemq:queue:newOrder"); Predicate isWidget = xpath("/order/product = 'widget");    from(newOrder) .choice() .when(isWidget).to(widget) .otherwise().to(gadget);

Content Based Router Endpoint newOrder = endpoint("activemq:queue:newOrder"); Predicate isWidget = xpath("/order/product = 'widget"); Endpoint widget = endpoint("activemq:queue:widget"); Endpoint gadget = endpoint("activemq:queue:gadget"); from(newOrder) .choice() .when(isWidget).to(widget) .otherwise().to(gadget);

Content Based Router import org.apache.camel.Endpoint; import org.apache.camel.Predicate; import org.apache.camel.builder.RouteBuilder; public class MyRoute extends RouteBuilder { public void configure() throws Exception { Endpoint newOrder = endpoint("activemq:queue:newOrder"); Predicate isWidget = xpath("/order/product = 'widget"); Endpoint widget = endpoint("activemq:queue:widget"); Endpoint gadget = endpoint("activemq:queue:gadget"); from(newOrder) .choice() .when(isWidget).to(widget) .otherwise().to(gadget) .end(); } } }

Content Based Router: Java DSL import org.apache.camel.builder.RouteBuilder; public class MyRoute extends RouteBuilder { public void configure() throws Exception { from("activemq:queue:newOrder") .choice() .when(xpath("/order/product = 'widget")) .to("activemq:queue:widget") .otherwise() .to("activemq:queue:gadget") .end(); } } }

Content Based Router: XML DSL <route> <from uri="activemq:queue:newOrder"/> <choice> <when> <xpath>/order/product = 'widget'</xpath> <to uri="activemq:queue:widget"/> </when> <otherwise> <to uri="activemq:queue:gadget"/> </otherwise> </choice> </route>

Spring XML <camelContext xmlns="http://camel.apache.org/schema/spring"> <route>  <from uri="stream:in?promptMessage=Enter something: "/>  <transform> <simple>${body.toUpperCase()}</simple> </transform>  <to uri="stream:out"/> </route> </camelContext>

public static void main import org.apache.camel.spring.Main; /** * A main class to run the example from your editor. */ public final class CamelConsoleMain { private CamelConsoleMain() {} public static void main(String[] args) throws Exception { // Main makes it easier to run a Spring application Main main = new Main(); // configure the location of the Spring XML file main.setApplicationContextUri("META-INF/spring/camel-context.xml"); // enable hangup support allows Camel to detect when the JVM is terminated main.enableHangupSupport(); // run and block until Camel is stopped (or JVM terminated) main.run(); } }

Camel Components Task Automation: Timer, Quartz2 Amazon: AWS-CW, AWS-DDB, AWS-SES, AWS-S3 Basics: Bean, Class, Context, Data Format, Exec, Language, Printer Chat: IRC, XMPP Endpoints: Direct, Direct-VM, Disruptor, SEDA ESB: JBI, NMR, Vert.x JMS: ActiveMQ, JMS, RabbitMQ, Stomp, ZeroMQ

What is Apache Camel? An integration framework Supports Enterprise Integration Patterns (EIP) Routing (many DSLs to choose from) Easy Conﬁguration (endpoints as URIs) Just Java, XML, Scala, etc. No Container Dependency Hundreds of components

My Experience First learned about Apache Camel from Bruce Snyder in 2008 at Colorado Software Summit Asked to replace IBM Message Broker last year (2014) for a client

James Strachan Me: “Do you know of any particular guides about migrating from IBM Message Broker to Fuse?” Strachan: “TBH the easiest thing really is to just start using Apache Camel for this kinda stuﬀ” Result: Recommended Apache Camel to my client and went to work

Getting Started Started with a Camel Archetype Used Java 7 Used Java DSL to deﬁne routes Development Strategy 1. Write an integration test against existing service 2. Implement the service with Camel; unit test 3. Copy logic from step 1; write integration test

Legacy Integration Test @Test public void sendGPIRequestUsingSoapApi() throws Exception { SOAPElement bodyChildOne = getBody(message).addChildElement("gpiRequest", "m"); SOAPElement bodyChildTwo = bodyChildOne.addChildElement("args0", "m"); bodyChildTwo.addChildElement("NDC", "ax22").addTextNode("54561237201"); SOAPMessage reply = connection.call(message, getUrlWithTimeout(SERVICE_NAME)); if (reply != null) { Iterator itr = reply.getSOAPBody().getChildElements(); Map resultMap = TestUtils.getResults(itr); assertEquals("66100525123130", resultMap.get("GPI")); } }

Drug Service Implementation @WebService public interface DrugService { @WebMethod(operationName = "gpiRequest") GpiResponse findGpiByNdc(GpiRequest request); }

Spring Conﬁguration @Configuration @ImportResource("classpath:META-INF/cxf/cxf.xml") @ComponentScan("com.raibledesigns.camel") public class CamelConfig extends CamelConfiguration { @Override protected void setupCamelContext(CamelContext camelContext) throws Exception { PropertiesComponent pc = new PropertiesComponent(); pc.setLocation("classpath:application.properties"); camelContext.addComponent("properties", pc); super.setupCamelContext(camelContext); } }

CXF Servlet @Override public void onStartup(ServletContext servletContext) throws ServletException { servletContext.addListener(new ContextLoaderListener(getContext())); ServletRegistration.Dynamic servlet = servletContext.addServlet("CXFServlet", new CXFServlet()); servlet.setLoadOnStartup(1); servlet.setAsyncSupported(true); servlet.addMapping("/api/*"); } private AnnotationConfigWebApplicationContext getContext() { AnnotationConfigWebApplicationContext context = new AnnotationConfigWebApplicationContext(); context.setConfigLocation("com.raibledesigns.camel.config"); return context; } }

Drug Route @Component public class DrugRoute extends RouteBuilder { private String uri = "cxf:/drugs?serviceClass=" + DrugService.class.getName(); @Override public void configure() throws Exception { from(uri) .recipientList(simple("direct:${header.operationName}")); from("direct:gpiRequest").routeId("gpiRequest") .process(new Processor() { public void process(Exchange exchange) throws Exception { // get the ndc from the input String ndc = exchange.getIn().getBody(GpiRequest.class).getNDC(); exchange.getOut().setBody(ndc); } }) .to("sql:{{sql.selectGpi}}") .to("log:output") .process(new Processor() { public void process(Exchange exchange) throws Exception { // get the gpi from the input List<HashMap> data = (ArrayList<HashMap>) exchange.getIn().getBody(); DrugInfo drug = new DrugInfo(); if (data.size() > 0) { drug = new DrugInfo(String.valueOf(data.get(0).get("GPI"))); } GpiResponse response = new GpiResponse(drug); exchange.getOut().setBody(response); } }); } }

Unit Testing Hardest part was ﬁguring out Camel’s testing support Bought Camel in Action, read chapter 6 Went to work Eliminated dependency on datasource Modiﬁed route and intercepted SQL calls

Unit Testing: DrugRouteTests.java @RunWith(CamelSpringJUnit4ClassRunner.class) @ContextConfiguration(loader = CamelSpringDelegatingTestContextLoader.class, classes = CamelConfig.class) @DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD) @UseAdviceWith public class DrugRouteTests { @Autowired CamelContext camelContext; @Produce ProducerTemplate template; @EndpointInject(uri = "mock:result") MockEndpoint result; static List<Map> results = new ArrayList<Map>() {{ add(new HashMap<String, String>() {{ put("GPI", "123456789"); }}); }}; // continued on next slide }

Unit Testing: @Before @Before public void before() throws Exception { camelContext.setTracing(true); ModelCamelContext context = (ModelCamelContext) camelContext; RouteDefinition route = context.getRouteDefinition("gpiRequest"); route.adviceWith(context, new RouteBuilder() { @Override public void configure() throws Exception { interceptSendToEndpoint("sql:*").skipSendToOriginalEndpoint().process(new Processor() { @Override public void process(Exchange exchange) throws Exception { exchange.getOut().setBody(results); } }); } }); route.to(result); camelContext.start(); }

Unit Testing: @Test @Test public void testMockSQLEndpoint() throws Exception { result.expectedMessageCount(1); GpiResponse expectedResult = new GpiResponse(new DrugInfo("123456789")); result.allMessages().body().contains(expectedResult); GpiRequest request = new GpiRequest(); request.setNDC("123"); template.sendBody("direct:gpiRequest", request); MockEndpoint.assertIsSatisfied(camelContext); }

Code Coverage <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>cobertura-maven-plugin</artifactId> <configuration> <instrumentation> <excludes> <exclude>**/model/*.class</exclude> <exclude>**/AppInitializer.class</exclude> <exclude>**/StoredProcedureBean.class</exclude> <exclude>**/SoapActionInterceptor.class</exclude> </excludes> </instrumentation> <check/> </configuration> <version>2.6</version> </plugin> ... <reporting> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>cobertura-maven-plugin</artifactId> <version>2.6</version> </plugin>

Integration Testing public class DrugRouteITest { private static final String URL = "http://localhost:8080/api/drugs"; protected static DrugService createCXFClient() { JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean(); factory.setBindingId("http://schemas.xmlsoap.org/wsdl/soap12/"); factory.setServiceClass(DrugService.class); factory.setAddress(getTestUrl(URL)); return (DrugService) factory.create(); } @Test public void findGpiByNdc() throws Exception { // create input parameter GpiRequest input = new GpiRequest(); input.setNDC("54561237201"); // create the webservice client and send the request DrugService client = createCXFClient(); GpiResponse response = client.findGpiByNdc(input); assertEquals("66100525123130", response.getDrugInfo().getGPI()); } }

Integrating Spring Boot Had to upgrade to Spring 4 Camel 2.13.1 didn’t support Spring 4 Camel 2.14-SNAPSHOT, CXF 3.0 Found issues with camel-test-spring, ﬁxed, created pull request <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> - <camel.version>2.13.1</camel.version> - <cxf.version>2.7.11</cxf.version> - <spring.version>3.2.8.RELEASE</spring.version> + <camel.version>2.14-SNAPSHOT</camel.version> + <cxf.version>3.0.0</cxf.version> + <spring.version>4.0.5.RELEASE</spring.version> </properties> Learn more: http://raibledesigns.com/rd/entry/developing_services_with_apache_camel2

Microservices Deployment http://martinfowler.com/articles/microservices.html

Camel 2.15 includes Spring Boot Support Auto-conﬁguration of Camel context Auto-detects Camel routes Registers key Camel utilities Producer Template Consumer Template Type Converter Connects Spring Boot’s external conﬁguration with Camel properties <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-spring-boot</artifactId> <version>${camel.version}</version> </dependency>

High Performance Scala, Akka and Netty Nice HTML-based Reporting Scenario Recorder Load Testing with Gatling val scn = scenario("Test the Blog entity")  .exec(http("First unauthenticated request")  .get("/api/account")  .headers(headers_http)  .check(status.is(401)))  .pause(10)  .exec(http("Authentication")  .post("/api/authenticate")  .headers(headers_http_authentication)  .formParam("username", "admin")  .formParam("password", "admin")  .check(jsonPath("$.token").saveAs("x_auth_token")))  .pause(1)  .exec(http("Authenticated request")  .get("/api/account")  .headers(headers_http_authenticated)  .check(status.is(200)))  .pause(10)  .repeat(2) {  exec(http("Get all blogs")  .get("/api/blogs")  .headers(headers_http_authenticated)  .check(status.is(200)))  .pause(10 seconds, 20 seconds)  .exec(http("Create new blog")  .put("/api/blogs")  .headers(headers_http_authenticated)  .body(StringBody("""{"id":null, "name":"SAMPLE_TEXT", "handle"

1. Write tests to run against current system. Find the number of concurrent requests that make it fall over. 2. Run tests against new system and tune accordingly. 3. Throttle requests if there are remote connectivity issues with 3rd parties. If I needed to throttle requests, I was planning to use Camel's Throttler. Gatling Approach

Used Gatling’s Recorder Listened on port 8000 Changed DrugServiceTest to use same port Ran integration test Created AbstractSimulation.scala to allow changing parameters My Gatling Strategy

AbstractSimulation.scala import io.gatling.core.scenario.Simulation import io.gatling.http.Predef._ /** * Base Simulation class that allows passing in parameters. */ class AbstractSimulation extends Simulation { val host = System.getProperty("host", "localhost:8080") val serviceType = System.getProperty("service", "modern") val nbUsers = Integer.getInteger("users", 10).toInt val rampRate = java.lang.Long.getLong("ramp", 30L).toLong val httpProtocol = http .baseURL("http://" + host) .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") .doNotTrackHeader("1") .acceptLanguageHeader("en-US,en;q=0.5") .acceptEncodingHeader("gzip, deflate") .userAgentHeader("Gatling 2.0") val headers = Map( """Cache-Control""" -> """no-cache""", """Content-Type""" -> """application/soap+xml; charset=UTF-8""", """Pragma""" -> """no-cache""") }

DrugServiceSimulation.scala import io.gatling.core.Predef._ import io.gatling.http.Predef._ import scala.concurrent.duration._ class DrugServiceSimulation extends AbstractSimulation { val service = if ("modern".equals(serviceType)) "/api/drugs" else "/axis2/services/DrugService" val scn = scenario("Drug Service :: findGpiByNdc") .exec(http(host) .post(service) .headers(headers) .body(RawFileBody("DrugServiceSimulation_request.xml"))) setUp(scn.inject(ramp(nbUsers users) over (rampRate seconds))).protocols(httpProtocol) }

Executing Simulations Legacy service with 100 users over 60 seconds mvn test -Dhost=legacy.server:7802 -Dservice=legacy -Dusers=100 -Dramp=60 Local drug service with 100 users over 30 seconds (defaults used) mvn test -Dusers=100 Results Legacy service started failing at 400 requests per second (rps) Local service started throwing errors at 4000/rps

Data Feeders JDBC Feeder allowed making requests contain unique data for each user ELFileBody allows substituting a ${NDC} variable in XML ﬁle val feeder = jdbcFeeder("jdbc:db2://server:50002/database", "username", "password", "SELECT NDC FROM GENERICS") val scn = scenario("Drug Service") .feed(feeder) .exec(http(host) .post(service) .headers(headers) .body(ELFileBody("DrugServiceSimulation_request.xml")))

Performance Results 100 users over 30 seconds No failures Max response time: 389ms for legacy, 172ms for Camel service 1000 users over 60 seconds Legacy service: 50% of requests failed, avg. response time over 40s New service: all requests succeeded, response mean time 100ms

Monitoring   “a modular web console for managing your Java stuﬀ” Camel plugin shows routes and metrics, route source editable (XML)

Summary Open source wins again! Pleasant experience developing with Apache Camel and Spring Integration tests were essential to verifying functionality Gatling Feeders helped discover edge cases with real data What about Spring Integration? Never looked into it, Camel was good enough

Action! Don’t be afraid to try new things Try Apache Camel if you need Enterprise Integration Spring Boot Rocks! Try it if you’re already using Spring Test, Test, Test, then test some more Gatling is a great way to ﬁnd our performance and test with real data

Contact Information http://raibledesigns.com @mraible Presentations http://slideshare.net/mraible Code http://github.com/mraible Questions?

Additional Information Camel in Action 2nd edition on its way (summer 2016)! Microservices with Spring Boot, Dropwizard, Jetty, CDI Cloud, Docker, Kubernetes, Fabric8, Hawtio Reactive with RxJava and Vert.x Follow its authors: @davsclaus and @jon_anstey http://camel.apache.org

Devoxx4Kids Denver Teaching Kids to Program Java, Minecraft, robots, oh my! Non-proﬁt, looking for speakers! http://www.meetup.com/Devoxx4Kids-Denver/

Developing, Testing and Scaling with Apache Camel - UberConf 2015

More Related Content

What's hot

Viewers also liked

Similar to Developing, Testing and Scaling with Apache Camel - UberConf 2015

More from Matt Raible

Recently uploaded

Developing, Testing and Scaling with Apache Camel - UberConf 2015