Downloaded 263 times
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-11-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-12-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-13-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-14-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-15-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-16-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-17-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-18-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-19-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication
@EnableBinding(Sink.class)
public class DemoSinkApp { @StreamListener(Sink.INPUT)
void receive(Message<String> message) {
System.out.println("Received " + message);
}
public static void main(String[] args) {
SpringApplication.run(DemoSinkApp.class, args);
}
}](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-35-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
public class DemoSourceApp {
@Autowired Source source; @GetMapping void send(@RequestParam String text) {
source.output() .send(MessageBuilder.withPayload(text).build());
}
public static void main(String[] args) {
SpringApplication.run(DemoSourceApp.class, args);
}
}](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-37-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class)
public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() {
return () -> MessageBuilder.withPayload("Hi").build());
}
public static void main(String[] args) {
SpringApplication.run(DemoSourceApp.class, args);
}}](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-38-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication
@EnableBinding(Processor.class)
public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT)
void receive(String text) {
return "[[" + text + "]]";
}
public static void main(String[] args) {
SpringApplication.run(DemoProcessorApp.class, args);
}
}](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-43-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication
@EnableBinding(Processor.class)
public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT)
public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) {
return stream.map(text -> "[[" + text + "]]");
}
public static void main(String[] args) {
SpringApplication.run(DemoProcessorRxApp.class, args);
}
}](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-46-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication
@EnableTask public class DemoTaskApp {
@Bean
CommandLineRunner clr() {
return args -> System.out.println("Task!");
}
public static void main(String[] args) {
SpringApplication.run(DemoTaskApp.class, args);
}
} spring.application.name=hello](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-99-2048.jpg)
![© 2016 Pivotal Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing
@EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory;
@Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()}
public static void main(String[] args) {
SpringApplication.run(DemoTaskApp.class, args);
}} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch](https://image.slidesharecdn.com/2016-11-18springdaysscdf-161118091515/75/Data-Microservices-with-Spring-Cloud-Stream-Task-and-Data-Flow-jsug-springday-101-2048.jpg)
Uploaded byToshiaki Maki
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
The document discusses Spring Cloud Data Flow, which provides a framework for building data pipelines using microservices. It describes how Spring Cloud Data Flow orchestrates long-lived streaming applications using Spring Cloud Stream and short-lived batch processing applications using Spring Cloud Task. It also discusses how these applications can be deployed on modern platforms using deployment-specific Spring Cloud Deployers.
In this document
Powered by AI
Introduction to the presentation and the speaker's background, focusing on Spring Cloud technologies.
Discussion on Spring XD's architecture for data ingestion & processing, including various input/output sources and modern platform integration.
Introduction of modern platforms like Cloud Foundry and Kubernetes, emphasizing a cloud-native redesign approach.
In-depth look at microservices architecture, real-time data processing, and using message brokers.
Overview of the Spring Cloud Data Flow features, including distributed data pipelines and components used.
Details on the architecture of Spring Cloud Data Flow emphasizing deployment strategies using various platforms.
Introduction to Spring Cloud Stream and its capabilities for event-driven microservices.
Examples of using Spring Cloud Stream with Twitter and Cassandra applications.Exploration of binder implementations including RabbitMQ and Apache Kafka for message handling.
Overview of programming models for source, sink, and processor components in Spring Cloud Stream.Introduction to Spring Cloud Stream app starters for diverse input/output sources.Details on creating and deploying streams in Spring Cloud Data Flow with practical commands.
Techniques for managing resources in Spring Cloud Data Flow when deploying applications.
Description of core features such as persistent pub-sub messaging and its importance.
Overview of consumer groups in Spring Cloud Stream and persistent subscriptions for message handling.
Strategies for executing real-time analytics processes using Spring Cloud Data Flow.
Introduction to Spring Cloud Task and its integration with Spring Boot for short-lived tasks.
Details on job executions and management features within Spring Cloud Task.
Illustration of orchestrating tasks using command line in Spring Cloud Data Flow.
Guide for installing and setting up Spring Cloud Data Flow on PCF for development.
Discussion on upcoming features for Spring Cloud Data Flow and harnessing new analytics capabilities.
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
- 1. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
- 2. © 2016 PivotalSoftware, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
- 3. © 2016 PivotalSoftware, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
- 4. © 2016 PivotalSoftware, Inc. All rights reserved. Spring XD (eXtreme Data)
- 5. © 2016 PivotalSoftware, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
- 6. © 2016 PivotalSoftware, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
- 7. © 2016 PivotalSoftware, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
- 8. © 2016 PivotalSoftware, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
- 9. © 2016 PivotalSoftware, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
- 10. © 2016 PivotalSoftware, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
- 11. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
- 12. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
- 13. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
- 14. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
- 15. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
- 16. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
- 17. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
- 18. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
- 19. © 2016 PivotalSoftware, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
- 20. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
- 21. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
- 22. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
- 23. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
- 24. © 2016 PivotalSoftware, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
- 25. © 2016 PivotalSoftware, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
- 26. © 2016 PivotalSoftware, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
- 27. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
- 28. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
- 29. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Stream
- 30. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
- 31. © 2016 PivotalSoftware, Inc. All rights reserved. JJUG CCC in 2 weeks
- 32. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
- 33. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
- 34. © 2016 PivotalSoftware, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
- 35. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication @EnableBinding(Sink.class) public class DemoSinkApp { @StreamListener(Sink.INPUT) void receive(Message<String> message) { System.out.println("Received " + message); } public static void main(String[] args) { SpringApplication.run(DemoSinkApp.class, args); } }
- 36. © 2016 PivotalSoftware, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
- 37. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Autowired Source source; @GetMapping void send(@RequestParam String text) { source.output() .send(MessageBuilder.withPayload(text).build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); } }
- 38. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() { return () -> MessageBuilder.withPayload("Hi").build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); }}
- 39. © 2016 PivotalSoftware, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
- 40. © 2016 PivotalSoftware, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
- 41. © 2016 PivotalSoftware, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
- 42. © 2016 PivotalSoftware, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
- 43. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) void receive(String text) { return "[[" + text + "]]"; } public static void main(String[] args) { SpringApplication.run(DemoProcessorApp.class, args); } }
- 44. © 2016 PivotalSoftware, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
- 45. © 2016 PivotalSoftware, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
- 46. © 2016 PivotalSoftware, Inc. All rights reserved. Reactive API Support @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT) public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) { return stream.map(text -> "[[" + text + "]]"); } public static void main(String[] args) { SpringApplication.run(DemoProcessorRxApp.class, args); } }
- 47. © 2016 PivotalSoftware, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT) public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) { return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group)); }
- 48. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
- 49. © 2016 PivotalSoftware, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
- 50. © 2016 PivotalSoftware, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
- 51. © 2016 PivotalSoftware, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
- 52. © 2016 PivotalSoftware, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
- 53. © 2016 PivotalSoftware, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
- 54. © 2016 PivotalSoftware, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
- 55. © 2016 PivotalSoftware, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
- 56. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
- 57. © 2016 PivotalSoftware, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
- 58. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
- 59. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
- 60. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
- 61. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
- 62. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
- 63. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
- 64. © 2016 PivotalSoftware, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
- 65. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
- 66. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
- 67. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
- 68. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
- 69. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
- 70. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
- 71. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
- 72. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
- 73. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
- 74. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
- 75. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
- 76. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
- 77. © 2016 PivotalSoftware, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
- 78. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average
- 79. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 80. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 81. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 82. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 83. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 84. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
- 85. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
- 86. © 2016 PivotalSoftware, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
- 87. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
- 88. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 89. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 90. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 91. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 92. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
- 93. © 2016 PivotalSoftware, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
- 94. © 2016 PivotalSoftware, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
- 95. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. Demo (Twitter Stream!)
- 96. © 2016 PivotalSoftware, Inc. All rights reserved.
- 97. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Task
- 98. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
- 99. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication @EnableTask public class DemoTaskApp { @Bean CommandLineRunner clr() { return args -> System.out.println("Task!"); } public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); } } spring.application.name=hello
- 100. © 2016 PivotalSoftware, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
- 101. © 2016 PivotalSoftware, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory; @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()} public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
- 102. © 2016 PivotalSoftware, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
- 103. © 2016 PivotalSoftware, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
- 104. © 2016 PivotalSoftware, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
- 105. © 2016 PivotalSoftware, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
- 106. © 2016 PivotalSoftware, Inc. All rights reserved. Task Execution List
- 107. © 2016 PivotalSoftware, Inc. All rights reserved. Job Execution List
- 108. © 2016 PivotalSoftware, Inc. All rights reserved. Job Execution Details
- 109. © 2016 PivotalSoftware, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
- 110. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. Demo (Scalable Tasks)
- 111. © 2016 PivotalSoftware, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
- 112. ‹#›© 2016 PivotalSoftware, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
- 113. © 2016 PivotalSoftware, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
- 114. © 2016 PivotalSoftware, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
- 115. © 2016 PivotalSoftware, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
- 116. © 2016 PivotalSoftware, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
- 117. © 2016 PivotalSoftware, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry