Building resilient scheduling in distributed systems with Spring

Building resilient scheduling in distributed systems with Spring Marek Jeszka @logic_marc

Agenda 一 Background 一 Approach 一 Results 一 Conclusion

Use-case 一 asynchronous communication 一 reliable processing 一 better visibility

@Component public class SimpleService { @Scheduled(cron = "0 * * * * *") public void runQuiteOften() { // process events } }

Distributed Systems 一 vertical scaling • actually it doesn’t scale… 一 horizontal scaling • cost-efficiency • higher reliability • easier to expand

Running on a single node 一 How to select the node? 一 Where to keep information about the selected node?

Instance 1 Instance 2 Instance 3

Instance 1 Instance 2 Instance 3 Am I a leader?

Leader election in Spring-based application

@Component public class SimpleService { @Scheduled(cron = "0 * * * * *") public void runQuiteOften() { // process events } @RunIfLeader

@Retention(RetentionPolicy.RUNTIME) @Target({ ElementType.METHOD }) public @interface RunIfLeader { }

Aspect Oriented Programming with Spring 一 Aspect - crosscutting concern • Logging • Transaction management <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-aop</artifactId> </dependency> 一 Enabled with dependency:

Types of advices 一 After 一 Around 一 Before

@Aspect public class RunIfLeaderAspect { @Around("@annotation(com.n26.RunIfLeader) && execution(void *(..))") public void annotatedMethod(ProceedingJoinPoint joinPoint) throws Throwable { if (isLeader()) { joinPoint.proceed(); } // do not execute }

Why we didn’t like it? 一 No clear separation between business and scheduling logic 一 Hard to test 一 Scheduled jobs spread across the application

Issue with the @SqsListener @SqsListener(value = "eventsQueue", deletionPolicy = ON_SUCCESS) @RunIfLeader void onEvent(String eventAsJson) { // process event }

Selecting a leader in programmatic approach

Programmatic approach 一 SchedulingConfigurer from org.springframework.scheduling.annotation public interface SchedulingConfigurer { void configureTasks( ScheduledTaskRegistrar taskRegistrar); }

@Configuration @EnableScheduling public class SchedulingConfig implements SchedulingConfigurer { @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.addCronTask( new CronTask(() -> { // process events }, "0 * * * * *")); }

@Autowired private Runnable processEventsTask; @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.addCronTask( new CronTask(processEventsTask, "0 * * * * *")); } @Component public class ProcessEventsTask implements Runnable { @Override public void run() { // process events } }

What are the benefits of programmatic approach? 一 Tasks are scheduled in one place 一 Custom executor service

@Configuration @EnableScheduling public class SchedulingConfig implements SchedulingConfigurer { @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.setScheduler(taskScheduler()); } @Bean(destroyMethod = "shutdown") public ExecutorService taskScheduler() { return Executors.newScheduledThreadPool( 4, // pool size new ThreadFactoryBuilder() .setNameFormat("scheduler-thread-%d").build()); } }

What are the benefits of programmatic approach? 一 Tasks are scheduled in one place 一 Custom executor service 一 Convenient testing

@RunWith(MockitoJUnitRunner.class) public class SchedulingConfigTest { @InjectMocks private SchedulingConfig underTest; @Mock private ScheduledTaskRegistrar taskRegistrarMock; @Mock private ProcessEventsTask processEventsTaskMock; @Test public void schedulesCronTask() { underTest.configureTasks(taskRegistrarMock); verify(taskRegistrarMock) .addCronTask(processEventsTaskMock, "0 * * * * *"); }

@RunWith(MockitoJUnitRunner.class) public class SchedulingConfigTest { @InjectMocks private SchedulingConfig underTest; @Mock private ScheduledTaskRegistrar taskRegistrarMock; @Test public void usesScheduledThreadPoolExecutor() { ArgumentCaptor<ScheduledThreadPoolExecutor> captor = forClass(ScheduledThreadPoolExecutor.class); underTest.configureTasks(taskRegistrarMock); verify(taskRegistrarMock).setScheduler(captor.capture()); assertThat(captor.getValue().getCorePoolSize()).isEqualTo(4); }

@Configuration @EnableScheduling public class SchedulingConfig implements SchedulingConfigurer { @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.addCronTask( new CronTask(() -> { if (isLeader()) { // process events } }, "0 * * * * *")); }

@Configuration @EnableScheduling public class SchedulingConfig implements SchedulingConfigurer { @Autowired private Runnable processEventsTask; @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { Runnable leaderAwareTask = new LeaderAwareTaskDecorator(processEventsTask); taskRegistrar.addCronTask( new CronTask(leaderAwareTask, "0 * * * * *")); }

public final class LeaderAwareTaskDecorator implements Runnable { private Runnable delegate; public LeaderAwareTaskDecorator(Runnable delegate) { this.delegate = delegate; } @Override public void run() { if (isLeader()) { delegate.run(); } }

Resiliency 一 What if the response didn’t come? 一 Can we safely repeat? • Duplicate entries created 一 Is the action idempotent? • One or multiple identical requests give the same result

Improvements 一 Distribute the jobs SELECT * FROM events FOR UPDATE SKIP LOCKED;

Further improvements SQS queue Instance 1 Instance 2 Instance 3

What have we learned? 一 Annotation-driven development is hard 一 Keep (code) consistency 一 Increase resilience & predictability 一 Think about observability

References 一 AOP: https://docs.spring.io/spring/docs/2.5.x /reference/aop.html 一 SchedulingConfigurer: https://docs.spring.io/springframework/docs/current/javadoc- api/org/springframework/scheduling/a nnotation/SchedulingConfigurer.html 一 Postgresql select: https://www.postgresql.org/docs/9.5/s ql-select.html

Building resilient scheduling in distributed systems with Spring

More Related Content

What's hot

Similar to Building resilient scheduling in distributed systems with Spring

Recently uploaded

Building resilient scheduling in distributed systems with Spring

Editor's Notes