Implicit trigger dispatch using ScheduledDataLoaderRegistry

[aisensiy]

Hi, I am trying to use dataloader without graphql. I know there is a ScheduledDataLoaderRegistry to automatically call the dispatch in every DataLoader but I still don't know how to use it. The code below seems not making the registry automatically trigger the dispatchAll as expected.

@SpringBootApplication public class DataloaderAndMapstructApplication { public static void main(String[] args) { SpringApplication.run(DataloaderAndMapstructApplication.class, args); } @Bean public ScheduledDataLoaderRegistry registry(UserRepository userRepository) { DispatchPredicate depthOrTimePredicate = DispatchPredicate .dispatchIfDepthGreaterThan(10) .or(DispatchPredicate.dispatchIfLongerThan(Duration.ofMillis(200))); ScheduledDataLoaderRegistry registry = ScheduledDataLoaderRegistry.newScheduledRegistry() .dispatchPredicate(depthOrTimePredicate) .schedule(Duration.ofMillis(40)) .build(); registry.register("userLoader", DataLoaderFactory.newDataLoader((BatchLoader<String, User>) (userIds) -> CompletableFuture.supplyAsync(() -> { return userRepository.findAllById(userIds); }))); return registry; } @Slf4j @Service @RequiredArgsConstructor public static class SchedulingRegistryRunner { private final ScheduledDataLoaderRegistry registry; @Scheduled(fixedDelay = 200) public void run() { log.info("run...."); // When calling 10 times of the dataloader, I still got 10 repository.findByUsername and 2 seconds to finish it. registry.dispatchAll(); } } }

[bbakerman]

ScheduledDataLoaderRegistry registry = ScheduledDataLoaderRegistry.newScheduledRegistry() .dispatchPredicate(depthOrTimePredicate) .schedule(Duration.ofMillis(40)) .build(); 

ScheduledDataLoaderRegistry requires a ScheduledExecutorService to run - this is the repeatable code pump that will periodically invoke the "predicate" to see if there is anything to dispatch.

Out of the box it has private ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); but really you should be deliberater in choosing one.

You should NOT your SchedulingRegistryRunner in this example code. You should not have another periodic code pump callinf dispatch. I mean it should not hurt but its not needed.

In this code example I cant see how you call the data loader. I guess its registry.getDataLoader("userLoader") somewhere.

Also note that re-using data loader instances between requests can lead to security problems - eg request A loads data X and then later request B will see that (potentially) cached data - which may be allowed or may not.

[aisensiy]

In this code example I cant see how you call the data loader.

Thanks for your reply. And I am trying to use mapstruct and dataloader together to make it easier to translate domain object to DTO.

// ============================ Domain =================== @Entity @Getter @NoArgsConstructor(access = AccessLevel.PROTECTED) public class Comment { @Id @GeneratedValue private Long id; private String content; private Long articleId; private String authorId; private Instant createdAt; } @Entity @Getter @NoArgsConstructor(access = AccessLevel.PROTECTED) public class User { @Id private String username; private String email; private Instant createdAt; } // ============================ DTO =================== @Data public class CommentDTO { private Long id; private String content; private Instant createdAt; private UserDTO author; } @Data public class UserDTO { String username; String email; } // ============================ Mapper =================== @Mapper(componentModel = "spring") public abstract class UserDTOMapper { private @Autowired DataLoaderRegistry dataLoaderRegistry; abstract UserDTO toDTO(User user); @SneakyThrows User toEntity(String id) { DataLoader<String, User> userDataLoader = dataLoaderRegistry.getDataLoader("userLoader"); return userDataLoader.load(id).get(); // <-------- This is wrong, this will stuck the thread } } @Mapper(componentModel = "spring", uses = {UserDTOMapper.class}) public abstract class CommentDTOMapper { @Autowired private DataLoaderRegistry registry; @Mapping(source = "authorId", target = "author") abstract CommentDTO commentToCommentDTO(Comment comment); abstract List<CommentDTO> map(List<Comment> comments); }

I know I am wrong to call dataloader.load(xx).get immediately. But I still do not have a good idea to make this work.

[bbakerman]

DataLoader is about 2 things - caching results and BATCHING a set of individual loads

In order to make a batch you need some time window to decide how long a batch is? eg do we wait 5 ms and then all outstanding dataloader calls are dispatched then?

or in grpahql do we dispatcher per field level say?

This is how batching works - combine N calls into one batch call on some boundary of choice.

userDataLoader.load(id).get(); 

This will never work if not one ever dispatches. But if should work if your dataLoaderRegistry is being dispatched periodically say. You need to make sure thats the case.