Skip to content

Conversation

albertzaharovits
Copy link
Contributor

@albertzaharovits albertzaharovits commented Jan 16, 2025

Highlights:

  • there is one merge scheduler per shard
  • for IO throttling, across shards, this uses static variables (the set of "active" merges)
  • there is no merge size cutoff for IO throttling, all merges are IO throttled (except for forced merges, and merges when the shard is closing)
  • the max merge threads setting is respected per scheduler (per shard)
  • merges are ordered by size per scheduler (per shard), smaller ones are handled first
  • new merges throttling is not implemented (i.e. to prevent new merges comming in), just some hooks are exposed (that the Internal Engine implements)
@albertzaharovits albertzaharovits force-pushed the threadpool-merge-scheduler branch from b0423c4 to c0667bf Compare January 18, 2025 17:02
@albertzaharovits albertzaharovits added v8.18.0 :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Jan 20, 2025
@albertzaharovits
Copy link
Contributor Author

We've ultimately merged a different alternative: #120869

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. v8.19.0 v9.1.0 WIP

2 participants