Native multi-threading backend #382
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
…ng a new backend Multithreading. In your `config.ini`, set `backend=Multithreading` instead of `Multiprocessing` under `[vec]` to take advantage of it. For envs that are CPU deep (i.e. envs that do a lot of computation per `c_step`) but not GPU wide (i.e. not too many params), this offers a nice speed up anywhere from 1.1x to 3x as fast. For envs that have a very few CPU cycles as part of `c_step`, this may not provide a good speedup and in fact may be slower (see below for results). You can also limit the max number of threads say `max_num_threads=4` under `[vec]` if your env is very short and still take advantage of native multi-threading. env SPS BEFORE AFTER (MT only) AFTER (torch pin) rlplays 44K 112K (^ my env) go 520K 690K pacman 800K 890K drone_swarm 1.1M 940K (*) enduro 720K 520K (*) terraform 370K 380K (**) (*) For some envs, it's better to do Multiprocessing or even Serial with a worker just going through the envs one by one rather than shard it across too many envs. I verified this match the perf when limiting num_threads to be a small number (or just try plain Serial) rather than spread to all cores. (**) GPU bound. Most time spent copying/learning.
rlplays commented Oct 18, 2025
| | ||
| | ||
| // TODO(perumaal): Should this be multi-thread aware as well? (see vec_step below). | ||
| // Main issue is that srand is not thread-safe. But do we care? |
Author
There was a problem hiding this comment.
It feels like we should have a puffer_rnd or something like that so it's TLS aware. But envs may be complex that use rnd a lot. I am not sure if it matters though?
rlplays changed the title added 10 commits November 4, 2025 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
To use this: In your
config.ini, setbackend=Multithreadinginstead ofMultiprocessingunder[vec]to take advantage of it. For envs that are CPU deep (i.e. envs that do a lot of computation perc_step) but not GPU wide (i.e. not too many params), this offers a nice speed up anywhere from 1.1x to 3x as fast.For envs that have a very few CPU cycles as part of
c_step, this may not provide a good speedup and in fact may be slower (see below for results). You can also limit the max number of threads saymax_num_threads=4under[vec]if your env is very short and still take advantage of native multi-threading.Performance Comparison: Multithreading vs. Multiprocessing
Notes:
(*) For some envs, it's better to do Multiprocessing or even Serial with a worker just going through the envs one by one rather than shard it across too many threads. I verified this matches the perf when limiting
max_num_threadsto be a small number (or even 0 to force Serial mode) rather than spread to all cores.(**) GPU bound. Most time spent copying/learning.