Ensure all runtime_modules loaded before mapping their paths

[naserca]

During a typical recompilation in our codebase, we identified a long wait unaccounted for during logs emitted by mix compile --profile time. You can see the discrepancy here:

$ time mix compile --profile time Compiling 53 files (.ex) [profile] 152ms compiling + 0ms waiting for # super/secret/file.ex # 52 more lines of compiling logs [profile] Finished compilation cycle of 61 modules in 13129ms # very long wait! [profile] Finished group pass check of 4749 modules in 3693ms mix compile --profile time 105.24s user 49.85s system 135% cpu 1:54.71 total

We tracked this down to

elixir/lib/elixir/lib/kernel/parallel_compiler.ex

Lines 275 to 278 in c02ea54

	for module <- runtime_modules,
	path = :code.which(module),
	is_list(path) and path != [],
	do: {module, path}

, where we eventually learned that most of the :code.which/1 calls were taking ~25ms each because the module was not already loaded.

By using the already-optimized :code.ensure_modules_loaded/1, this change cut a huge amount of time during this pre-group-pass-check stage of the compilation process.

$ time mix compile --profile time Compiling 53 files (.ex) [profile] 142ms compiling + 0ms waiting for # super/secret/file.ex # 52 more lines of compiling logs [profile] Finished compilation cycle of 61 modules in 13466ms [profile] Finished group pass check of 4749 modules in 2354ms mix compile --profile time 37.48s user 9.48s system 210% cpu 22.291 total

Implementation notes:

• Considering the current :code.which/1 gracefully filters out non-path return values, I opted to similarly not check (or log) any non-:ok return value from :code.ensure_modules_loaded/1
• Don't think this requires any new tests, but could be wrong!

Miscellanea:

• Yes, ~4700 runtime modules checked after compilation of 61 modules is surprising and something we're working on fixing though I think this fix is useful even with a much smaller number.

[josevalim]

💚 💙 💜 💛 ❤️

[josevalim]

@naserca unfortunately I had to revert this because we cannot assume we can load all modules. The good news is that I believe we have a simpler fix which I will publish soon for you to try out. :)

[josevalim]

I have pushed a fix here: 8f86710 - can you please let me know how it goes? Thank you!

[naserca]

Not unfortunate at all! I didn't really expect for my naive one-liner to be the best approach :) The good news is yes:

I've cherry-picked your commit locally into our current version v1.14.2 (adjusting some given the changes in main) and seeing the same general super-exciting effects on performance during a typical recompilation in our application:

Before:

mix compile 86.53s user 42.16s system 140% cpu 1:31.87 total

After:

mix compile 25.17s user 5.17s system 241% cpu 12.567 total

Huge thanks for taking a look! 💚 💙 💜 💛 ❤️

[josevalim]

Since you patched v1.14 already, could you send a PR backporting it to v1.14? :)

[naserca]

Good idea! https://github.com/elixir-lang/elixir/pull/12294/files