[C++20 modules] Large module compilations run into repeated header include limits. #127561
Description
I am working on converting the deal.II library (a large and widely used scientific computing library, see https://www.dealii.org) to use modules. deal.II consists of ~600 header and ~400 source files, and contains about 800,000 lines of code. On the plus side, after converting these to interface and implementation partition units, I can compile all but perhaps a dozen of these files. On the downside, most of the ones that fail run into errors that fundamentally look like this:
/usr/bin/clang++-20 -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_SERIALIZATION_DYN_LINK -DBOOST_SERIALIZATION_NO_LIB -DBOOST_SYSTEM_DYN_LINK -DBOOST_SYSTEM_NO_LIB -DBOOST_THREAD_DYN_LINK -DBOOST_THREAD_NO_LIB -DDEBUG -D_GLIBCXX_ASSERTIONS -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -Ddealii_module_debug_EXPORTS -I/home/bangerth/p/deal.II/2/build/include -I/home/bangerth/p/deal.II/2/build/source -I/home/bangerth/p/deal.II/2/cxx-modules/include -isystem /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/algorithms/src -isystem /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/containers/src -isystem /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/core/src -isystem /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/simd/src -isystem /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/tpls/desul/include -isystem /usr/include/trilinos -std=c++20 -fPIC -pedantic -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-deprecated-declarations -Wno-psabi -Wfloat-conversion -Qunused-arguments -Wno-unsupported-friend -Wno-pass-failed -Wno-unused-local-typedefs -openmp-simd -std=c++20 -ffp-exception-behavior=strict -Og -ggdb -Wa,--compress-debug-sections -MD -MT cxx20-modules/CMakeFiles/dealii_module_debug.dir/interface_module_partitions/matrix_free/portable_matrix_free.templates.ccm.o -MF cxx20-modules/CMakeFiles/dealii_module_debug.dir/interface_module_partitions/matrix_free/portable_matrix_free.templates.ccm.o.d @cxx20-modules/CMakeFiles/dealii_module_debug.dir/interface_module_partitions/matrix_free/portable_matrix_free.templates.ccm.o.modmap -o cxx20-modules/CMakeFiles/dealii_module_debug.dir/interface_module_partitions/matrix_free/portable_matrix_free.templates.ccm.o -c /home/bangerth/p/deal.II/2/build/cxx20-modules/interface_module_partitions/matrix_free/portable_matrix_free.templates.ccm remark: source manager location address space usage: [-Rsloc-usage] note: 16065395B (16.07MB) in local locations, 2116191482B (2.12GB) in locations loaded from AST files, for a total of 2132256877B (2.13GB) (99% of available space) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/version.h:1997:54: note: file entered 6016 times using 485683712B (485.68MB) of space plus 11974477B (11.97MB) for macro expansions 1997 | # if (__cplusplus > 202302L) && _GLIBCXX_HOSTED && (__glibcxx_to_chars) | ^ /usr/include/x86_64-linux-gnu/bits/mathcalls.h:433:10: note: file entered 861 times using 12538743B (12.54MB) of space plus 65289138B (65.29MB) for macro expansions 433 | && !__MATH_DECLARING_FLOATN | ^ /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/cmath:1:1: note: file entered 502 times using 48662876B (48.66MB) of space plus 114636B (114.64kB) for macro expansions 1 | // -*- C++ -*- C forwarding header. | ^ /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/core/src/impl/KokkosExp_Host_IterateTile.hpp:2393:5: note: file entered 123 times using 14962581B (14.96MB) of space plus 32871012B (32.87MB) for macro expansions 2393 | #if KOKKOS_ENABLE_NEW_LOOP_MACROS | ^ /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/core/src/Kokkos_CopyViews.hpp:1:1: note: file entered 246 times using 41607456B (41.61MB) of space plus 509097B (509.10kB) for macro expansions 1 | /* | ^ /home/bangerth/p/deal.II/2/cxx-modules/bundled/kokkos-3.7.00/core/src/Kokkos_View.hpp:1:1: note: file entered 492 times using 39372300B (39.37MB) of space plus 2503173B (2.50MB) for macro expansions 1 | /* | ^ [...] I initially though that that must surely be some kind of internal compiler error, but I am no longer sure whether that's the case. /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/version.h is a header file that does not have an include guard against repeated inclusion:
// Copyright (C) 2023-2024 Free Software Foundation, Inc. // This file is part of the GNU ISO C++ Library. This library is free // software; you can redistribute it and/or modify it under the // terms of the GNU General Public License as published by the // Free Software Foundation; either version 3, or (at your option) // any later version. // This library is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // Under Section 7 of GPL version 3, you are granted additional // permissions described in the GCC Runtime Library Exception, version // 3.1, as published by the Free Software Foundation. // You should have received a copy of the GNU General Public License and // a copy of the GCC Runtime Library Exception along with this program; // see the files COPYING3 and COPYING.RUNTIME respectively. If not, see // <http://www.gnu.org/licenses/>. // DO NOT EDIT THIS FILE (version.h) // // It has been AutoGen-ed // From the definitions version.def // and the template file version.tpl /** @file bits/version.h * This is an internal header file, included by other library headers. * Do not attempt to use it directly. @headername{version} */ // Usage guide: // // In your usual header, do something like: // // #define __glibcxx_want_ranges // #define __glibcxx_want_concepts // #include <bits/version.h> // // This will generate the FTMs you named, and let you use them in your code as // if it was user code. All macros are also exposed under __glibcxx_NAME even // if unwanted, to permit bits and other FTMs to depend on them for condtional // computation without exposing extra FTMs to user code. #pragma GCC system_header #include <bits/c++config.h> #if !defined(__cpp_lib_incomplete_container_elements) [...] In other words, it is credible that via the headers directly included by the source file portable_matrix_free.templates.ccm (an interface partition unit) and by the interface partitions that are imported by portable_matrix_free.templates.ccm, we really do reach 6016 repeated inclusions of this file, using the claimed 485683712B (485.68MB) of space plus 11974477B (11.97MB) for macro expansions. At the same time, I also have no way of actually verifying this.
How would I go about providing you with a stripped down example of this error? I could, of course, create a copy of my source directories (containing only this .ccm file, plus the ones it transitively depends on via partition imports) with a stripped down CMake file that reproduces the issue. That might still require a hundred files or more.
Or do you see a better way in which I can help provide testcases? I've got my fair share of experience in dealing with compiler bugs (on a competitor project: https://gcc.gnu.org/bugzilla/buglist.cgi?bug_status=__all__&content=bangerth&no_redirect=1&order=Importance&query_format=specific) and some interest in making sure Clang can actually compile our project, so quite happy to help provide inputs for these sorts of bugs that only show up on non-trivially large code bases.
Thank you in advance!