[clang] Cherry-pick FS sandboxing (pt. 1) #11633
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| @swift-ci please test |
The dependency scanner was initially using a fair amount of infrastructure provided by the `clangTooling` library. Over time, the needs for bespoke handling of command lines grew and the overlap with the tooling library kept shrinking. I don't think the library provides any value anymore. I decided to remove the dependency and only reimplement the small bits required by the scanner. This allowed for a nice simplification, where we no longer need to create temporary dummy `FileManager` instances (mis-named as `DriverFileMgr` in some parts) and `SourceManager` instances to attach to the `DiagnosticsEngine`. That code was copied from the tooling library to support `DiagnosticConsumers` that expect these to exist. The scanner uses a closed set of consumers and none need these objects to exist. The motivation for this (hopefully NFC) patch are some new restrictions to how VFS's can be propagated in Clang that I'm working on.
This PR is part of an effort to remove file system usage from the command line parsing code. The reason for that is that it's impossible to do file system access correctly without a configured VFS, and the VFS can only be configured after the command line is parsed. I don't want to intertwine command line parsing and VFS configuration, so I decided to perform the file system access after the command line is parsed and the VFS is configured - ideally right before the file system entity is used for the first time. This patch delays checking that `model-path` is an existing directory. (cherry picked from commit 2e96cd6)
This PR is part of an effort to remove file system usage from the command line parsing code. The reason for that is that it's impossible to do file system access correctly without a configured VFS, and the VFS can only be configured after the command line is parsed. I don't want to intertwine command line parsing and VFS configuration, so I decided to perform the file system access after the command line is parsed and the VFS is configured - ideally right before the file system entity is used for the first time. This patch delays checking that `ctu-dir` is an existing directory. (cherry picked from commit 1fc090f)
This PR is part of an effort to remove file system usage from the command line parsing code. The reason for that is that it's impossible to do file system access correctly without a configured VFS, and the VFS can only be configured after the command line is parsed. I don't want to intertwine command line parsing and VFS configuration, so I decided to perform the file system access after the command line is parsed and the VFS is configured - ideally right before the file system entity is used for the first time. This patch delays opening the OpenMP host IR file until codegen. (cherry picked from commit a24e11f)
(cherry picked from commit b0f85be)
…50123)" This reverts commit 613caa9, essentially reapplying 4a4bdde after moving `normalizeModuleCachePath` from clangFrontend to clangLex. This PR is part of an effort to remove file system usage from the command line parsing code. The reason for that is that it's impossible to do file system access correctly without a configured VFS, and the VFS can only be configured after the command line is parsed. I don't want to intertwine command line parsing and VFS configuration, so I decided to perform the file system access after the command line is parsed and the VFS is configured - ideally right before the file system entity is used for the first time. This patch delays normalization of the module cache path until `CompilerInstance` is asked for the cache path in the current compilation context. (cherry picked from commit 55bef46)
This is a follow-up to llvm#150124. This PR makes it so that the `-fopenmp-host-ir-file-path` respects VFS overlays, like any other input file. (cherry picked from commit d1c0b1b)
…lvm#158372) This PR uses the new-ish `llvm::vfs::FileSystem::visit()` interface to collect VFS overlay entries from an existing `FileSystem` instance rather than parsing the VFS YAML file anew. This prevents duplicate diagnostics as observed by `clang/test/VFS/broken-vfs-module-dep.c`. (cherry picked from commit 4957c47)
…llvm#158695) This PR changes the behavior of `clang::ExecuteCompilerInvocation()` so that it only returns early when the `DiagnosticsEngine` emitted errors **within** the function. Handling errors emitted before the function got called is a responsibility of the caller. Necessary for llvm#158381. (cherry picked from commit f33fb0d)
Normalizing an empty modules cache path results in an incorrect non-empty path (the working directory). This PR conditionalizes more code to avoid this. Tested downstream by swift/llvm-project and the `DependencyScanningCAPITests.DependencyScanningFSCacheOutOfDate` unit test. (cherry picked from commit 5a339b0)
(cherry picked from commit 95ea104)
[clang] Fix CAS initialization after upstream llvm#158381 (cherry picked from commit 6d73002)
This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()` required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does.
This PR passes the VFS down to `llvm::cl` functions so that they don't assume the real file system.
This PR ensures that the Clang static analyzer loads the config file through the properly-configured VFS rather than through the bare real file system. This enables correctly going through VFS overlays, unifying the behavior with the rest of the compiler.
…lvm#160146) This PR uses the correctly-configured VFS to load the file specified via `-fms-secure-hotpatch-functions-file=`, matching other input files of the compiler.
Some LLVM passes need access to the filesystem to read configuration files and similar. In some places, this is achieved by grabbing the VFS from `PGOOptions`, but some passes don't have access to these and resort to just calling `vfs::getRealFileSystem()`. This PR allows setting the VFS directly on `PassBuilder` that's able to pass it down to all passes that need it.
This PR changes `llvm::FileCollector` to use the `llvm::vfs::FileSystem` API for making file paths absolute instead of using `llvm::sys::fs::make_absolute()` directly. This matches the behavior of the compiler on most other input files.
This PR loads the path from `-fembed-offload-object=<path>` through the VFS rather than going straight to the real file system. This matches the behavior of other input files of the compiler. This technically changes behavior in that `-fembed-offload-object=-` no longer loads the file from stdin, but I don't think that was the intention of the original code anyways.
This PR uses the VFS to create the OpenMP target entry instead of going straight to the real file system. This matches the behavior of other input files of the compiler.
This PR uses the VFS/`FileManager` to check the system framework marker instead of going straight to the real file system. This matches the behavior of other input files of the compiler.
This PR uses the VFS to check `.model` files in the Clang static analyzer to match the compiler's behavior for other input files.
This PR uses the VFS to get the unique file ID when printing externalized decls in CUDA instead of going straight to the real file system. This matches the behavior of other input files of the compiler.
This PR uses the VFS to get the OpenMP entry info instead of going straight to the real file system. This matches the behavior of other input files of the compiler.
This PR starts using the correct VFS for getting file's real path in `FileCollector` instead of using the real FS directly. This matches compiler's behavior for other input files.
This PR starts using the correct VFS in `ModuleDependencyCollector` instead of using the real FS directly. This matches compiler's behavior for other input files.
…m` (llvm#160942) When the root node of the `RedirectingFileSystem` is to be resolved to the current working directory, we previously consulted the real FS instead of the provided underlying VFS. This PR fixes that issue.
This PR starts using the correct VFS in `GCOVProfilerPass` instead of using the real FS directly. This matches compiler's behavior for other input files.
For the redirecting VFS, the `'overlay-relative'` option controls whether external paths should be appended to the overlay directory. This didn't always work as expected: when the overlay file path itself was relative, its absolute path was decided by the real FS, not the underlying VFS, and the resulting external path didn't exist in the underlying VFS. This PR fixes this issue.
The modular build was failing due to a missing include.
This PR virtualizes module cache pruning via the new `ModuleCache` interface. Currently this is an NFC, but I left a FIXME in `InProcessModuleCache` to make this more efficient for the dependency scanner.
The PR llvm#160935 incorrectly replaced `llvm::sys::fs::getUniqueID()` with `llvm::vfs::FileSystem::exists()` in a condition. That's incorrect, since the first function returns `std::error_code` that evaluates to `true` when there is an error (file doesn't exist), while the new code does the opposite. This PR fixes that issue by inverting the conditional. Co-authored-by: ronlieb <ron.lieberman@amd.com>
jansvoboda11 force-pushed the jan_svoboda/stable-21-sandbox-cherry-pick branch from ec81882 to 9530779 Compare jansvoboda11 requested review from JDevlieghere, QuietMisdreavus, cyndyishida and daniel-grumberg as code owners | swiftlang/swift#84995 |
jansvoboda11 force-pushed the jan_svoboda/stable-21-sandbox-cherry-pick branch from 9530779 to 6861e87 Compare | swiftlang/swift#84995 |
…lvm#161459) The `llvm::sys::fs::make_absolute(const Twine &, SmallVectorImpl<char> &)` functions doesn't perform any FS access - it only modifies the second parameter via path/string operations. This function should live in the `llvm::sys::path` namespace for consistency and for making it easier to spot function calls that perform IO.
…)` (llvm#160748) The `CompilerInstance::createSourceManager()` function currently accepts the `FileManager` to be used. However, all clients call `CompilerInstance::createFileManager()` prior to creating the `SourceManager`, and it never makes sense to use a `FileManager` in the `SourceManager` that's different from the rest of the compiler. Passing the `FileManager` explicitly is redundant, error-prone, and deviates from the style of other `CompilerInstance` initialization APIs. This PR therefore removes the `FileManager` parameter from `createSourceManager()` and also stops returning the `FileManager` pointer from `createFileManager()`, since that was its primary use. Now, `createSourceManager()` internally calls `getFileManager()` instead.
…59667) The frontend currently opens the path provided via `-fprofile-instrument-use-path=` to learn the kind of the instrumentation data and set the `CodeGenOptions::ProfileUse` value. This happens during command-line parsing, where we don't have a correctly configured VFS yet, so the behavior is quite different from all other frontend inputs. We need to move this logic out of the frontend command line parsing logic somewhere where we do have the configured VFS. The complication is that the `ProfileUse` flag is being used to set preprocessor macros, and there isn't a great place between command line parsing and preprocessor initialization to perform this logic. This PR solves the issue by deducing the kind of instrumentation data right in the driver and then passing it via a new flag to the frontend. This shouldn't change observable behavior of Clang on the driver level, and only affects the frontend command line interface, which is an implementation detail anyway.
Most `SourceMgr` clients don't make use of include files, but those that do might want to specify the file system to use. This patch enables that by making it possible to pass a `vfs::FileSystem` instance into `SourceMgr`.
jansvoboda11 force-pushed the jan_svoboda/stable-21-sandbox-cherry-pick branch from 6861e87 to c2250e3 Compare | swiftlang/swift#84995 |
| swiftlang/swift#84995 |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
No description provided.