[clang] Pass VFS into ASTUnit::LoadFromASTFile() #159166
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
llvmbot added clang 
Member
| @llvm/pr-subscribers-clang Author: Jan Svoboda (jansvoboda11) ChangesThis PR makes the Full diff: https://github.com/llvm/llvm-project/pull/159166.diff 9 Files Affected:
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h index 27bba8e64859d..f66df89aad904 100644 --- a/clang/include/clang/Frontend/ASTUnit.h +++ b/clang/include/clang/Frontend/ASTUnit.h @@ -729,16 +729,15 @@ class ASTUnit { /// \returns - The initialized ASTUnit or null if the AST failed to load. static std::unique_ptr<ASTUnit> LoadFromASTFile( StringRef Filename, const PCHContainerReader &PCHContainerRdr, - WhatToLoad ToLoad, std::shared_ptr<DiagnosticOptions> DiagOpts, + WhatToLoad ToLoad, IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS, + std::shared_ptr<DiagnosticOptions> DiagOpts, IntrusiveRefCntPtr<DiagnosticsEngine> Diags, const FileSystemOptions &FileSystemOpts, const HeaderSearchOptions &HSOpts, const LangOptions *LangOpts = nullptr, bool OnlyLocalDecls = false, CaptureDiagsKind CaptureDiagnostics = CaptureDiagsKind::None, bool AllowASTWithCompilerErrors = false, - bool UserFilesAreVolatile = false, - IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS = - llvm::vfs::getRealFileSystem()); + bool UserFilesAreVolatile = false); private: /// Helper function for \c LoadFromCompilerInvocation() and diff --git a/clang/lib/CrossTU/CrossTranslationUnit.cpp b/clang/lib/CrossTU/CrossTranslationUnit.cpp index 96e28fb10284f..847913d4b03fd 100644 --- a/clang/lib/CrossTU/CrossTranslationUnit.cpp +++ b/clang/lib/CrossTU/CrossTranslationUnit.cpp @@ -577,8 +577,8 @@ CrossTranslationUnitContext::ASTLoader::loadFromDump(StringRef ASTDumpPath) { DiagnosticIDs::create(), *DiagOpts, DiagClient); return ASTUnit::LoadFromASTFile( ASTDumpPath, CI.getPCHContainerOperations()->getRawReader(), - ASTUnit::LoadEverything, DiagOpts, Diags, CI.getFileSystemOpts(), - CI.getHeaderSearchOpts()); + ASTUnit::LoadEverything, CI.getVirtualFileSystemPtr(), DiagOpts, Diags, + CI.getFileSystemOpts(), CI.getHeaderSearchOpts()); } /// Load the AST from a source-file, which is supposed to be located inside the diff --git a/clang/lib/Frontend/ASTMerge.cpp b/clang/lib/Frontend/ASTMerge.cpp index 10c10458466bc..0dff69d613bcf 100644 --- a/clang/lib/Frontend/ASTMerge.cpp +++ b/clang/lib/Frontend/ASTMerge.cpp @@ -47,7 +47,8 @@ void ASTMergeAction::ExecuteAction() { /*ShouldOwnClient=*/true); std::unique_ptr<ASTUnit> Unit = ASTUnit::LoadFromASTFile( ASTFiles[I], CI.getPCHContainerReader(), ASTUnit::LoadEverything, - nullptr, Diags, CI.getFileSystemOpts(), CI.getHeaderSearchOpts()); + CI.getVirtualFileSystemPtr(), nullptr, Diags, CI.getFileSystemOpts(), + CI.getHeaderSearchOpts()); if (!Unit) continue; diff --git a/clang/lib/Frontend/ASTUnit.cpp b/clang/lib/Frontend/ASTUnit.cpp index 8b35af152cbc8..cb445682ac48b 100644 --- a/clang/lib/Frontend/ASTUnit.cpp +++ b/clang/lib/Frontend/ASTUnit.cpp @@ -808,12 +808,13 @@ void ASTUnit::ConfigureDiags(IntrusiveRefCntPtr<DiagnosticsEngine> Diags, std::unique_ptr<ASTUnit> ASTUnit::LoadFromASTFile( StringRef Filename, const PCHContainerReader &PCHContainerRdr, - WhatToLoad ToLoad, std::shared_ptr<DiagnosticOptions> DiagOpts, + WhatToLoad ToLoad, IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS, + std::shared_ptr<DiagnosticOptions> DiagOpts, IntrusiveRefCntPtr<DiagnosticsEngine> Diags, const FileSystemOptions &FileSystemOpts, const HeaderSearchOptions &HSOpts, const LangOptions *LangOpts, bool OnlyLocalDecls, CaptureDiagsKind CaptureDiagnostics, bool AllowASTWithCompilerErrors, - bool UserFilesAreVolatile, IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) { + bool UserFilesAreVolatile) { std::unique_ptr<ASTUnit> AST(new ASTUnit(true)); // Recover resources if we crash before exiting this method. diff --git a/clang/lib/Frontend/FrontendAction.cpp b/clang/lib/Frontend/FrontendAction.cpp index ca37e0661476d..6cc3b65a16cb2 100644 --- a/clang/lib/Frontend/FrontendAction.cpp +++ b/clang/lib/Frontend/FrontendAction.cpp @@ -864,7 +864,8 @@ bool FrontendAction::BeginSourceFile(CompilerInstance &CI, std::unique_ptr<ASTUnit> AST = ASTUnit::LoadFromASTFile( InputFile, CI.getPCHContainerReader(), ASTUnit::LoadPreprocessorOnly, - nullptr, ASTDiags, CI.getFileSystemOpts(), CI.getHeaderSearchOpts()); + CI.getVirtualFileSystemPtr(), nullptr, ASTDiags, CI.getFileSystemOpts(), + CI.getHeaderSearchOpts()); if (!AST) return false; @@ -931,9 +932,9 @@ bool FrontendAction::BeginSourceFile(CompilerInstance &CI, StringRef InputFile = Input.getFile(); std::unique_ptr<ASTUnit> AST = ASTUnit::LoadFromASTFile( - InputFile, CI.getPCHContainerReader(), ASTUnit::LoadEverything, nullptr, - Diags, CI.getFileSystemOpts(), CI.getHeaderSearchOpts(), - &CI.getLangOpts()); + InputFile, CI.getPCHContainerReader(), ASTUnit::LoadEverything, + CI.getVirtualFileSystemPtr(), nullptr, Diags, CI.getFileSystemOpts(), + CI.getHeaderSearchOpts(), &CI.getLangOpts()); if (!AST) return false; diff --git a/clang/tools/c-index-test/core_main.cpp b/clang/tools/c-index-test/core_main.cpp index 25104304322d4..5a3086a7fc08f 100644 --- a/clang/tools/c-index-test/core_main.cpp +++ b/clang/tools/c-index-test/core_main.cpp @@ -275,12 +275,13 @@ static bool printSourceSymbolsFromModule(StringRef modulePath, HeaderSearchOptions HSOpts; + auto VFS = llvm::vfs::getRealFileSystem(); + auto DiagOpts = std::make_shared<DiagnosticOptions>(); IntrusiveRefCntPtr<DiagnosticsEngine> Diags = - CompilerInstance::createDiagnostics(*llvm::vfs::getRealFileSystem(), - *DiagOpts); + CompilerInstance::createDiagnostics(*VFS, *DiagOpts); std::unique_ptr<ASTUnit> AU = ASTUnit::LoadFromASTFile( - modulePath, *pchRdr, ASTUnit::LoadASTOnly, DiagOpts, Diags, + modulePath, *pchRdr, ASTUnit::LoadASTOnly, VFS, DiagOpts, Diags, FileSystemOpts, HSOpts, /*LangOpts=*/nullptr, /*OnlyLocalDecls=*/true, CaptureDiagsKind::None, /*AllowASTWithCompilerErrors=*/true, diff --git a/clang/tools/clang-extdef-mapping/ClangExtDefMapGen.cpp b/clang/tools/clang-extdef-mapping/ClangExtDefMapGen.cpp index ddb2944e3820b..6d8f86b13fa36 100644 --- a/clang/tools/clang-extdef-mapping/ClangExtDefMapGen.cpp +++ b/clang/tools/clang-extdef-mapping/ClangExtDefMapGen.cpp @@ -157,8 +157,8 @@ static bool HandleAST(StringRef AstPath) { std::unique_ptr<ASTUnit> Unit = ASTUnit::LoadFromASTFile( AstPath, CI->getPCHContainerOperations()->getRawReader(), - ASTUnit::LoadASTOnly, DiagOpts, DiagEngine, CI->getFileSystemOpts(), - CI->getHeaderSearchOpts()); + ASTUnit::LoadASTOnly, CI->getVirtualFileSystemPtr(), DiagOpts, DiagEngine, + CI->getFileSystemOpts(), CI->getHeaderSearchOpts()); if (!Unit) return false; diff --git a/clang/tools/libclang/CIndex.cpp b/clang/tools/libclang/CIndex.cpp index 9526f629bda42..6518e7d4d530f 100644 --- a/clang/tools/libclang/CIndex.cpp +++ b/clang/tools/libclang/CIndex.cpp @@ -4180,13 +4180,14 @@ enum CXErrorCode clang_createTranslationUnit2(CXIndex CIdx, FileSystemOptions FileSystemOpts; HeaderSearchOptions HSOpts; + auto VFS = llvm::vfs::getRealFileSystem(); + auto DiagOpts = std::make_shared<DiagnosticOptions>(); IntrusiveRefCntPtr<DiagnosticsEngine> Diags = - CompilerInstance::createDiagnostics(*llvm::vfs::getRealFileSystem(), - *DiagOpts); + CompilerInstance::createDiagnostics(*VFS, *DiagOpts); std::unique_ptr<ASTUnit> AU = ASTUnit::LoadFromASTFile( ast_filename, CXXIdx->getPCHContainerOperations()->getRawReader(), - ASTUnit::LoadEverything, DiagOpts, Diags, FileSystemOpts, HSOpts, + ASTUnit::LoadEverything, VFS, DiagOpts, Diags, FileSystemOpts, HSOpts, /*LangOpts=*/nullptr, CXXIdx->getOnlyLocalDecls(), CaptureDiagsKind::All, /*AllowASTWithCompilerErrors=*/true, /*UserFilesAreVolatile=*/true); diff --git a/clang/unittests/Frontend/ASTUnitTest.cpp b/clang/unittests/Frontend/ASTUnitTest.cpp index 7160453b17daa..dfdbe90e72f1f 100644 --- a/clang/unittests/Frontend/ASTUnitTest.cpp +++ b/clang/unittests/Frontend/ASTUnitTest.cpp @@ -98,7 +98,8 @@ TEST_F(ASTUnitTest, SaveLoadPreservesLangOptionsInPrintingPolicy) { std::unique_ptr<ASTUnit> AU = ASTUnit::LoadFromASTFile( ASTFileName, PCHContainerOps->getRawReader(), ASTUnit::LoadEverything, - DiagOpts, Diags, FileSystemOptions(), HSOpts); + llvm::vfs::getRealFileSystem(), DiagOpts, Diags, FileSystemOptions(), + HSOpts); if (!AU) FAIL() << "failed to load ASTUnit"; |
| static std::unique_ptr<ASTUnit> LoadFromASTFile( | ||
| StringRef Filename, const PCHContainerReader &PCHContainerRdr, | ||
| WhatToLoad ToLoad, std::shared_ptr<DiagnosticOptions> DiagOpts, | ||
| WhatToLoad ToLoad, IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS, |
Collaborator
There was a problem hiding this comment.
I feel like it is better to pass vfs::FileSystem& to avoid ref counting? The caller should hold onto VFS during the function call and no ownership is transferred here.
Contributor Author
There was a problem hiding this comment.
I agree in principle, but ASTUnit is special when it comes to lifetimes and actually typically does outlive the caller. Moreover, the VFS is used to create a FileManager that currently does take the ownership.
cachemeifyoucan approved these changes Sep 16, 2025
Collaborator
| LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/27/builds/16242 Here is the relevant piece of the build log for the reference |
qiongsiwu pushed a commit to qiongsiwu/llvm-project that referenced this pull request Oct 16, 2025
This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()` required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does. (cherry picked from commit cda542d)
jansvoboda11 added a commit to swiftlang/llvm-project that referenced this pull request Oct 18, 2025
This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()` required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does.
jansvoboda11 added a commit to swiftlang/llvm-project that referenced this pull request Oct 23, 2025
This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()` required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does.
qiongsiwu added a commit to swiftlang/llvm-project that referenced this pull request Oct 27, 2025
…Instance Sharing (#11631) * [clang][deps] Remove dependency on `tooling::ToolAction` (llvm#149904) The dependency scanner was initially using a fair amount of infrastructure provided by the `clangTooling` library. Over time, the needs for bespoke handling of command lines grew and the overlap with the tooling library kept shrinking. I don't think the library provides any value anymore. I decided to remove the dependency and only reimplement the small bits required by the scanner. This allowed for a nice simplification, where we no longer need to create temporary dummy `FileManager` instances (mis-named as `DriverFileMgr` in some parts) and `SourceManager` instances to attach to the `DiagnosticsEngine`. That code was copied from the tooling library to support `DiagnosticConsumers` that expect these to exist. The scanner uses a closed set of consumers and none need these objects to exist. The motivation for this (hopefully NFC) patch are some new restrictions to how VFS's can be propagated in Clang that I'm working on. (cherry picked from commit aa1b416) * Reland "[clang] Delay normalization of `-fmodules-cache-path` (llvm#150123)" This reverts commit 613caa9, essentially reapplying 4a4bdde after moving `normalizeModuleCachePath` from clangFrontend to clangLex. This PR is part of an effort to remove file system usage from the command line parsing code. The reason for that is that it's impossible to do file system access correctly without a configured VFS, and the VFS can only be configured after the command line is parsed. I don't want to intertwine command line parsing and VFS configuration, so I decided to perform the file system access after the command line is parsed and the VFS is configured - ideally right before the file system entity is used for the first time. This patch delays normalization of the module cache path until `CompilerInstance` is asked for the cache path in the current compilation context. (cherry picked from commit 55bef46) * NFC: Clean up of IntrusiveRefCntPtr construction from raw pointers. (llvm#151545) Handles clang::DiagnosticsEngine and clang::DiagnosticIDs. For DiagnosticIDs, this mostly migrates from `new DiagnosticIDs` to convenience method `DiagnosticIDs::create()`. Part of cleanup llvm#151026 (cherry picked from commit c7f3437) Conflicts: clang/tools/driver/cc1_main.cpp clang/unittests/Driver/DXCModeTest.cpp clang/unittests/Driver/SimpleDiagnosticConsumer.h clang/unittests/Frontend/SearchPathTest.cpp clang/unittests/Lex/HeaderSearchTest.cpp clang/unittests/Tooling/RewriterTestContext.h * NFC: Clean up of IntrusiveRefCntPtr construction from raw pointers. (llvm#151782) This commit handles the following types: - clang::ExternalASTSource - clang::TargetInfo - clang::ASTContext - clang::SourceManager - clang::FileManager Part of cleanup llvm#151026 (cherry picked from commit 4205da0) Conflicts: clang/lib/Frontend/ASTUnit.cpp clang/lib/Frontend/ChainedIncludesSource.cpp clang/lib/Frontend/CompilerInstance.cpp * Merge commit '30633f308941' from llvm.org/main into next (cherry picked from commit 95ea104) Conflicts: clang/include/clang/Frontend/CompilerInstance.h clang/lib/Frontend/CompilerInstance.cpp * Merge pull request #11450 from swiftlang/jan_svoboda/cas-fix-early-vfs [clang] Fix CAS initialization after upstream llvm#158381 (cherry picked from commit 6d73002) * [clang] Avoid reparsing VFS overlay files for module dep collector (llvm#158372) This PR uses the new-ish `llvm::vfs::FileSystem::visit()` interface to collect VFS overlay entries from an existing `FileSystem` instance rather than parsing the VFS YAML file anew. This prevents duplicate diagnostics as observed by `clang/test/VFS/broken-vfs-module-dep.c`. (cherry picked from commit 4957c47) * [clang] Don't fail `ExecuteCompilerInvocation()` due to caller errors (llvm#158695) This PR changes the behavior of `clang::ExecuteCompilerInvocation()` so that it only returns early when the `DiagnosticsEngine` emitted errors **within** the function. Handling errors emitted before the function got called is a responsibility of the caller. Necessary for llvm#158381. (cherry picked from commit f33fb0d) * [clang] Only set non-empty bypass to scan VFS (llvm#159605) Normalizing an empty modules cache path results in an incorrect non-empty path (the working directory). This PR conditionalizes more code to avoid this. Tested downstream by swift/llvm-project and the `DependencyScanningCAPITests.DependencyScanningFSCacheOutOfDate` unit test. (cherry picked from commit 5a339b0) * Merge commit '0e35f56d40d3' from llvm.org/main into next (cherry picked from commit 3efcc0f) Conflicts: clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp * [clang] NFCI: Clean up `CompilerInstance::create{File,Source}Manager()` (llvm#160748) The `CompilerInstance::createSourceManager()` function currently accepts the `FileManager` to be used. However, all clients call `CompilerInstance::createFileManager()` prior to creating the `SourceManager`, and it never makes sense to use a `FileManager` in the `SourceManager` that's different from the rest of the compiler. Passing the `FileManager` explicitly is redundant, error-prone, and deviates from the style of other `CompilerInstance` initialization APIs. This PR therefore removes the `FileManager` parameter from `createSourceManager()` and also stops returning the `FileManager` pointer from `createFileManager()`, since that was its primary use. Now, `createSourceManager()` internally calls `getFileManager()` instead. (cherry picked from commit b86ddae) Conflicts: clang/lib/Tooling/DependencyScanning/DependencyScannerImpl.cpp * Merge commit '436861645247' from llvm.org/main into next (cherry picked from commit 286ea7d) Conflicts: clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp * [clang] Pass VFS into `ASTUnit::LoadFromASTFile()` (llvm#159166) This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()` required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does. (cherry picked from commit cda542d) * Fix a line missing when merging 30633f3 * [clang][deps] Fix a use-after-free from expanding response files (llvm#164676) In 4368616 we accidentally moved uses of command-line args saved into a bump pointer allocator during response file expansion out of scope of the allocator. Also, the test that should have caught this (at least with asan) was not working correctly because clang-scan-deps was expanding response files itself during argument adjustment rather than the underlying scanner library. rdar://162720059 (cherry picked from commit 3e6f696) --------- Co-authored-by: Jan Svoboda <jan_svoboda@apple.com> Co-authored-by: James Y Knight <jyknight@google.com> Co-authored-by: git apple-llvm automerger <am@git-apple-llvm> Co-authored-by: Ben Langmuir <blangmuir@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This PR makes the
VFSparameter toASTUnit::LoadFromASTFile()required and explicit, rather than silently defaulting to the real file system. This makes it easy to correctly propagate the fully-configured VFS and load any input files like the rest of the compiler does.