- Notifications
You must be signed in to change notification settings - Fork 13.8k
std: Use libc
for filesystem ops on WASI targets #147572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
alexcrichton wants to merge 1 commit into rust-lang:master Choose a base branch from alexcrichton:wasi-use-libc
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
Open
+751 −635
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
rustbot has assigned @Mark-Simulacrum. Use |
This comment has been minimized.
This comment has been minimized.
This commit is a large change to the implementation of filesystem and other system-related operations on WASI targets. Previously the standard library explicitly used the `wasi` crate at the 0.11.x version track which means that it used WASIp1 APIs directly. This meant that `std` was hard-coded to use WASIp1 syscalls and there was no separate implementation for the WASIp{2,3} targets, for example. The high-level goal of this commit is to decouple this interaction and avoid the use of the `wasi` crate on the WASIp2 target. Historically when WASIp1 was originally added to Rust the wasi-libc library was in a much different position than it is today. Nowadays Rust already depends on wasi-libc on WASI targets for things like memory allocation and environment variable management. As a libc library it also has all the functions necessary to implement all filesystem operations Rust wants. Recently wasi-libc additionally was updated to use WASIp2 APIs directly on the `wasm32-wasip2` target instead of using `wasm32-wasip1` APIs. This commit is leveraging this work by enabling Rust to completely sever the dependence on WASIp1 APIs when compiling for `wasm32-wasip2`. This is also intended to make it easier to migrate to `wasm32-wasip3` internally in the future where now only libc need be updated and Rust doesn't need to explicitly change as well. This commit fairly heavily changes the internals of the implementation of WASI filesystem operations. The basis of implementation is now libc-style APIs as opposed to WASIp1-style-APIs which necessitated a number of changes throughout. Additionally the `std::os::wasi::fs` module has had a few API-breaking changes as well, but this module is also unstable. While this module has been "stable" since inception in the sense that it hasn't changed, this PR is what the unstable status was sort of trying to buy in terms of future flexibility to make changes. For users of stable `std::fs` interfaces this is not intended to be a breaking change but there nevertheless may be minor issues here and there. Concrete changes here are: * `std` for `wasm32-wasip2` no longer depends on `wasi 0.11.x` * The implementation of `std::os::wasi::fs`, which was previously unstable and still is, is now only available on `wasm32-wasip1` and is implemented largely directly in terms of WASIp1 APIs by directly using the `wasi 0.11.x` crate. * `std::os::wasi::fs::MetadataExt`, which was previously unstable and still is, has removed methods that duplicate functionality in `std::fs::Metadata` as there's no point in having them. * `std::os::wasi::fs::FileTypeExt`, which was previously unstable and still is, lost the ability to distinguish between dgram/stream sockets. WASIp1 sockets were never really supported, though, so this isn't much of a loss. * The internal `WasiFd` type has been "gutted" to have far fewer methods internally. This still represents a file descriptor owned by `wasi-libc`, however. * The `std::sys::fs::wasi` implementation is overhauled to largely use the `libc` crate and its APIs directly (which go to wasi-libc). This drastically changes the `ReadDir` implementation, for example. * The `std::sys::fs::wasi::OpenOptions` API now has two ways of opening files. By default `libc::open` is used but of `std::os::wasi::fs::OpenOptionsExt` is used then a WASIp1-specific API is used to open a file. * Types like `FileAttr` and `FileType` now use libc representations at-rest instead of WASIp1 internals. * Various transmutations for iovec-style APIs are consolidated into a WASI-specific module close to the definition of the `IoSlice{,Mut}` types to make it easier to audit unsafe guarantees. * The `wasip2` OS module gained a custom implementation of the `helpers` module as the WASIp1 version of this module is no longer applicable. I'd like to do follow-up PRs after this one to reorganize `std::os::wasi*` a bit to be more amenable for WASIp{2,3} targets as well, but that'll come in a future PR. For now this is tasked with the high-level goal of removing the dependency on WASIp1 APIs in the standard library by relying on wasi-libc to natively use WASIp2 APIs.
c87de05
to cb4a430
Compare Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rustdoc-search Area: Rustdoc's search feature O-wasi Operating system: Wasi, Webassembly System Interface O-wasm Target: WASM (WebAssembly), http://webassembly.org/ S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. T-rustdoc-frontend Relevant to the rustdoc-frontend team, which will review and decide on the web UI/UX output.
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This commit is a large change to the implementation of filesystem and other system-related operations on WASI targets. Previously the standard library explicitly used the
wasi
crate at the 0.11.x version track which means that it used WASIp1 APIs directly. This meant thatstd
was hard-coded to use WASIp1 syscalls and there was no separate implementation for the WASIp{2,3} targets, for example. The high-level goal of this commit is to decouple this interaction and avoid the use of thewasi
crate on the WASIp2 target.Historically when WASIp1 was originally added to Rust the wasi-libc library was in a much different position than it is today. Nowadays Rust already depends on wasi-libc on WASI targets for things like memory allocation and environment variable management. As a libc library it also has all the functions necessary to implement all filesystem operations Rust wants. Recently wasi-libc additionally was updated to use WASIp2 APIs directly on the
wasm32-wasip2
target instead of usingwasm32-wasip1
APIs. This commit is leveraging this work by enabling Rust to completely sever the dependence on WASIp1 APIs when compiling forwasm32-wasip2
. This is also intended to make it easier to migrate towasm32-wasip3
internally in the future where now only libc need be updated and Rust doesn't need to explicitly change as well.This commit fairly heavily changes the internals of the implementation of WASI filesystem operations. The basis of implementation is now libc-style APIs as opposed to WASIp1-style-APIs which necessitated a number of changes throughout. Additionally the
std::os::wasi::fs
module has had a few API-breaking changes as well, but this module is also unstable. While this module has been "stable" since inception in the sense that it hasn't changed, this PR is what the unstable status was sort of trying to buy in terms of future flexibility to make changes. For users of stablestd::fs
interfaces this is not intended to be a breaking change but there nevertheless may be minor issues here and there.Concrete changes here are:
std
forwasm32-wasip2
no longer depends onwasi 0.11.x
std::os::wasi::fs
, which was previously unstable and still is, is now only available onwasm32-wasip1
and is implemented largely directly in terms of WASIp1 APIs by directly using thewasi 0.11.x
crate.std::os::wasi::fs::MetadataExt
, which was previously unstable and still is, has removed methods that duplicate functionality instd::fs::Metadata
as there's no point in having them.std::os::wasi::fs::FileTypeExt
, which was previously unstable and still is, lost the ability to distinguish between dgram/stream sockets. WASIp1 sockets were never really supported, though, so this isn't much of a loss.WasiFd
type has been "gutted" to have far fewer methods internally. This still represents a file descriptor owned bywasi-libc
, however.std::sys::fs::wasi
implementation is overhauled to largely use thelibc
crate and its APIs directly (which go to wasi-libc). This drastically changes theReadDir
implementation, for example.std::sys::fs::wasi::OpenOptions
API now has two ways of opening files. By defaultlibc::open
is used but ofstd::os::wasi::fs::OpenOptionsExt
is used then a WASIp1-specific API is used to open a file.FileAttr
andFileType
now use libc representations at-rest instead of WASIp1 internals.IoSlice{,Mut}
types to make it easier to audit unsafe guarantees.wasip2
OS module gained a custom implementation of thehelpers
module as the WASIp1 version of this module is no longer applicable.I'd like to do follow-up PRs after this one to reorganize
std::os::wasi*
a bit to be more amenable for WASIp{2,3} targets as well, but that'll come in a future PR. For now this is tasked with the high-level goal of removing the dependency on WASIp1 APIs in the standard library by relying on wasi-libc to natively use WASIp2 APIs.