Skip to content

Conversation

kateinoigakukun
Copy link
Member

wasi-libc is a libc implementation widely used in WASI toolchains. This patch adds a new Swift and C module for it to import and use it in Swift code. This module is intentionally separate from the existing Glibc module because wasi-libc is actually based on musl libc.

This PR changed ClangImporter also to inject the modulemap and auxiliary header for wasi-libc into VFS to recognize it as well as Glibc.

@kateinoigakukun
Copy link
Member Author

@swift-ci Please smoke test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@kateinoigakukun kateinoigakukun marked this pull request as ready for review June 11, 2023 07:35
@finagolfin
Copy link
Member

@kateinoigakukun, do you mind updating this to remove the SwiftWASILibc.h header file and just listing the wasi-libc headers in the modulemap, as we discussed before and @egorzhdan has already done with the libstdc++ modulemap?

@finagolfin
Copy link
Member

I just tried switching glibc.modulemap to listing the Bionic headers directly and most everything from the compiler validation suite kept working on my Android phone. However, it broke around 70 C++ Interop tests that worked before with the SwiftGlibc.h header, outputting this error:

<testcase classname="Swift(android-aarch64).Interop/Cxx/class" name="constructors-executable.swift" time="8.35"> <failure><![CDATA[Script: -- : 'RUN: at line 1'; rm -rf "/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp" && mkdir -p "/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp" && /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swiftc -target aarch64-unknown-linux-android -toolchain-stdlib-rpath -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4 -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache /data/data/com.termux/files/home/swift/test/Interop/Cxx/class/constructors-executable.swift -I /data/data/com.termux/files/home/swift/test/Interop/Cxx/class/Inputs/ -Xfrontend -enable-experimental-cxx-interop -o /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out -module-name main && echo /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out && /usr/bin/env LD_LIBRARY_PATH='/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/lib/swift/android:/data/data/com.termux/files/home/build/Ninja-Release/libdispatch-android-aarch64' /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out -- Exit Code: 1 Command Output (stderr): -- <unknown>:0: warning: the '-enable-experimental-cxx-interop' flag is deprecated; please pass '-cxx-interoperability-mode=' instead <unknown>:0: note: Swift will maintain source compatibility for imported APIs based on the selected compatibility mode, so updating the Swift compiler will not change how APIs are imported /data/data/com.termux/files/usr/include/bits/termios_inlines.h:32:10: note: while building module 'std' imported from /data/data/com.termux/files/usr/include/bits/termios_inlines.h:32: #include <errno.h> ^ <module-includes>:2:10: note: in file included from <module-includes>:2: #include "ctype.h" ^ /data/data/com.termux/files/usr/include/c++/v1/ctype.h:38:15: error: cyclic dependency in module 'SwiftGlibc': SwiftGlibc -> std -> SwiftGlibc #include_next <ctype.h> ^ <module-includes>:11:10: note: in file included from <module-includes>:11: #include "pty.h" ^ /data/data/com.termux/files/usr/include/pty.h:38:10: note: in file included from /data/data/com.termux/files/usr/include/pty.h:38: #include <termios.h> ^ /data/data/com.termux/files/usr/include/termios.h:154:10: note: in file included from /data/data/com.termux/files/usr/include/termios.h:154: #include <android/legacy_termios_inlines.h> ^ /data/data/com.termux/files/usr/include/android/legacy_termios_inlines.h:43:10: note: in file included from /data/data/com.termux/files/usr/include/android/legacy_termios_inlines.h:43: #include <bits/termios_inlines.h> ^ /data/data/com.termux/files/usr/include/bits/termios_inlines.h:32:10: error: could not build module 'std' #include <errno.h> ^ <unknown>:0: error: could not build C module 'SwiftGlibc' <module-includes>:1:10: note: in file included from <module-includes>:1: #include "LibcOverlayShims.h" ^ -- ]]></failure> </testcase> 

This has to do with the two different ctype.h headers in libc++ versus Bionic. I then tried removing the ones that forwarded to the C header with #include_next and failed from glibc.modulemap, until I got to an error that the libc++ fenv.h forwarded to the Bionic one, which ultimately included sys/types.h, causing another cyclic dependency error like above, ie having libc++ headers forwarding to libc headers alone causes build issues.

@egorzhdan, how do you work around this: don't have any libc++ headers that forward to libc headers in the C++ modulemap?

The other failure is some oddity related to stdbool.h:

<failure><![CDATA[Script: -- : 'RUN: at line 1'; /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swift-frontend -target aarch64-unknown-linux-android -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4 -define-availability 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -define-availability 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -define-availability 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -define-availability 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -define-availability 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -define-availability 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -define-availability 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -define-availability 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -define-availability 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -define-availability 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -define-availability 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -typo-correction-limit 10 -import-objc-header /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift -disable-llvm-optzns -emit-ir | /data/data/com.termux/files/usr/bin/python3.11 /data/data/com.termux/files/home/swift/utils/PathSanitizingFileCheck --allow-unused-prefixes --sanitize BUILD_DIR=/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64 --sanitize SOURCE_DIR=/data/data/com.termux/files/home/swift --use-filecheck /data/data/com.termux/files/home/build/Ninja-Release/llvm-android-aarch64/bin/FileCheck /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift : 'RUN: at line 4'; /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swift-frontend -target aarch64-unknown-linux-android -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4 -define-availability 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -define-availability 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -define-availability 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -define-availability 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -define-availability 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -define-availability 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -define-availability 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -define-availability 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -define-availability 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -define-availability 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -define-availability 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -typo-correction-limit 10 -import-objc-header /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift -c -- Exit Code: 2 Command Output (stderr): -- /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h:3:17: error: cannot combine with previous 'type-name' declaration specifier typedef uint8_t bool; ^ /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/lib/swift/clang/include/stdbool.h:20:14: note: expanded from macro 'bool' #define bool _Bool ^ /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h:3:1: warning: typedef requires a name typedef uint8_t bool; ^ <unknown>:0: error: failed to import bridging header '/data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h' FileCheck error: '<stdin>' is empty. FileCheck command line: /data/data/com.termux/files/home/build/Ninja-Release/llvm-android-aarch64/bin/FileCheck --allow-unused-prefixes /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift -- ]]></failure> 

@kateinoigakukun, I don't know if you run these C/C++ interop tests on wasm, but everything else in the validation suite worked for me.

I can slap up my libc modulemap changes if you guys are interested in trying it with your libc's, particularly @3405691582 and @mhjacobson with the BSDs and maybe @stevapple with Windows.

@finagolfin
Copy link
Member

I've submitted my glibc modulemap changes in #66665, you may want to try something similar with wasi-libc and see if it works.

@MaxDesiatov MaxDesiatov added the WebAssembly Platform: WebAssembly label Sep 19, 2023
kateinoigakukun and others added 2 commits September 23, 2023 11:51
[wasi-libc](https://github.com/WebAssembly/wasi-libc) is a libc implementation widely used in WASI toolchains. This patch adds a new module named WASILibc. Some extra emulation libraries are also added as wasi_emulated_xxx.
@finagolfin
Copy link
Member

Nice, working well with the headers listed in the module map, @kateinoigakukun?

@egorzhdan, please review.

@kateinoigakukun
Copy link
Member Author

@swift-ci Please smoke test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

Copy link
Contributor

@al45tair al45tair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kateinoigakukun kateinoigakukun merged commit a7d006e into swiftlang:main Sep 25, 2023
@kateinoigakukun kateinoigakukun deleted the katei/patch-series/wasi-libc-module branch September 25, 2023 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WebAssembly Platform: WebAssembly

4 participants