Apple: Always pass SDK root when linking with cc, and pass it via SDKROOT env var

[madsmtm]

Fixes #80817, fixes #96943, and generally simplifies our linker invocation on Apple platforms.

Part of #129432.

Necessary background on trampoline binaries

The developer binaries such as /usr/bin/cc and /usr/bin/clang are actually trampolines (similar in spirit to the Rust binaries in ~/.cargo/bin) which effectively invokes xcrun to get the current Xcode developer directory, which allows it to find the actual binary under /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/*.

This binary is then launched with the following environment variables set (but none of them are set if SDKROOT is set explicitly):

• SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
• LIBRARY_PATH=/usr/local/lib (appended)
• CPATH=/usr/local/include (appended)
• MANPATH=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/share/man:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/usr/share/man:/Applications/Xcode.app/Contents/Developer/usr/share/man:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/share/man: (prepended)

This allows the user to type e.g. clang foo.c in their terminal on macOS, and have it automatically pick up a suitable Clang binary and SDK from either an installed Xcode.app or the Xcode Command Line Tools.
(It acts roughly as-if you typed xcrun -sdk macosx clang foo.c).

Finding a suitable SDK

All compilation on macOS is cross-compilation using SDKs, there are no system headers any more (/usr/include is gone), and the system libraries are elsewhere in the file system (/usr/lib is basically empty). Instead, the logic for finding the SDK is handled by the /usr/bin/cc trampoline (see above).

But relying on the cc trampoline doesn't work when:

• Cross-compiling, since a different SDK is needed there.
• Invoking the linker directly, since the linker doesn't understand SDKROOT.
• Linking build scripts inside Xcode (see Linker errors when cross-compiling from Xcode on Big Sur #80817), since Xcode prepends /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin to PATH, which means cc refers to the actual Clang binary, and we end up with the wrong SDK root specified.

Basically, we cannot rely on the trampoline at all, so the last commit removes the special-casing that was done when linking with cc for macOS (i.e. the most common path), so that we now always invoke xcrun (if SDKROOT is not explicitly specified) to find the SDK root.

Making sure this is non-breaking has a few difficulties though, namely that the user might not have Xcode installed, and that the compiler driver may not understand the -isysroot flag. These difficulties are explored below.

No Xcode

There are several compiler drivers which work without Xcode by bundling their own SDK, including zig cc, Nixpkgs' clang and Homebrew's llvm package. Additionally, xcrun is rarely available when cross-compiling from non-macOS and instead the user might provide a downloaded SDK manually with -Clink-args=....

We do still want to try to invoke xcrun if possible, since it is usually the SDK that the user wants (and if not, the environment should override xcrun, such as is done by Nixpkgs). But we do not want failure to invoke xcrun to stop the linking process. This is changed in the second-to-last commit.

SDKROOT vs. -isysroot

The exact reasoning why we do not always pass the SDK root when linking on macOS eludes me (the git history dead ends in #100286), but I suspect it's because we want to support compiler drivers which do not support the -isysroot option.

To make sure that such use-cases continue to work, we now pass the SDK root via the SDKROOT environment variable. This way, compiler drivers that support setting the SDK root (such as Clang and GCC) can use it, while compiler drivers that don't (presumably because they figure out the SDK in some other way) can just ignore it.

One small danger here would be if there's some compiler driver out there which works with the -isysroot flag, but not with the SDKROOT environment variable. I am not aware of any?

In a sense, this also shifts the blame; if a compiler driver does not understand SDKROOT, it won't work with e.g. xcrun -sdk macosx15.0 $tool either, so it can more clearly be argued that this is incorrect behaviour on the part of the tool.

Note also that this overrides the behaviour discussed above (/usr/bin/cc sets some extra environment variables), I will argue that is fine since MANPATH and CPATH is useless when linking, and /usr/local/lib is empty on a default system at least since macOS 10.14 (it might be filled by extra libraries installed by the user, but I'll argue that if we want it to be part of the default library search path, we should set it explicitly so that it's also set when linking with -Clinker=ld).

Considered alternatives

• Invoke /usr/bin/cc instead of cc.
  • This breaks many other use-cases though where overriding cc in the PATH is desired.
• Look up which cc, and do special logic if in Xcode toolchain.
  • Seems brittle, and besides, it's not the cc in the Xcode toolchain that's wrong, it's the /usr/bin/cc behaviour that is a bit too magical.
• Invoke xcrun --sdk macosx cc.
  • This completely ignores SDKROOT, so we'd still have to parse that first to figure out if it's suitable or not, but would probably be workable.
• Maybe somehow configure the linker with extra flags such that it'll be able to link regardless of linking for macOS or e.g. iOS? Though I doubt this is possible.
• Bundle the SDK, similar to zig-cc.
  • Comes with it's own host of problems.

Testing

Tested that this works with the following -Clinker=...:

• Default (cc)
• /usr/bin/ld
• Actual Clang from Xcode (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang)
• /usr/bin/clang (invoked via clang instead of cc)
• Homebrew's llvm package (ignores SDKROOT, uses their own SDK)
• Homebrew's gcc package (SDKROOT is preferred over their own SDK)
• Macports clang Couldn't get it to build
• Macports gcc (SDKROOT is preferred over their own SDK)
• Zig CC installed via. homebrew (ignores both -isysroot and SDKROOT, uses their own SDK)
• Nixpkgs clang (ignores SDKROOT, uses their own SDK)
• Nixpkgs gcc (ignores SDKROOT, uses their own SDK)
• cosmocc? Doesn't accept common flags (like -arch)

CC @BlackHoleFox @thomcc

[rustbot]

r? @nnethercote

rustbot has assigned @nnethercote.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

These commits modify compiler targets.
(See the Target Tier Policy.)

[nnethercote]

This looks reasonable, but I don't really have the first idea about any of this stuff and that means I'm not a great reviewer for it. @thomcc, are you able to take a look? I can rubber-stamp it if you are happy.

One question: there are no tests in this PR or in #131433. Should there be?

[rustbot]

This PR modifies tests/run-make/. If this PR is trying to port a Makefile
run-make test to use rmake.rs, please update the
run-make port tracking issue
so we can track our progress. You can either modify the tracking issue
directly, or you can comment on the tracking issue and link this PR.

cc @jieyouxu

[madsmtm]

One question: there are no tests in this PR or in #131433. Should there be?

There are a few unit tests under compiler/rustc_codegen_ssa/src/back/apple/tests.rs, but your point stands, there are no integration tests for this.

The main reason for that is that this is really hard to write a robust test for, since this is so heavily environment specific - but I guess a test that adds the Xcode paths to PATH as a regression test for #80817 would be doable, I've done that and rebased, thanks for the nudge to do it ;).

[rustbot]

Some changes occurred in src/doc/rustc/src/platform-support

cc @Noratrieb

[thomcc]

I approve of the macOS specific parts of this. I'm not a compiler reviewer so I haven't done any review for style/etc. That said that part does look fine to me. It's a great PR overall. Great PR description too, clarified some things I had always wondered about.

[nnethercote]

Thanks, @thomcc. Once #131433 is merged I'm happy for this to merge.

@bors delegate=madsmtm

[aeubanks]

hi, in Chromium this PR caused https://crbug.com/442128549 where rustc was running xcrun when we didn't want it to. we ended up passing the SDK root via the SDKROOT env var, but that's not the nicest way to pass the SDK root path because (1) it's an environment variable as opposed to a real rustc flag, and (2) the environment variable only allows an absolute path which isn't good for build systems that expect everything to be a relative path. is it possible to add a flag like clang's -isysroot to rustc?

[madsmtm]

Hmm, I really think that SDKROOT is really the right way to go about it, it interacts much better with nested invocations (such as here where cargo is invoking rustc which is invoking the linker).

I'm not adamant about not adding a flag for this, but I'd like to avoid it if possible. I'd much rather that we accepted relative paths here (in both rustc and clang).

[nico]

For what it's worth, over in LLVM land, flags are preferred over env vars for these reasons:

1. It makes it easier to reproduce behavior of subcommands by pasting -v (or whatever the equivalent is) lines from bots
2. Tools like ninja track changes in command line flags, but not changes in envs
3. For flags, it's pretty clear that relative paths are resolved against cwd at program start, while it's less clear for env vars. (Granted, most toolchain-y tools don't change cwd while they're running.)

[madsmtm]

I looked into allowing SDKROOT to be relative, but xcrun doesn't seem to support it (it errors with xcodebuild: error: SDK "./Foo.sdk" cannot be located.), so that's probably a no-go.

Have opened #146345 for discussing/tracking adding a flag to help with your use-case.