Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote · 2024-04-18T23:28:21Z

A third attempt at this; the first attempt was #96724 and the second was #114647.

rustbot · 2024-04-18T23:28:25Z

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

ijackson · 2024-05-10T15:28:11Z

❤️ @nnethercote for working on this. Thank you! I'm not sure if there's a way for me to help, as someone who doesn't really know much about the compiler innards, but please LMK if you think of something.

nnethercote · 2024-05-10T20:45:22Z

@ijackson: thanks! I'm curious why you are interested in this change, given that it's a compiler internals rearrangement?

nnethercote · 2024-05-10T20:47:31Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

ijackson · 2024-05-10T22:40:41Z

@ijackson: Oh, I see, you are interested in #67062 being fixed. Unfortunately my current thoughts are that this PR alone won't be enough to fix that issue, though it's a necessary stepping stone.

Right. It seems ... quite nontrivial. So, thanks.

oriongonza · 2024-05-12T21:49:53Z

After this is done TokenKind will become Copy right?

nnethercote · 2024-05-12T22:35:11Z

After this is done TokenKind will become Copy right?

Yes.

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for the `c2!` macro. - tests/ui/macros/trace_faulty_macros.rs: there is some sub-optimal spacing in the printing of `A { a : a, b : 0, c : _, .. }`, which will be fixed in the next commit. The spacing of `1+1` improves -- it now matches the formatting in the source code. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

…, r=<try> Print `token::Interpolated` with token stream pretty printing. This is a step towards removing `token::Interpolated` (rust-lang#124141). It unavoidably changes the output of the `stringify!` macro, generally for the better. r? `@petrochenkov`

nnethercote · 2024-05-17T06:47:27Z

#125174 carves off a piece of this PR so it can be merged separately.

Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Rollup merge of rust-lang#125236 - dtolnay:expandtest, r=nnethercote Add tests for `-Zunpretty=expanded` ported from stringify's tests This PR adds a new set of tests for the AST pretty-printer. Previously, pretty-printer edge cases were tested by way of `stringify!` in [tests/ui/macros/stringify.rs](https://github.com/rust-lang/rust/blob/1.78.0/tests/ui/macros/stringify.rs), such as the tests added by rust-lang@419b269 and rust-lang@527e2ea. Those tests will no longer provide effective coverage of the AST pretty-printer after rust-lang#124141. `Nonterminal` and `TokenKind::Interpolated` are being removed, and a consequence is that `stringify!` will perform token stream pretty printing, instead of AST pretty printing, in all of the `stringify!` cases including $:expr and all other interpolations. This PR adds 2 new ui tests with `compile-flags: -Zunpretty=expanded`: - **tests/ui/unpretty/expanded-exhaustive.rs** — this test aims for exhaustive coverage of all the variants of `ExprKind`, `ItemKind`, `PatKind`, `StmtKind`, `TyKind`, and `VisibilityKind`. Some parts could use being fleshed out further, but the current state is roughly on par with what exists in the old stringify-based tests. - **tests/ui/unpretty/expanded-interpolation.rs** — this test covers tricky macro metavariable edge cases that require the AST pretty printer to synthesize parentheses in order for the printed code to be valid Rust syntax. r? `@nnethercote`

Instead of using AST pretty printing. This is a step towards removing `token::Interpolated`, which will eventually (in rust-lang#124141) be replaced with a token stream within invisible delimiters. This changes (improves) the output of the `stringify!` macro in some cases. This is allowed. As the `stringify!` docs say: "Note that the expanded results of the input tokens may change in the future. You should be careful if you rely on the output." Test changes: - tests/ui/macros/stringify.rs: this used to test both token stream pretty printing and AST pretty printing via different ways of invoking of `stringify!` (i.e. `$expr` vs `$tt`). But those two different invocations now give the same result, which is a nice consistency improvement. This removes the need for all the `c2*` macros. The AST pretty printer now has more thorough testing thanks to rust-lang#125236. - tests/ui/proc-macro/*: minor improvements where small differences between `INPUT (DISPLAY)` output and `DEEP-RE-COLLECTED (DISPLAY)` output disappear.

petrochenkov · 2024-05-23T20:01:24Z

It's great to see that enum InvisibleOrigin allows to migrate the parser to delimited groups relatively simply, with just the maybe_whole to maybe_reparse_metavar_seq replacement.

Of course it prevents a lot of interesting stuff like reparsing expr as pat and similar, like it would work in a purely token-based model, but all that can be carefully introduced later, when it's possible to do backward compatibly.

petrochenkov · 2024-05-23T20:02:23Z

How hard would it be to get this to a perf run?
(With or without the NtExpr/NtLiteral stuff.)

petrochenkov · 2024-05-23T20:03:04Z

Blocked on #125174.
@rustbot blocked

rust-timer · 2025-04-14T08:23:04Z

Finished benchmarking commit (f836ae4): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	1.0%	[0.3%, 2.1%]	11
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.5%	[-1.4%, -0.2%]	48
Improvements ✅ (secondary)	-1.0%	[-2.4%, -0.3%]	36
All ❌✅ (primary)	-0.2%	[-1.4%, 2.1%]	59

Max RSS (memory usage)

Results (secondary -1.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	1.5%	[0.9%, 2.0%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-7.5%	[-7.5%, -7.5%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (primary -1.3%, secondary 3.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	5.9%	[1.6%, 9.7%]	12
Improvements ✅ (primary)	-1.3%	[-1.3%, -1.3%]	1
Improvements ✅ (secondary)	-3.8%	[-4.5%, -2.2%]	4
All ❌✅ (primary)	-1.3%	[-1.3%, -1.3%]	1

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 780.383s -> 779.942s (-0.06%)
Artifact size: 365.54 MiB -> 365.12 MiB (-0.11%)

It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141.

ast: Remove token visiting from AST visitor It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141. r? `@nnethercote`

Rollup merge of rust-lang#140450 - petrochenkov:vistok, r=nnethercote ast: Remove token visiting from AST visitor It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141. r? `@nnethercote`

…olated_BETA, r=nnethercote [beta] [also fit for beta rollup] [HOTFIX] Don't delay a bug on malformed meta items involving interpolated tokens Directly fixes rust-lang#140612. <details><summary>Outdated Information</summary> Directly fixes the 3 crater regressions reported in rust-lang#137687 (comment) (NB: The containing issue rust-lang#137687 is in fact *not* an instance of these regressions, see rust-lang#137687 (comment)). </details> **Why is this a separate PR for `beta`**? Well, the crater regressions were already fixed on master albeit unintentionally so, namely by PR rust-lang#124141 which we **certainly** don't want to backport! So this is simply a hotfix. PR rust-lang#140584 will then provide the regression test for master, too, so it doesn't get 'lost'. [`@]T-release,` if/once accepted by T-compiler, this PR will be fit for beta rollup and can be cherry-picked. <details><summary>Slightly Outdated & Irrelevant Information</summary> FYI, we may also want to (separately) backport PR rust-lang#140584 to [fix] rust-lang#137687 (which, again, is not an instance of the crater regressions) but it's unclear if it's really necessary (since it's fuzzer-generated and I don't know of any real users who are impacted). </details> cc `@jdonszelmann` r? fmease

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [rust](https://github.com/rust-lang/rust) | minor | `1.86.0` -> `1.87.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>rust-lang/rust (rust)</summary> ### [`v1.87.0`](https://github.com/rust-lang/rust/blob/HEAD/RELEASES.md#Version-1870-2025-05-15) [Compare Source](rust-lang/rust@1.86.0...1.87.0) \========================== <a id="1.87.0-Language"></a> ## Language - [Stabilize `asm_goto` feature](rust-lang/rust#133870) - [Allow parsing open beginning ranges (`..EXPR`) after unary operators `!`, `-`, and `*`](rust-lang/rust#134900). - [Don't require method impls for methods with `Self: Sized` bounds in `impl`s for unsized types](rust-lang/rust#135480) - [Stabilize `feature(precise_capturing_in_traits)` allowing `use<...>` bounds on return position `impl Trait` in `trait`s](rust-lang/rust#138128) <a id="1.87.0-Compiler"></a> ## Compiler - [x86: make SSE2 required for i686 targets and use it to pass SIMD types](rust-lang/rust#135408) <a id="1.87.0-Platform-Support"></a> ## Platform Support - [Remove `i586-pc-windows-msvc` target](rust-lang/rust#137957) Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. [platform-support-doc]: https://doc.rust-lang.org/rustc/platform-support.html <a id="1.87.0-Libraries"></a> ## Libraries - [Stabilize the anonymous pipe API](rust-lang/rust#127154) - [Add support for unbounded left/right shift operations](rust-lang/rust#129375) - [Print pointer metadata in `Debug` impl of raw pointers](rust-lang/rust#135080) - [`Vec::with_capacity` guarantees it allocates with the amount requested, even if `Vec::capacity` returns a different number.](rust-lang/rust#135933) - Most `std::arch` intrinsics which don't take pointer arguments can now be called from safe code if the caller has the appropriate target features already enabled (rust-lang/stdarch#1714, rust-lang/stdarch#1716, rust-lang/stdarch#1717) - [Undeprecate `env::home_dir`](rust-lang/rust#137327) - [Denote `ControlFlow` as `#[must_use]`](rust-lang/rust#137449) - [Macros such as `assert_eq!` and `vec!` now support `const {...}` expressions](rust-lang/rust#138162) <a id="1.87.0-Stabilized-APIs"></a> ## Stabilized APIs - [`Vec::extract_if`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.extract_if) - [`vec::ExtractIf`](https://doc.rust-lang.org/stable/std/vec/struct.ExtractIf.html) - [`LinkedList::extract_if`](https://doc.rust-lang.org/stable/std/collections/struct.LinkedList.html#method.extract_if) - [`linked_list::ExtractIf`](https://doc.rust-lang.org/stable/std/collections/linked_list/struct.ExtractIf.html) - [`<[T]>::split_off`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off) - [`<[T]>::split_off_mut`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off_mut) - [`<[T]>::split_off_first`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off_first) - [`<[T]>::split_off_first_mut`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off_first_mut) - [`<[T]>::split_off_last`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off_last) - [`<[T]>::split_off_last_mut`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.split_off_last_mut) - [`String::extend_from_within`](https://doc.rust-lang.org/stable/alloc/string/struct.String.html#method.extend_from_within) - [`os_str::Display`](https://doc.rust-lang.org/stable/std/ffi/os_str/struct.Display.html) - [`OsString::display`](https://doc.rust-lang.org/stable/std/ffi/struct.OsString.html#method.display) - [`OsStr::display`](https://doc.rust-lang.org/stable/std/ffi/struct.OsStr.html#method.display) - [`io::pipe`](https://doc.rust-lang.org/stable/std/io/fn.pipe.html) - [`io::PipeReader`](https://doc.rust-lang.org/stable/std/io/struct.PipeReader.html) - [`io::PipeWriter`](https://doc.rust-lang.org/stable/std/io/struct.PipeWriter.html) - [`impl From<PipeReader> for OwnedHandle`](https://doc.rust-lang.org/stable/std/os/windows/io/struct.OwnedHandle.html#impl-From%3CPipeReader%3E-for-OwnedHandle) - [`impl From<PipeWriter> for OwnedHandle`](https://doc.rust-lang.org/stable/std/os/windows/io/struct.OwnedHandle.html#impl-From%3CPipeWriter%3E-for-OwnedHandle) - [`impl From<PipeReader> for Stdio`](https://doc.rust-lang.org/stable/std/process/struct.Stdio.html) - [`impl From<PipeWriter> for Stdio`](https://doc.rust-lang.org/stable/std/process/struct.Stdio.html#impl-From%3CPipeWriter%3E-for-Stdio) - [`impl From<PipeReader> for OwnedFd`](https://doc.rust-lang.org/stable/std/os/fd/struct.OwnedFd.html#impl-From%3CPipeReader%3E-for-OwnedFd) - [`impl From<PipeWriter> for OwnedFd`](https://doc.rust-lang.org/stable/std/os/fd/struct.OwnedFd.html#impl-From%3CPipeWriter%3E-for-OwnedFd) - [`Box<MaybeUninit<T>>::write`](https://doc.rust-lang.org/stable/std/boxed/struct.Box.html#method.write) - [`impl TryFrom<Vec<u8>> for String`](https://doc.rust-lang.org/stable/std/string/struct.String.html#impl-TryFrom%3CVec%3Cu8%3E%3E-for-String) - [`<*const T>::offset_from_unsigned`](https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from_unsigned) - [`<*const T>::byte_offset_from_unsigned`](https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.byte_offset_from_unsigned) - [`<*mut T>::offset_from_unsigned`](https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from_unsigned-1) - [`<*mut T>::byte_offset_from_unsigned`](https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.byte_offset_from_unsigned-1) - [`NonNull::offset_from_unsigned`](https://doc.rust-lang.org/stable/std/ptr/struct.NonNull.html#method.offset_from_unsigned) - [`NonNull::byte_offset_from_unsigned`](https://doc.rust-lang.org/stable/std/ptr/struct.NonNull.html#method.byte_offset_from_unsigned) - [`<uN>::cast_signed`](https://doc.rust-lang.org/stable/std/primitive.usize.html#method.cast_signed) - [`NonZero::<uN>::cast_signed`](https://doc.rust-lang.org/stable/std/num/struct.NonZero.html#method.cast_signed-5). - [`<iN>::cast_unsigned`](https://doc.rust-lang.org/stable/std/primitive.isize.html#method.cast_unsigned). - [`NonZero::<iN>::cast_unsigned`](https://doc.rust-lang.org/stable/std/num/struct.NonZero.html#method.cast_unsigned-5). - [`<uN>::is_multiple_of`](https://doc.rust-lang.org/stable/std/primitive.usize.html#method.is_multiple_of) - [`<uN>::unbounded_shl`](https://doc.rust-lang.org/stable/std/primitive.usize.html#method.unbounded_shl) - [`<uN>::unbounded_shr`](https://doc.rust-lang.org/stable/std/primitive.usize.html#method.unbounded_shr) - [`<iN>::unbounded_shl`](https://doc.rust-lang.org/stable/std/primitive.isize.html#method.unbounded_shl) - [`<iN>::unbounded_shr`](https://doc.rust-lang.org/stable/std/primitive.isize.html#method.unbounded_shr) - [`<iN>::midpoint`](https://doc.rust-lang.org/stable/std/primitive.isize.html#method.midpoint) - [`<str>::from_utf8`](https://doc.rust-lang.org/stable/std/primitive.str.html#method.from_utf8) - [`<str>::from_utf8_mut`](https://doc.rust-lang.org/stable/std/primitive.str.html#method.from_utf8\_mut) - [`<str>::from_utf8_unchecked`](https://doc.rust-lang.org/stable/std/primitive.str.html#method.from_utf8\_unchecked) - [`<str>::from_utf8_unchecked_mut`](https://doc.rust-lang.org/stable/std/primitive.str.html#method.from_utf8\_unchecked_mut) These previously stable APIs are now stable in const contexts: - [`core::str::from_utf8_mut`](https://doc.rust-lang.org/stable/std/str/fn.from_utf8\_mut.html) - [`<[T]>::copy_from_slice`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.copy_from_slice) - [`SocketAddr::set_ip`](https://doc.rust-lang.org/stable/std/net/enum.SocketAddr.html#method.set_ip) - [`SocketAddr::set_port`](https://doc.rust-lang.org/stable/std/net/enum.SocketAddr.html#method.set_port), - [`SocketAddrV4::set_ip`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV4.html#method.set_ip) - [`SocketAddrV4::set_port`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV4.html#method.set_port), - [`SocketAddrV6::set_ip`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.set_ip) - [`SocketAddrV6::set_port`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.set_port) - [`SocketAddrV6::set_flowinfo`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.set_flowinfo) - [`SocketAddrV6::set_scope_id`](https://doc.rust-lang.org/stable/std/net/struct.SocketAddrV6.html#method.set_scope_id) - [`char::is_digit`](https://doc.rust-lang.org/stable/std/primitive.char.html#method.is_digit) - [`char::is_whitespace`](https://doc.rust-lang.org/stable/std/primitive.char.html#method.is_whitespace) - [`<[[T; N]]>::as_flattened`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.as_flattened) - [`<[[T; N]]>::as_flattened_mut`](https://doc.rust-lang.org/stable/std/primitive.slice.html#method.as_flattened_mut) - [`String::into_bytes`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.into_bytes) - [`String::as_str`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.as_str) - [`String::capacity`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.capacity) - [`String::as_bytes`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.as_bytes) - [`String::len`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.len) - [`String::is_empty`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.is_empty) - [`String::as_mut_str`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.as_mut_str) - [`String::as_mut_vec`](https://doc.rust-lang.org/stable/std/string/struct.String.html#method.as_mut_vec) - [`Vec::as_ptr`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.as_ptr) - [`Vec::as_slice`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.as_slice) - [`Vec::capacity`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.capacity) - [`Vec::len`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.len) - [`Vec::is_empty`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.is_empty) - [`Vec::as_mut_slice`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.as_mut_slice) - [`Vec::as_mut_ptr`](https://doc.rust-lang.org/stable/std/vec/struct.Vec.html#method.as_mut_ptr) <a id="1.87.0-Cargo"></a> ## Cargo - [Add terminal integration via ANSI OSC 9;4 sequences](rust-lang/cargo#14615) - [chore: bump openssl to v3](rust-lang/cargo#15232) - [feat(package): add --exclude-lockfile flag](rust-lang/cargo#15234) <a id="1.87.0-Compatibility-Notes"></a> ## Compatibility Notes - [Rust now raises an error for macro invocations inside the `#![crate_name]` attribute](rust-lang/rust#127581) - [Unstable fields are now always considered to be inhabited](rust-lang/rust#133889) - [Macro arguments of unary operators followed by open beginning ranges may now be matched differently](rust-lang/rust#134900) - [Make `Debug` impl of raw pointers print metadata if present](rust-lang/rust#135080) - [Warn against function pointers using unsupported ABI strings in dependencies](rust-lang/rust#135767) - [Associated types on `dyn` types are no longer deduplicated](rust-lang/rust#136458) - [Forbid attributes on `..` inside of struct patterns (`let Struct { #[attribute] .. }) =`](rust-lang/rust#136490) - [Make `ptr_cast_add_auto_to_object` lint into hard error](rust-lang/rust#136764) - Many `std::arch` intrinsics are now safe to call in some contexts, there may now be new `unused_unsafe` warnings in existing codebases. - [Limit `width` and `precision` formatting options to 16 bits on all targets](rust-lang/rust#136932) - [Turn order dependent trait objects future incompat warning into a hard error](rust-lang/rust#136968) - [Denote `ControlFlow` as `#[must_use]`](rust-lang/rust#137449) - [Windows: The standard library no longer links `advapi32`, except on win7.](rust-lang/rust#138233) Code such as C libraries that were relying on this assumption may need to explicitly link advapi32. - [Proc macros can no longer observe expanded `cfg(true)` attributes.](rust-lang/rust#138844) - [Start changing the internal representation of pasted tokens](rust-lang/rust#124141). Certain invalid declarative macros that were previously accepted in obscure circumstances are now correctly rejected by the compiler. Use of a `tt` fragment specifier can often fix these macros. - [Don't allow flattened format_args in const.](rust-lang/rust#139624) <a id="1.87.0-Internal-Changes"></a> ## Internal Changes These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - [Update to LLVM 20](rust-lang/rust#135763) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

@cfg

The first rule of the `features` macro looks like this: ``` macro_rules! features { ( @target: $target:ident; @cfg: $cfg:meta; @MACRO_NAME: $macro_name:ident; @MACRO_ATTRS: $(#[$macro_attrs:meta])* $(@BIND_FEATURE_NAME: $bind_feature:tt; $feature_impl:tt; $(#[$deprecate_attr:meta];)?)* $(@NO_RUNTIME_DETECTION: $nort_feature:tt; )* $(@feature: #[$stability_attr:meta] $feature:ident: $feature_lit:tt; $(without cfg check: $feature_cfg_check:literal;)? $(implied by target_features: [$($target_feature_lit:tt),*];)? $(#[$feature_comment:meta])*)* ) => { ``` Notice all the `tt` specifiers. They are used because they are forwarded to another macro. Only `ident`, `lifetime`, and `tt` specifiers can be forwarded this way. But there is an exception: `$feature_lit:tt`, which was added recently. In theory it should cause an error like this: ``` error: no rules expected `literal` metavariable --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:54:91 | 51 | / macro_rules! $macro_name { 52 | | $( 53 | | ($feature_lit) => { 54 | | $crate::detect_feature!($feature, $feature_lit $(, without cfg check: $feature_cfg_check)? ... | | ^^^^^^^^^^^^^^^^^^ no rules expected this token in macro call ... | 88 | | }; 89 | | } | |_________- in this expansion of `is_x86_feature_detected!` | ::: std/tests/run-time-detect.rs:145:27 | 145 | println!("tsc: {:?}", is_x86_feature_detected!("tsc")); | ------------------------------- in this macro invocation | note: while trying to match keyword `true` --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:12:55 | 12 | ($feature:tt, $feature_lit:tt, without cfg check: true) => { | ^^^^ = note: captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens = note: see <https://doc.rust-lang.org/nightly/reference/macros-by-example.html#forwarding-a-matched-fragment> for more information ``` (The URL at the end of the error has more details about this forwarding limitation.) In practice it doesn't cause this error. I'm not sure why, but the existing macro implementation in rustc is far from perfect, so it's believable that it does the wrong thing here. Why does this matter? Because #124141 is modifying the macro implementation, and when that PR is applied the error *does* occur. (It's one of several cases I have found where the existing compiler accepts code it shouldn't, but #124141 causes that code to be rejected.) Fortunately the fix is simple: replace the `literal` specifier with `tt`.

Pkgsrc changes: * Adjust patches to adapt to upstream changes and new versions. * associated checksums Upstream changes relative to 1.87.0: Version 1.88.0 (2025-06-26) ========================== Language -------- - [Stabilize `#![feature(let_chains)]` in the 2024 edition.] (rust-lang/rust#132833) This feature allows `&&`-chaining `let` statements inside `if` and `while`, allowing intermixture with boolean expressions. The patterns inside the `let` sub-expressions can be irrefutable or refutable. - [Stabilize `#![feature(naked_functions)]`.] (rust-lang/rust#134213) Naked functions allow writing functions with no compiler-generated epilogue and prologue, allowing full control over the generated assembly for a particular function. - [Stabilize `#![feature(cfg_boolean_literals)]`.] (rust-lang/rust#138632) This allows using boolean literals as `cfg` predicates, e.g. `#[cfg(true)]` and `#[cfg(false)]`. - [Fully de-stabilize the `#[bench]` attribute] (rust-lang/rust#134273). Usage of `#[bench]` without `#![feature(custom_test_frameworks)]` already triggered a deny-by-default future-incompatibility lint since Rust 1.77, but will now become a hard error. - [Add warn-by-default `dangerous_implicit_autorefs` lint against implicit autoref of raw pointer dereference.] (rust-lang/rust#123239) The lint [will be bumped to deny-by-default] (rust-lang/rust#141661) in the next version of Rust. - [Add `invalid_null_arguments` lint to prevent invalid usage of null pointers.] (rust-lang/rust#119220) This lint is uplifted from `clippy::invalid_null_ptr_usage`. - [Change trait impl candidate preference for builtin impls and trivial where-clauses.] (rust-lang/rust#138176) - [Check types of generic const parameter defaults] (rust-lang/rust#139646) Compiler -------- - [Stabilize `-Cdwarf-version` for selecting the version of DWARF debug information to generate.] (rust-lang/rust#136926) Platform Support ---------------- - [Demote `i686-pc-windows-gnu` to Tier 2.] (https://blog.rust-lang.org/2025/05/26/demoting-i686-pc-windows-gnu/) Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. [platform-support-doc]: https://doc.rust-lang.org/rustc/platform-support.html Libraries --------- - [Remove backticks from `#[should_panic]` test failure message.] (rust-lang/rust#136160) - [Guarantee that `[T; N]::from_fn` is generated in order of increasing indices.] (rust-lang/rust#139099), for those passing it a stateful closure. - [The libtest flag `--nocapture` is deprecated in favor of the more consistent `--no-capture` flag.] (rust-lang/rust#139224) - [Guarantee that `{float}::NAN` is a quiet NaN.] (rust-lang/rust#139483) Stabilized APIs --------------- - [`Cell::update`] (https://doc.rust-lang.org/stable/std/cell/struct.Cell.html#method.update) - [`impl Default for *const T`] (https://doc.rust-lang.org/nightly/std/primitive.pointer.html#impl-Default-for-*const+T) - [`impl Default for *mut T`] (https://doc.rust-lang.org/nightly/std/primitive.pointer.html#impl-Default-for-*mut+T) - [`HashMap::extract_if`] (https://doc.rust-lang.org/stable/std/collections/struct.HashMap.html#method.extract_if) - [`HashSet::extract_if`] (https://doc.rust-lang.org/stable/std/collections/struct.HashSet.html#method.extract_if) - [`proc_macro::Span::line`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.line) - [`proc_macro::Span::column`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.column) - [`proc_macro::Span::start`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.start) - [`proc_macro::Span::end`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.end) - [`proc_macro::Span::file`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.file) - [`proc_macro::Span::local_file`] (https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.local_file) These previously stable APIs are now stable in const contexts: - [`NonNull<T>::replace`] (https://doc.rust-lang.org/stable/std/ptr/struct.NonNull.html#method.replace) - [`<*mut T>::replace`] (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.replace) - [`std::ptr::swap_nonoverlapping`] (rust-lang/rust#137280) - [`Cell::{replace, get, get_mut, from_mut, as_slice_of_cells}`] (rust-lang/rust#137928) Cargo ----- - [Stabilize automatic garbage collection.] (rust-lang/cargo#14287) - [use `zlib-rs` for gzip compression in rust code] (rust-lang/cargo#15417) Rustdoc ----- - [Doctests can be ignored based on target names using `ignore-*` attributes.] (rust-lang/rust#137096) - [Stabilize the `--test-runtool` and `--test-runtool-arg` CLI options to specify a program (like qemu) and its arguments to run a doctest.] (rust-lang/rust#137096) Compatibility Notes ------------------- - [Finish changing the internal representation of pasted tokens] (rust-lang/rust#124141). Certain invalid declarative macros that were previously accepted in obscure circumstances are now correctly rejected by the compiler. Use of a `tt` fragment specifier can often fix these macros. - [Fully de-stabilize the `#[bench]` attribute] (rust-lang/rust#134273). Usage of `#[bench]` without `#![feature(custom_test_frameworks)]` already triggered a deny-by-default future-incompatibility lint since Rust 1.77, but will now become a hard error. - [Fix borrow checking some always-true patterns.] (rust-lang/rust#139042) The borrow checker was overly permissive in some cases, allowing programs that shouldn't have compiled. - [Update the minimum external LLVM to 19.] (rust-lang/rust#139275) - [Make it a hard error to use a vector type with a non-Rust ABI without enabling the required target feature.] (rust-lang/rust#139309)

@cfg

The first rule of the `features` macro looks like this: ``` macro_rules! features { ( @target: $target:ident; @cfg: $cfg:meta; @MACRO_NAME: $macro_name:ident; @MACRO_ATTRS: $(#[$macro_attrs:meta])* $(@BIND_FEATURE_NAME: $bind_feature:tt; $feature_impl:tt; $(#[$deprecate_attr:meta];)?)* $(@NO_RUNTIME_DETECTION: $nort_feature:tt; )* $(@feature: #[$stability_attr:meta] $feature:ident: $feature_lit:tt; $(without cfg check: $feature_cfg_check:literal;)? $(implied by target_features: [$($target_feature_lit:tt),*];)? $(#[$feature_comment:meta])*)* ) => { ``` Notice all the `tt` specifiers. They are used because they are forwarded to another macro. Only `ident`, `lifetime`, and `tt` specifiers can be forwarded this way. But there is an exception: `$feature_lit:tt`, which was added recently. In theory it should cause an error like this: ``` error: no rules expected `literal` metavariable --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:54:91 | 51 | / macro_rules! $macro_name { 52 | | $( 53 | | ($feature_lit) => { 54 | | $crate::detect_feature!($feature, $feature_lit $(, without cfg check: $feature_cfg_check)? ... | | ^^^^^^^^^^^^^^^^^^ no rules expected this token in macro call ... | 88 | | }; 89 | | } | |_________- in this expansion of `is_x86_feature_detected!` | ::: std/tests/run-time-detect.rs:145:27 | 145 | println!("tsc: {:?}", is_x86_feature_detected!("tsc")); | ------------------------------- in this macro invocation | note: while trying to match keyword `true` --> /home/njn/dev/rust3/library/stdarch/crates/std_detect/src/detect/macros.rs:12:55 | 12 | ($feature:tt, $feature_lit:tt, without cfg check: true) => { | ^^^^ = note: captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens = note: see <https://doc.rust-lang.org/nightly/reference/macros-by-example.html#forwarding-a-matched-fragment> for more information ``` (The URL at the end of the error has more details about this forwarding limitation.) In practice it doesn't cause this error. I'm not sure why, but the existing macro implementation in rustc is far from perfect, so it's believable that it does the wrong thing here. Why does this matter? Because rust-lang#124141 is modifying the macro implementation, and when that PR is applied the error *does* occur. (It's one of several cases I have found where the existing compiler accepts code it shouldn't, but rust-lang#124141 causes that code to be rejected.) Fortunately the fix is simple: replace the `literal` specifier with `tt`.

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 18, 2024

nnethercote marked this pull request as draft April 18, 2024 23:28

nnethercote mentioned this pull request Apr 18, 2024

Start removing Nonterminal #114647

Closed

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch 2 times, most recently from 42a623a to c133e16 Compare April 23, 2024 03:51

petrochenkov self-assigned this Apr 28, 2024

nnethercote force-pushed the rm-Nonterminal-and-TokenKind-Interpolated branch from c133e16 to 7aef5db Compare May 16, 2024 10:50

This comment has been minimized.

Sign in to view

nnethercote mentioned this pull request May 17, 2024

Print token::Interpolated with token stream pretty printing. #125174

Merged

This comment was marked as resolved.

Sign in to view

dtolnay mentioned this pull request May 18, 2024

Add tests for -Zunpretty=expanded ported from stringify's tests #125236

Merged

rustbot removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 23, 2024

This was referenced Apr 14, 2025

Include whitespace in "remove |" suggestion and make it hidden #137872

Merged

Actually ensure that TokenKind's fields are safely copyable #139743

Closed

Mark-Simulacrum added the perf-regression-triaged The performance regression has been triaged. label Apr 14, 2025

nnethercote deleted the rm-Nonterminal-and-TokenKind-Interpolated branch April 15, 2025 06:21

nnethercote mentioned this pull request Apr 23, 2025

regression: error: expected one of !, ., ;, ?, {, }, or an operator, found :: #140223

Closed

jieyouxu mentioned this pull request Apr 24, 2025

regression: error: attempted to repeat an expression containing no syntax variables matched as repeating at this depth #140221

Closed

petrochenkov added a commit to petrochenkov/rust that referenced this pull request Apr 29, 2025

ast: Remove token visiting from AST visitor

0c8a204

It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141.

petrochenkov mentioned this pull request Apr 29, 2025

ast: Remove token visiting from AST visitor #140450

Merged

petrochenkov added a commit to petrochenkov/rust that referenced this pull request Apr 29, 2025

ast: Remove token visiting from AST visitor

4e06fcb

It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141.

nnethercote added the relnotes Marks issues that should be documented in the release notes of the next release. label Apr 29, 2025

rustbot mentioned this pull request Apr 29, 2025

Tracking issue for release notes of #124141: Remove Nonterminal and TokenKind::Interpolated (part 2) #140488

Closed

This was referenced Apr 29, 2025

Draft release notes for 1.87 #140133

Closed

Tracking issue for release notes of #124141: Remove Nonterminal and TokenKind::Interpolated (part 1) #140241

Closed

petrochenkov added a commit to petrochenkov/rust that referenced this pull request Apr 30, 2025

ast: Remove token visiting from AST visitor

6668d13

It's no longer necessary after the removal of nonterminal tokens in rust-lang#124141.

jieyouxu mentioned this pull request May 3, 2025

regression: ICE: expr in place where literal is expected (builtin attr parsing) #140612

Closed

Mark-Simulacrum mentioned this pull request May 26, 2025

Draft release notes for 1.88.0 #141598

Closed

petrochenkov mentioned this pull request Jun 4, 2025

Macro docs use term "literal token" which is undefined and confusing rust-lang/reference#1840

Open

petrochenkov mentioned this pull request Sep 26, 2025

Allow passing expr metavariable to cfg #146961

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Uh oh!

nnethercote commented Apr 18, 2024 •

edited

Loading

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 •

edited

Loading

This comment was marked as resolved.

petrochenkov commented May 23, 2024 •

edited

Loading

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

rust-timer commented Apr 14, 2025

Labels

11 participants

Remove Nonterminal and TokenKind::Interpolated #124141

Remove Nonterminal and TokenKind::Interpolated #124141

Uh oh!

Conversation

nnethercote commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

rustbot commented Apr 18, 2024

ijackson commented May 10, 2024

nnethercote commented May 10, 2024

nnethercote commented May 10, 2024

ijackson commented May 10, 2024

oriongonza commented May 12, 2024

nnethercote commented May 12, 2024

This comment has been minimized.

nnethercote commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This comment was marked as resolved.

petrochenkov commented May 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

petrochenkov commented May 23, 2024

petrochenkov commented May 23, 2024

rust-timer commented Apr 14, 2025

Overall result: ❌✅ regressions and improvements - please read the text below

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Labels

11 participants

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

Remove `Nonterminal` and `TokenKind::Interpolated` #124141

nnethercote commented Apr 18, 2024 •

edited

Loading

nnethercote commented May 17, 2024 •

edited

Loading

petrochenkov commented May 23, 2024 •

edited

Loading