Make ast::TokenKind more like lexer::TokenKind #137902
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
`BinOpToken` is badly named, because it only covers the assignable binary ops and excludes comparisons and `&&`/`||`. Its use in `ast::TokenKind` does allow a small amount of code sharing, but it's a clumsy factoring. This commit removes `ast::TokenKind::BinOp{,Eq}`, replacing each one with 10 individual variants. This makes `ast::TokenKind` more similar to `rustc_lexer::TokenKind`, which has individual variants for all operators. Although the number of lines of code increases, the number of chars decreases due to the frequent use of shorter names like `token::Plus` instead of `token::BinOp(BinOpToken::Plus)`. For consistency with `rustc_lexer::TokenKind::Bang`, and because other `ast::TokenKind` variants generally have syntactic names instead of semantic names (e.g. `Star` and `DotDot` instead of `Mul` and `Range`).
rustbot added S-waiting-on-review 
| Some changes occurred in src/tools/rustfmt cc @rust-lang/rustfmt |
compiler-errors approved these changes Mar 3, 2025
| r? compiler-errors @bors r+ rollup |
bors added S-waiting-on-bors 
bors added a commit to rust-lang-ci/rust that referenced this pull request Mar 4, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Mar 4, 2025
Rollup merge of rust-lang#137902 - nnethercote:ast-lexer-TokenKind, r=compiler-errors Make `ast::TokenKind` more like `lexer::TokenKind` This is step 2 of rust-lang/compiler-team#831. r? `@spastorino`
github-actions bot pushed a commit to model-checking/verify-rust-std that referenced this pull request Mar 14, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
flip1995 pushed a commit to flip1995/rust that referenced this pull request Mar 20, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
nnethercote added a commit to nnethercote/rust that referenced this pull request Apr 17, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ``` nnethercote added a commit to nnethercote/rust that referenced this pull request Apr 17, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ``` GuillaumeGomez pushed a commit to GuillaumeGomez/rust that referenced this pull request Apr 18, 2025
…iaskrgr Rollup of 12 pull requests Successful merges: - rust-lang#135767 (Future incompatibility warning `unsupported_fn_ptr_calling_conventions`: Also warn in dependencies) - rust-lang#137852 (Remove layouting dead code for non-array SIMD types.) - rust-lang#137863 (Fix pretty printing of unsafe binders) - rust-lang#137882 (do not build additional stage on compiler paths) - rust-lang#137894 (Revert "store ScalarPair via memset when one side is undef and the other side can be memset") - rust-lang#137902 (Make `ast::TokenKind` more like `lexer::TokenKind`) - rust-lang#137921 (Subtree update of `rust-analyzer`) - rust-lang#137922 (A few cleanups after the removal of `cfg(not(parallel))`) - rust-lang#137939 (fix order on shl impl) - rust-lang#137946 (Fix docker run-local docs) - rust-lang#137955 (Always allow rustdoc-json tests to contain long lines) - rust-lang#137958 (triagebot.toml: Don't label `test/rustdoc-json` as A-rustdoc-search) r? `@ghost` `@rustbot` modify labels: rollup
nnethercote added a commit to nnethercote/rust that referenced this pull request Apr 20, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ``` bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 22, 2025
… r=petrochenkov Remove `token::{Open,Close}Delim` By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ``` r? `@petrochenkov` andrew-otiv pushed a commit to andrew-otiv/rust that referenced this pull request Apr 22, 2025
By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR rust-lang#137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ``` Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
This is step 2 of rust-lang/compiler-team#831.
r? @spastorino