[Parser] Do not eagerly lex keywords #6541
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
| | ||
| std::optional<std::string_view> Lexer::takeKeyword() { | ||
| if (curr) { | ||
| return std::nullopt; |
There was a problem hiding this comment.
Can you explain this part? It reads as "if there is a current token, ignore it and return nullopt". Why is that correct?
There was a problem hiding this comment.
This goes away at the end of the patch sequence, but it is required for correctness at these intermediate steps. Let's say the lexer reaches an integer and eagerly lexes it into a token stored in curr. That integer may be followed by a keyword, but takeKeyword() should fail until takeU32() (or similar) has been called first. Without this check, takeKeyword() would succeed even though the preceeding integer was never consumed.
There was a problem hiding this comment.
I see, thanks. Which is the PR where I can see the final step where it goes away, to get an idea for the direction?
| Looks like this causes the spec tests to hang forever for some reason. Will investigate. |
tlively force-pushed the parser-no-tok-parens branch from 45f9c5a to d1df91c Compare tlively force-pushed the parser-no-tok-keyword branch 2 times, most recently from e47dc44 to b08c1d7 Compare 
Lex them on demand instead to avoid wasted work.
tlively force-pushed the parser-no-tok-keyword branch from b08c1d7 to 5fa865a Compare 
2 participants
Lex them on demand instead to avoid wasted work.