feat(parser): Don't error on frontmatter

[epage]

The hope is this will improve the editor experience for cargo scripts even if there isn't full support yet. This is a first step towards #15318.

This copies over Cargo's frontmatter parser, rather than using rustc's because rustc's is embedded in the tokenizer and requires the lexer to be stateful. r-a intentionally prevents the tokenizer from being stateful.

 // Re-create the tokenizer from scratch every token because `GuardedStrPrefix` is one token in the lexer // but we want to split it to two in edition <2024.

Edits to Cargo's frontmatter parser were kept to a minimum with documentation linking it back to the source, for easier editing.

Cargo could expose this as a crate but not feeling up for taking on a new crate atm and we could get into a weird position with release cycles as most internal crates follow the release train, being published on stabilization.

Cargo's frontmatter parser counts the end of a shebang slightly differently than rustc_lexer, causing some other tests to be updated. Cargo's behavior more closely represents the Reference. While that is non-normative, I think this representation makes more logical sense.

[ShoyuVanilla]

I believe that the implementation is good but I'm not totally confident with this "extra parser for frontmatter" design choice(I think it would be okay though). @rust-lang/rust-analyzer sorry for the ping, but do you have any ideas?

[Veykril]

Yea I think this is fine. I don't think lifting this out of cargo as a crate is worth it either (even if there is a risk of "desyncing" here (maybe the cargo side could get a comment saying that changes there should ideally be mirrored to r-a). Either way I think this is a decent way to handle this for now especially given that this is a non-invasive change 👍

Thanks Ed!

[lnicola]

changelog feat (first contribution) parse script frontmatter