Lexer lookahead refactor

[edsrzf]

This re-works the lexer to use a buffer (peek_buf method) for lookahead instead of a single byte. This allows us to use one big pattern match for most things instead of having sub-branches within each case.

There's definitely a chance I've broken something, but all the existing tests pass.

I've also pulled out some of the larger cases into separate methods and factored out common number tokenization logic.

I really want to re-work how the column and line tracking is done but when I tried I ended up going down a rabbit hole, and will look at that as a separate piece.

[edsrzf]

I kinda wonder what this type of match actually compiles down to, but assume we're not too worried about performance at this stage. It may be worth revisiting later.

It makes the code pretty nice IMHO.

[ryangjchandler]

If you were truly interested, you could look at similar code in https://godbolt.org/ for the assembly output.

[edsrzf]

Yep, I do intend to at some stage. But for now I think I'm happy with the nicer code.

[ryangjchandler]

This is definitely a worthwhile refactor - like you say, performance isn't of huge importance to me right now. Once we've got everything working correctly, performance can be a priority.

Even in it's current state, it's tonnes faster than a PHP implementation.