Lex and parse in terms of bytes rather than characters #57
Conversation
| Wow, appreciate the work on this @edsrzf. I definitely think a custom Debug impl would be useful, for convenience. |
| } | ||
| | ||
| pub fn tokenize(&mut self, input: &str) -> Result<Vec<Token>, LexerError> { | ||
| pub fn tokenize<B: ?Sized + AsRef<[u8]>>( |
There was a problem hiding this comment.
Making this generic means it can now accept strings or byte slices.
trunk_lexer/src/lexer.rs Outdated
| }; | ||
| } | ||
| | ||
| fn var<B: ?Sized + AsRef<[u8]>>(v: &B) -> TokenKind { |
There was a problem hiding this comment.
I found it was actually easier to write this as a generic function than a macro, when it needs to deal with both string and byte literals. Alternatively we could always use byte literals, I guess.
There was a problem hiding this comment.
This is fine. The macros were just originally a quick and dirty way of removing repetition, most of them could probably be removed so that the tests are explicit anyway.
| Thoughts on the Is there a reason to prefer doing that instead of pulling in bstr? |
| @edsrzf I'd ideally like to avoid dependencies where possible. Given that this is really just a case of wrapping a byte array / vec, I think we can just handle roll it with a |
| Great work, thanks @edsrzf! |
2 participants
Fixes #26.
I've tried to do as straightforward of conversion as possible so this might be non-optimal in some aspects, but I figure we can do further refactoring later. All tests pass.
One thing that's worse now is test failures. The
Debugimpl forVec<u8>prints things like[50, 51, 52]so it's not very human-readable.We could look at pulling in a dependency like bstr, or otherwise adding a custom
Debugimpl. Let me know what you think and I can either add it onto this PR or do it as a separate one.