PHP Parser VS tree-sitter

Once rector gets 8.4 rules out, this will be pretty awesome:

https://github.com/rectorphp/rector/issues/8701

https://github.com/nikic/PHP-Parser/commit/7b0384cdbe03431c4...

heh, I'll be straight that I'm sad it doesn't parse its own source in the tests ... or maybe it does <https://github.com/nikic/PHP-Parser/blob/v5.1.0/test/bootstr...>, who knows (joking but not joking)

Interesting. One of the reasons I stopped considering Rector is because of how memory, CPU, and time intensive it is for a non-trivial project. Instead I've been using Nikita's PHP Parser directly and getting much better results even though it isn't multi-threaded out of the box.

nikic/PHP-Parser uses a Bison equivalent for PHP parsing. See the grammar file https://github.com/nikic/PHP-Parser/blob/4.x/grammar/php7.y.

Maybe nikic/PHP-Parser is an alternative. If you only need to tokenizer part, PHP has an extension too.

nikic/PHP-Parser

Code parsing. Many linters and code builders use php-parser. It uses YACC (analog Bison) to build AST.

We use the tree-sitter[1] for parsing C declarations in Rizin[2] (see the "td" command, for example). See our custom grammar[3] (modified mainstream tree-sitter-c). The custom grammar was sadly necessary, due to the inability of Tree-Sitter to have the alternate roots[4].

P.S. Please add a license for your code.

[1] https://tree-sitter.github.io/

[2] https://github.com/rizinorg/rizin/tree/dev/librz/type/parser

[3] https://github.com/rizinorg/tree-sitter-c/

[4] https://github.com/tree-sitter/tree-sitter/issues/711

This is the default and recommended strategy. Through the tree-sitter parser, it understands the syntactic structure of code and splits according to semantic units.

Paste this into the Playground (try it here). You’ll get something like:

This code sets up a parser for Python code. The tree-sitter-python package provides a precompiled grammar, so no manual compilation is needed. If you’re targeting another language, check for its grammar package on Tree-Sitter Wiki.

The API is defined in tree_sitter/api.h (available on GitHub). It revolves around a few key concepts: parsers, trees, nodes, and queries. Let’s dive into the essentials.

Tree-sitter is a parser generator tool and an incremental parsing library, it is available in Rust 🦀 - GitHub. CocoIndex has built-in Rust integration with Tree-sitter to efficiently parse code and extract syntax trees for various programming languages.

I think the main problem is that highligthing framework used by tree-sitter (https://github.com/tree-sitter/tree-sitter/tree/master/highl...) is not easily pluggable into emacs font-lock-mode.

Although I am not certain if tree-sitter allows matching on comments, but https://github.com/tree-sitter/tree-sitter/issues/1138 seems to imply that it does

Hard agree. Even without going deep on a "serious language" there's a universe of DSLs that's mostly unexplored.

Debuggers are the outlier in your group but there's not exactly a void for those other wishes. As just one slice, building a tree-sitter [1] grammar gives the basis for good editor integration [2], formatters [3], structural diff [4] and other dev tools. Similarly if you're expressing some form of program, mapping to LLVM IR connects your creation with a fairly extensive compiler toolchain.

Language agnostic tooling exists, but there still needs to be some abstraction layer.

[1]: https://tree-sitter.github.io/

[2]: https://zed.dev/blog/syntax-aware-editing

[3]: https://topiary.tweag.io

[4]: https://difftastic.wilfred.me.uk

For those that want to explore the grammars listed at https://github.com/tree-sitter/tree-sitter/wiki/List-of-pars... in a more friendly railroad diagram format I made https://mingodad.github.io/plgh/json2ebnf.html that reads the "src/grammar.json" and try it's best to generate an EBNF understood by (IPV6) https://www.bottlecaps.de/rr/ui or (IPV4) https://rr.red-dove.com/ui where we get a nice navigable railroad diagram (see https://github.com/GuntherRademacher/rr for offline usage).