Skip to content

bpo-40902: Speed up PEG parser by using Dijkstra's shunting yard algorithm #20696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 19 commits into from
Closed

bpo-40902: Speed up PEG parser by using Dijkstra's shunting yard algorithm #20696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
83337b3
WIP of operator precedence
pablogsal Jun 7, 2020
9e25541
Generate precedence table
pablogsal Jun 7, 2020
0366b96
Fix test_ast and test_type_comments and address feedback
pablogsal Jun 7, 2020
2f979f9
Use a struct to keep track of the augmented info
pablogsal Jun 7, 2020
c51f1ad
Remove the macro into a static inline function
pablogsal Jun 7, 2020
d0e3ef5
Minor cleanup
gvanrossum Jun 7, 2020
8f12437
Replace operators[i] with values[i].oper
gvanrossum Jun 7, 2020
2372f5e
Eliminate stack_pos -- it's always output_pos - 1
gvanrossum Jun 7, 2020
9398d8f
Tiny cleanup
gvanrossum Jun 7, 2020
1554dea
Rename values->stack, output_ptr->stack_top
gvanrossum Jun 7, 2020
0b6f84a
Use member assignments instead of creating a new ValueItem
gvanrossum Jun 7, 2020
8dd799b
Slight tweak of _reduction_step()
gvanrossum Jun 7, 2020
08e588b
Raise for now if the associativity of the operators is not 'left'
pablogsal Jun 7, 2020
882c740
Add a huge block comment explaining the operator precedence parser
gvanrossum Jun 8, 2020
13de6ab
Merge remote-tracking branch 'upstream/master' into operator_precedence
pablogsal Jun 8, 2020
10bc620
Implement right-associative operators
pablogsal Jun 8, 2020
5320f37
Better error handling
pablogsal Jun 8, 2020
fe2d89c
Remove TODO item
pablogsal Jun 8, 2020
6fee838
Do not use yet '**' in the operators list in the grammar
pablogsal Jun 8, 2020
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 11 additions & 23 deletions Grammar/python.gram
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ _PyPegen_parse(Parser *p)

// The end
'''
@operators '''
left |
left ^
left &
left << >>
left + -
left * / // % @
'''

file[mod_ty]: a=[statements] ENDMARKER { _PyPegen_make_module(p, a) }
interactive[mod_ty]: a=statement_newline { Interactive(a, p->arena) }
eval[mod_ty]: a=expressions NEWLINE* ENDMARKER { Expression(a, p->arena) }
Expand Down Expand Up @@ -415,30 +424,9 @@ in_bitwise_or[CmpopExprPair*]: 'in' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, I
isnot_bitwise_or[CmpopExprPair*]: 'is' 'not' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, IsNot, a) }
is_bitwise_or[CmpopExprPair*]: 'is' a=bitwise_or { _PyPegen_cmpop_expr_pair(p, Is, a) }

operator: '|' | '^' | '&' | '<<' | '>>' | '+' | '-' | '*' | '/' | '//' | '%' | '@'
bitwise_or[expr_ty]:
| a=bitwise_or '|' b=bitwise_xor { _Py_BinOp(a, BitOr, b, EXTRA) }
| bitwise_xor
bitwise_xor[expr_ty]:
| a=bitwise_xor '^' b=bitwise_and { _Py_BinOp(a, BitXor, b, EXTRA) }
| bitwise_and
bitwise_and[expr_ty]:
| a=bitwise_and '&' b=shift_expr { _Py_BinOp(a, BitAnd, b, EXTRA) }
| shift_expr
shift_expr[expr_ty]:
| a=shift_expr '<<' b=sum { _Py_BinOp(a, LShift, b, EXTRA) }
| a=shift_expr '>>' b=sum { _Py_BinOp(a, RShift, b, EXTRA) }
| sum

sum[expr_ty]:
| a=sum '+' b=term { _Py_BinOp(a, Add, b, EXTRA) }
| a=sum '-' b=term { _Py_BinOp(a, Sub, b, EXTRA) }
| term
term[expr_ty]:
| a=term '*' b=factor { _Py_BinOp(a, Mult, b, EXTRA) }
| a=term '/' b=factor { _Py_BinOp(a, Div, b, EXTRA) }
| a=term '//' b=factor { _Py_BinOp(a, FloorDiv, b, EXTRA) }
| a=term '%' b=factor { _Py_BinOp(a, Mod, b, EXTRA) }
| a=term '@' b=factor { CHECK_VERSION(5, "The '@' operator is", _Py_BinOp(a, MatMult, b, EXTRA)) }
| a=factor b=(o=operator f=factor {_PyPegen_operator_term_pair(p, o, f)})+ { _PyPegen_operator_precedence_expr(p, a, b) }
| factor
factor[expr_ty] (memo):
| '+' a=factor { _Py_UnaryOp(UAdd, a, EXTRA) }
Expand Down
Loading