Tokenizer: derefencing incorrectly identified as short array #1381
Description
The array dereferencing square brackets in the following code are incorrectly changed to short array tokens.
// Array and string literal dereferencing. echo [1, 2, 3][0]; echo 'PHP'[0];Debug output:
*** START LEVEL MAP *** Process token 0 on line 1 [col:1;len:5;lvl:0;]: T_OPEN_TAG => <?php\n Process token 1 on line 2 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n Process token 2 on line 3 [col:1;len:42;lvl:0;]: T_COMMENT => // Array and string literal de referencing.\n Process token 3 on line 4 [col:1;len:4;lvl:0;]: T_ECHO => echo Process token 4 on line 4 [col:5;len:1;lvl:0;]: T_WHITESPACE => Process token 5 on line 4 [col:6;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [ Process token 6 on line 4 [col:7;len:1;lvl:0;]: T_LNUMBER => 1 Process token 7 on line 4 [col:8;len:1;lvl:0;]: T_COMMA => , Process token 8 on line 4 [col:9;len:1;lvl:0;]: T_WHITESPACE => Process token 9 on line 4 [col:10;len:1;lvl:0;]: T_LNUMBER => 2 Process token 10 on line 4 [col:11;len:1;lvl:0;]: T_COMMA => , Process token 11 on line 4 [col:12;len:1;lvl:0;]: T_WHITESPACE => Process token 12 on line 4 [col:13;len:1;lvl:0;]: T_LNUMBER => 3 Process token 13 on line 4 [col:14;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ] Process token 14 on line 4 [col:15;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [ Process token 15 on line 4 [col:16;len:1;lvl:0;]: T_LNUMBER => 0 Process token 16 on line 4 [col:17;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ] Process token 17 on line 4 [col:18;len:1;lvl:0;]: T_SEMICOLON => ; Process token 18 on line 4 [col:19;len:0;lvl:0;]: T_WHITESPACE => \n Process token 19 on line 5 [col:1;len:4;lvl:0;]: T_ECHO => echo Process token 20 on line 5 [col:5;len:1;lvl:0;]: T_WHITESPACE => Process token 21 on line 5 [col:6;len:5;lvl:0;]: T_CONSTANT_ENCAPSED_STRING => 'PHP' Process token 22 on line 5 [col:11;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [ Process token 23 on line 5 [col:12;len:1;lvl:0;]: T_LNUMBER => 0 Process token 24 on line 5 [col:13;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ] Process token 25 on line 5 [col:14;len:1;lvl:0;]: T_SEMICOLON => ; *** END LEVEL MAP *** *** START ADDITIONAL PHP PROCESSING *** * token 22 on line 5 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY * token 24 on line 5 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY * token 14 on line 4 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY * token 16 on line 4 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY * token 5 on line 4 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY * token 13 on line 4 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY *** END ADDITIONAL PHP PROCESSING ***