Skip to content

Conversation

jrfnl
Copy link
Member

@jrfnl jrfnl commented May 15, 2024

Description

Tokenizers/Comment: add tests

The Tokenizers\Comment class did not have any tests associated with it. This commit fixes that and documents the existing behaviour.

Note: code coverage is as high as it can go, but not 100%. The reason for this, is the tokenizer debug statements, which are conditional on a verbosity flag, which is turned off for the tests.

Loosely related to #484.

Tokenizers/PHP: bug fix - empty block comment

This commit fixes an edge case tokenizer bug, where a - completely empty, not even whitespace - block comment, would be tokenized as a docblock.

Without this commit, the /**/ code snippet was tokenized as:

 8 | L07 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 4]: /**/ 9 | L07 | C 5 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 0]: 

With the fix applied, it will be tokenized as:

 8 | L07 | C 1 | CC 0 | ( 0) | T_COMMENT | [ 4]: /**/ 

Tokenizers/Comment: bug fix - empty docblock

This commit fixes an edge case tokenizer bug, where a - completely empty, not even whitespace - DocBlock, would not be tokenized correctly.

Without this commit, the /***/ code snippet was tokenized as:

 13 | L10 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 5]: /***/ 14 | L10 | C 6 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 0]: 

With the fix applied, it will be tokenized as:

 13 | L10 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 3]: /** 14 | L10 | C 4 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 2]: */ 

Tokenizers/Comment: minor tweaks

Girlscouting.

  • Remove an unnecessary interim variable.
  • Add a comment clarifying certain code.
  • Specify the array format in the @return tags.

Suggested changelog entry

  • Fixed an edge case bug where an empty block comment would not be tokenized correctly.
  • Fixed an edge case bug where an empty single-line DocBlock would not be tokenized correctly.

Related issues/external references

Preliminary work to allow for #484 later on.

Related #146

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
jrfnl added 4 commits May 18, 2024 11:37
The `Tokenizers\Comment` class did not have any tests associated with it. This commit fixes that and documents the existing behaviour. Note: code coverage is as high as it can go, but not 100%. The reason for this, is the tokenizer debug statements, which are conditional on a verbosity flag, which is turned off for the tests. Loosely related to 484.
This commit fixes an edge case tokenizer bug, where a - completely empty, not even whitespace - _block comment_, would be tokenized as a docblock. Without this commit, the `/**/` code snippet was tokenized as: ``` 8 | L07 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 4]: /**/ 9 | L07 | C 5 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 0]: ``` With the fix applied, it will be tokenized as: ``` 8 | L07 | C 1 | CC 0 | ( 0) | T_COMMENT | [ 4]: /**/ ```
This commit fixes an edge case tokenizer bug, where a - completely empty, not even whitespace - _DocBlock_, would not be tokenized correctly. Without this commit, the `/***/` code snippet was tokenized as: ``` 13 | L10 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 5]: /***/ 14 | L10 | C 6 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 0]: ``` With the fix applied, it will be tokenized as: ``` 13 | L10 | C 1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG | [ 3]: /** 14 | L10 | C 4 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG | [ 2]: */ ```
Girlscouting. * Remove an unnecessary interim variable. * Add a comment clarifying certain code. * Specify the array format in the `@return` tags.
@jrfnl
Copy link
Member Author

jrfnl commented May 18, 2024

Rebased without changes. Merging once the build passes.

@jrfnl jrfnl force-pushed the feature/tokenizer-comments-add-tests branch from 3cecd91 to bef6fff Compare May 18, 2024 09:38
@jrfnl jrfnl merged commit b7c2356 into master May 18, 2024
@jrfnl jrfnl deleted the feature/tokenizer-comments-add-tests branch May 18, 2024 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment