Performance enhancements. Rebuild the word parser and replace the whitespace checker in the match finder. #102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
SavageTiger force-pushed the sven/feature/html_diff_update branch 2 times, most recently from a81629e to 317fdba Compare added 4 commits April 5, 2021 22:23
Added typehints, strict type checking, and removed else code-flow
While finding blocks we have todo a whitespace check of part of the old sentence, this is done loads of times. This is done by refrerencing the oldWords array, and temporarely making a string from part of that array, and then checking if that string is only whitespace. I have replaced this by a loop that iterates over the part of the sentence item by item, and when one of the items is not a space (usually the first item), it immediately reports false and caching the result, speeding up the algorithm in some cases by up to 50%
SavageTiger force-pushed the sven/feature/html_diff_update branch from 59073d6 to 7ef0cd5 Compare 
SavageTiger force-pushed the sven/feature/html_diff_update branch 2 times, most recently from 6cc072f to 7ef0cd5 Compare The old parser split up the html into a big string of characters, and then going over the string parsing it character by character, using a big complicated switch statement, and loads of if statements. Most of this has been replaced by regular expression parsing greatly simplifying the code, and reducing the execution time by 98%
SavageTiger force-pushed the sven/feature/html_diff_update branch from 7ef0cd5 to a16330a Compare SavageTiger changed the title Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Description
While investigating if I could improve the performance for this PR #101 I stumbled upon two bottlenecks.
1. The html to word parser was a big for loop that walked over every character individually
The code was really complex, and hard to comprehend. It was a huge loop with loads of code-flow inside that was affected by what character was being processed, what character was processed previously, and what process was comming up.
I replace it by mostly regex parsing that does the heavy lifting, speeding this method up by 98%
Old
New
2. Whitespace checking was resource intensive
While finding blocks we have todo a whitespace check of part of the old sentence, this is done loads of times. This is done by referencing the oldWords array, and temporarily making a string from part of that array, and then checking if that string is only whitespace.
A couple of years ago I already added caching here, to speed up the algorithm allot, but I took some more time to investigate if this can be improved further.
I have replaced this by a loop that iterates over the part of the sentence item by item, and when one of the items is not a space (usually the first item), it immediately reports false and caching the result, speeding up the algorithm in some cases by up to 50%
Old
New
