Add Tests to measure activation times of extension #1813
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| console.log(`Analysis Engine Loaded in ${analysisEngineActivationAvgTime}ms`); | ||
| | ||
| expect(devActivationAvgTime - releaseActivationAvgTime).to.be.lessThan(AllowedIncreaaseInActivationDelayInMS, 'Activation times have increased above allowed threshold.'); | ||
| expect(devActivationAvgTime).to.be.lessThan(1000, 'Activation time has exceed 1s.'); |
There was a problem hiding this comment.
With analysis engine may be a bit more than a second, depends on the machine/VM.
MikhailArkhipov approved these changes May 31, 2018
brettcannon suggested changes May 31, 2018
| import { commands, extensions } from 'vscode'; | ||
| import { StopWatch } from '../../client/common/stopWatch'; | ||
| | ||
| const AllowedIncreaaseInActivationDelayInMS = 50; |
| For Extension activation times, we load both extensions x times, and re-load the window y times in each x load. | ||
| I.e. capture averages by giving the extensions sufficient time to warm up. | ||
| This block of code merely launches the tests by using either the dev or release version of the extension, | ||
| and spawning the tests (mimic user starting tests from command line), this way we can run tests multiple times. |
There was a problem hiding this comment.
We can also set a maximum perf number to help deal with skew, e.g. we never want this to average above 500 ms and we fail if that occurs. This prevents drift by saying we allow up to a 10% shift by being absolute instead of relative. It would also potentially negate needing to do a comparison against a current version.
Codecov Report
@@ Coverage Diff @@ ## master #1813 +/- ## ========================================== - Coverage 74.37% 74.32% -0.06% ========================================== Files 282 282 Lines 13218 13191 -27 Branches 2375 2346 -29 ========================================== - Hits 9831 9804 -27 Misses 3256 3256 Partials 131 131
Continue to review full report at Codecov.
|
brettcannon approved these changes May 31, 2018
DonJayamanne force-pushed the testAgainstInsiders branch 2 times, most recently from 1927481 to 3707b2c Compare 
DonJayamanne force-pushed the testAgainstInsiders branch from c85a3cd to f05e27a Compare 
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Fixes #932
This pull request: