[CI] Reduce Flakiness For test_spec_decode.py::test_suffix_decoding_acceptance #30367
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
| Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request aims to reduce flakiness in test_suffix_decoding_acceptance by lowering the acceptance rate threshold. The change is justified and seems correct. I have one high-severity comment regarding a stale comment and a magic number that should be addressed to improve maintainability.
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
shaharmor98 pushed a commit to shaharmor98/smor-vllm that referenced this pull request Dec 11, 2025
…cceptance (vllm-project#30367) Signed-off-by: Micah Williamson <micah.williamson@amd.com>
TheCodeWrangler pushed a commit to TheCodeWrangler/vllm that referenced this pull request Dec 12, 2025
…cceptance (vllm-project#30367) Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: Nathan Price <nathan@abridge.com>
TheCodeWrangler pushed a commit to TheCodeWrangler/vllm that referenced this pull request Dec 12, 2025
…cceptance (vllm-project#30367) Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: Nathan Price <nathan@abridge.com> Signed-off-by: Nathan Price <nathan@abridge.com>
TheCodeWrangler pushed a commit to TheCodeWrangler/vllm that referenced this pull request Dec 12, 2025
…cceptance (vllm-project#30367) Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: Nathan Price <nathan@abridge.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
v1/e2e/test_spec_decode.py::test_suffix_decoding_acceptancehas been a bit flaky, failing about 5.6% of the time for almost the same exact reason each time despite passing after a retry. The final acceptance threshold is currentlylast_accept_rate > 0.825. Here are some examples of recent failures in buildkite for this test:Build 42551:
FAILED v1/e2e/test_spec_decode.py::test_suffix_decoding_acceptance - assert 0.8142514011208967 > 0.825Build 42535
FAILED v1/e2e/test_spec_decode.py::test_suffix_decoding_acceptance - assert 0.8142514011208967 > 0.825Build 42714:
FAILED v1/e2e/test_spec_decode.py::test_suffix_decoding_acceptance - assert 0.8105515587529976 > 0.825I believe I looked through almost all of the failures from the past week and all of them were 0.81 or above. I think if we can just set the threshold to 0.80, the success rate over the past week would have been 100%