[Misc] Add metrics for request queue time, forward time, and execute time #9659
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Abatom requested review from WoosukKwon, alexm-redhat, comaniac, njhill, youkaichao and zhuohan123 as code owners 
| 👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Abatom changed the title model_execute_time and time_in_queue. Abatom changed the title 
| @WoosukKwon, Could you help me review the code? |
Abatom changed the title | @njhill , Could you help me review the code? Thanks! |
| @youkaichao , Could you help me review the code? Thanks! |
simon-mo reviewed Oct 25, 2024
simon-mo approved these changes Oct 28, 2024
rasmith pushed a commit to rasmith/vllm that referenced this pull request Oct 30, 2024
…time (vllm-project#9659) Signed-off-by: Randall Smith <Randall.Smith@amd.com>
hissu-hyvarinen pushed a commit to ROCm/vllm that referenced this pull request Nov 6, 2024
JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024
…time (vllm-project#9659) Signed-off-by: Loc Huynh <jc1da.3011@gmail.com>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
…time (vllm-project#9659) Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
markmc added a commit to markmc/vllm that referenced this pull request Mar 3, 2025
vllm:time_in_queue_requests appears to be an exact duplicate of vllm:request_queue_time_seconds. Both record first_scheduled_time-arrival_time: ``` if seq_group.is_finished(): time_queue_requests.append( seq_group.metrics.first_scheduled_time - seq_group.metrics.arrival_time) ``` ``` def maybe_set_first_scheduled_time(self, time: float) -> None: if self.metrics.first_scheduled_time is None: self.metrics.first_scheduled_time = time self.metrics.time_in_queue = time - self.metrics.arrival_time ``` vllm:time_in_queue_requests was added by vllm-project#9659 and vllm:request_queue_time_seconds was later added by vllm-project#4464. However, neither existed when each PR was first created. The latter seems like the right one to keep since it is implemented in V1, used in the Grafana dashboard, and has test coverage. Signed-off-by: Mark McLoughlin <markmc@redhat.com>
markmc added a commit to markmc/vllm that referenced this pull request Mar 3, 2025
Metrics originally added by vllm-project#9659 These seem to be of questionable value relative to the existing prefill, decode, and inference time metrics. And since they would be challenging to implement in V1, and they don't conform to the standard of using seconds as units, let's deprecate them Signed-off-by: Mark McLoughlin <markmc@redhat.com>
LeiWang1999 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Mar 26, 2025
…time (vllm-project#9659) Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
/metricsAPI.collect_model_forward_timeandcollect_model_execute_timedue to the--otlp-traces-endpointflag, so that metrics can also collect information aboutmodel_forward_timeandmodel_execute_time.