[2/N] Proper handling of placeholders in merged multi-modal processor #10485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 added the ready 
| 👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 changed the title Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 force-pushed the fix-mm-processor branch from 1d13132 to 8c493d1 Compare Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 force-pushed the fix-mm-processor branch from 8c493d1 to a85c542 Compare Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 changed the title DarkLight1337 changed the title DarkLight1337 changed the title DarkLight1337 force-pushed the fix-mm-processor branch from 9a47b6e to f910463 Compare Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 force-pushed the fix-mm-processor branch from f910463 to 4861d54 Compare 
DarkLight1337 force-pushed the fix-mm-processor branch from 9eb2ba6 to dc619cc Compare DarkLight1337 changed the title Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 force-pushed the fix-mm-processor branch from 3481896 to 4b7d00a Compare Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
DarkLight1337 changed the title DarkLight1337 changed the title DarkLight1337 changed the title 
Isotr0py reviewed Nov 22, 2024
DarkLight1337 force-pushed the fix-mm-processor branch from 1122c3f to bdb38fb Compare DarkLight1337 force-pushed the fix-mm-processor branch from bdb38fb to a29daa4 Compare 
ywang96 approved these changes Nov 22, 2024
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
…ject#10485) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
anko-intel pushed a commit to HabanaAI/vllm-fork that referenced this pull request Feb 12, 2025
…ject#10485) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Part of #10114
This PR updates #10044 to handle transformers v4.47 processors (huggingface/transformers#30962), which directly contain placeholder tokens in the processed
input_ids.I have also updated our code to properly replace targets with placeholders inside the processed prompt, and added unit tests for it.