[Model] Support tensor parallel for timm ViT in Deepseek_vl2

[wzqd]

Support tensor parallel for timm ViT

• Wrap external ViT from timm library to support tensor parallel
• Apply tensor parallel to mlp blocks
• Apply tensor parallel to attention blocks

Purpose

The vision transformer from timm library does not support tensor parallel and has bad performance. Currently minicpmv2_0 and deepseek_vl2 are the only two models using timm ViT, but minicpmv2_6 has changed to idefics ViT. The purpose of this pr is to enhance performance of the ViT in deepseekvl2 without changing the model structure.

[gemini-code-assist]

Code Review

This pull request introduces tensor parallelism support for the timm Vision Transformer's MLP blocks within the deepseek_vl2 model. The implementation correctly monkey-patches timm.layers.Mlp to substitute nn.Linear with vLLM's ColumnParallelLinear and RowParallelLinear. A wrapper class is also added to adapt the output format of these parallel layers for compatibility with the timm library.

The overall approach is solid and directly addresses the goal of improving performance through tensor parallelism. However, I have identified a critical issue in the exception handling logic that could lead to a runtime error, as well as a high-severity issue in the implementation of the wrapper class that could introduce subtle bugs. Addressing these points will significantly improve the robustness and correctness of the code.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[Isotr0py]

LGTM now!