v1.0.0rc3

[QiJune]

Announcement Highlights:

• Model Support
  • Support Mistral3.1 VLM model (feat(models): Mistral3.1 VLM pytorch backend support #5529)
  • Add TensorRT-Engine Qwen3 (dense) model support ([feat] Add TensorRT-Engine Qwen3 (dense) model support #5650)
• Feature
  • Add support for MXFP8xMXFP4 in pytorch (fix (NvBug 5354925): Fix static EPLB #5411)
  • Log stack trace on error in openai server (chore: log stack trace on error in openai server #5749)
  • Refactor the topk parallelization part for the routing kernels ([https://nvbugs/5365714] fix(scaffolding): use default LLM rather than trt backend LLM #5705)
  • Adjust free GPU memory fraction in KvCacheConfig for DeepSeek R1 tests (fix: Adjust free GPU memory fraction in KvCacheConfig for DeepSeek R1 tests #5774)
  • Support FP8 row-wise dense GEMM in torch flow ([TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow #5615)
  • Move DeepEP from Docker images to wheel building (Refactor: move DeepEP from Docker images to wheel building #5534)
  • Add user-provided speculative decoding support ([TRTLLM-6291] feat: Add user-provided speculative decoding support #5204)
  • Add optional module cache for TRT-LLM Gen Gemm interfaces ([feat] Adds optional module cache for TRT-LLM Gen Gemm interfaces #5743)
  • Add streaming scaffolding_llm.generate_async support (feat(scaffolding): add streaming scaffolding_llm.generate_async support #5345)
  • Detokenize option in /v1/completions request ([feat]: Detokenize option in /v1/completions request #5382)
  • Support n-gram speculative decoding with disagg ([TRTLLM-5847][feat] Support n-gram speculative decoding with disagg #5732)
  • Return context response immediately when stream_interval > 1 (feat: Return context response immediately when stream_interval > 1 #5836)
  • Add support for sm121 ([TRTLLM-5366][feat]Add support for sm121 #5524)
  • Add LLM speculative decoding example (chore [TRTLLM-6161]: add LLM speculative decoding example #5706)
  • Update xgrammar version to 0.1.19 (feat: TRTLLM-6224 update xgrammar version to 0.1.19 #5830)
  • Some refactor on WideEP (chore: some refactor on WideEP #5727)
  • Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner ([TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner #5764)
  • Update transformers to 4.53.0 (Update transformers to 4.53.0 #5747)
  • Share PyTorch tensor between processes ([1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes #5396)
  • Custom masking utils for Gemma3 VLM (feat: Custom masking utils for Gemma3 VLM #5853)
  • Remove support for llmapi + TRT backend in Triton (chore: remove support for llmapi + TRT backend in Triton #5856)
  • Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE ([feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE #5723)
  • Enable kvcache to be reused during request generation (feat:enable kvcache to be reused during request generation #4028)
  • Simplify speculative decoding configs ([refactor] Simplification of Speculative decoding configs #5639)
  • Add binding type build argument (pybind, nanobind) (feat: binding type build argument (pybind, nanobind) #5802)
  • Add the ability to write a request timeline ([enhance] Add the ability to write a request timeline. #5258)
  • Support deepEP fp4 post quant all2all dispatch (deepEP fp4 post quant all2all dispatch #5881)
  • Refactor the rest routing part for the routing kernels in the MoE TRT-LLM backend (Refactor the rest routing part for the routing kernels in the MoE TRT-LLM backend #5771)
  • Move vision parts from processor to model for Gemma3 ([refactor] Move vision parts from processor to model for Gemma3 #5888)
• API
  • [BREAKING CHANGE] Rename mixed_sampler to enable_mixed_sampler ([TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler #5751)
  • [BREAKING CHANGE] Rename LLM.autotuner_enabled to enable_autotuner ([TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner #5876)
• Bug Fixes
  • Fix test_generate_with_seed CI failure. (fix: [nvbug/5368507] Fix test_generate_with_seed CI failure. #5772)
  • Improve fp4_block_scale_moe_runner type check ([fix] improve fp4_block_scale_moe_runner type check #5681)
  • Fix prompt adapter TP2 case ([NvBug 5362426] fix: Fix prompt adapter TP2 case #5782)
  • Fix disaggregate serving with attention DP (fix: Disaggregate serving with attention DP #4993)
  • Ignore nvshmem_src_*.txz from confidentiality-scan (Fix: ignore nvshmem_src_*.txz from confidentiality-scan #5831)
  • Fix a quote error introduced in Refactor: move DeepEP from Docker images to wheel building #5534 (Fix a quote error introduced in #5534 #5816)
  • Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. ([5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. #5801)
  • Fix lost requests for disaggregated serving (Fix lost requests for disaggregated serving #5815)
  • Update unit tests: skip all_close assert for dropout in attention, increase tolerance for rope op test (fix: [5376140] [AutoDeploy] Update unit tests: skip all_close assert for dropout in attention, increase tolerance for rope op test #5855)
  • Fix GEMM+AR fusion on blackwell (Fix GEMM+AR fusion on blackwell #5563)
  • Catch inference failures in trtllm-bench ([fix] Catch inference failures in trtllm-bench #5841)
  • Add is_fp8_output key to XQA kernel cubin hashing (solves Eagle3-one-engine Hopper fp8 bug) (Add is_fp8_output key to XQA kernel cubin hashing (solves Eagle3-one-engine Hopper fp8 bug) #5813)
  • Skip rope scaling for local layers in Gemma3 VLM (fix: Skip rope scaling for local layers in Gemma3 VLM #5857)
  • Fix llama4 multimodal support ([Bugfix] LLama4: fix for llama4 multimodal support #5809)
  • Fix Llama4 Scout FP4 crash issue ([nvbugs/5321981] Cherrypick fix: Fix the Llama3.1 405B hanging issue. (#5698) #5925)
  • Fix max batch size and max tokens in kv cache estimations for Nemotron-H ([TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H #5371)
  • Fix moe regression for sm120 (Fix : fix moe regression for sm120 #5823)
  • Fix Qwen2.5VL FP8 support ([NVBUG-5304516/5319741]Qwen2.5VL FP8 support #5029)
  • Fix the illegal memory access issue in moe gemm on SM120 ([fix] WAR to fix the illegal memory access issue in moe gemm on SM120 #5636)
  • Avoid nesting NCCL group in allgather and reduce scatter OPs (avoid nesting NCCL group in allgather and reduce scatter OPs #5866)
  • Fix tileN cannot % 16==0 & support sm89 deepgemm bmm ([fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm #5531)
  • Fix incremental detokenization ([NvBug 5370718, 5371538] fix: Fix incremental detokenization #5825)
  • Fix MoE workspace info by storing Torch tensor itself instead of data_ptr ([fix] Fix MoE workspace info by storing Torch tensor itself instead of data_ptr #5900)
  • Make the bench serving script compatible with different usages (fix: Make the bench serving script compatible with different usages #5905)
  • Fix mistral unit tests due to transformers upgrade ([fix] Fix mistral unit tests due to transformers upgrade #5904)
  • Fix the Llama3.1 405B hanging issue. ([5321981] fix: Fix the Llama3.1 405B hanging issue. #5698) ([nvbugs/5321981] Cherrypick fix: Fix the Llama3.1 405B hanging issue. (#5698) #5925)
  • Fix Gemma3 unit tests due to transformers upgrade (test: Fix Gemma3 unit tests due to transformers upgrade #5921)
  • Extend triton exit time for test_llava ([nvbug/5308432] fix: extend triton exit time for test_llava #5971)
  • Fix alltoall for llama4 (apply_router_weight_on_input=True) ([NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) #5902)
  • Remove SpecConfig and fix thread leak issues ([fix] Remove SpecConfig and fix thread leak issues #5931)
  • Fast redux detection in trtllm gen routing kernel (fix: fast redux detection in trtllm gen routing kernel #5941)
  • Fix cancel request logic (fix cancel request logic #5800)
  • Fix errors in wide-ep scripts (Fix errors in wide-ep scripts #5992)
  • Fix error in post-merge-tests ([BUG5388075][fix] Fix error in post-merge-tests #5949)
• Benchmark
• Performance
  • Optimize TRTLLM Sampler perf single beam single step (feat: Optimize TRTLLM Sampler perf single beam single step #5550)
• Infrastructure
  • Fix a syntax issue in the image check ([Infra] - Fix a syntax issue in the image check #5775)
  • Speedup fused moe tests ([ci] speedup fused moe tests #5726)
  • Set the label community action to only run on upstream TRTLLM ([None][infra] Set the label community action to only run on upstream TRTLLM #5806)
  • Update namelist in blossom-ci (update namelist in blossom-ci #5838)
  • Update nspect version ([TRTLLM-5878] update nspect version #5832)
  • Reduce redundant test cases for TRTLLM Gen FP8 MoE (test: reduce redundant test cases for TRTLLM Gen FP8 MoE #5845)
  • Parallelize torch unittests ([ci] parallelize torch unittests #5714)
  • Use current_image_tags.properties in rename_docker_images.py (fix: use current_image_tags.properties in rename_docker_images.py #5846)
  • Fix two known NSPECT high vulnerability issues and reduce image size (infra: [TRTLLM-6054][TRTLLM-5804] Fix two known NSPECT high vulnerability issues and reduce image size #5434)
• Documentation
  • Update the document of qwen3 and cuda_graph usage ([https://nvbugs/5365714] fix(scaffolding): use default LLM rather than trt backend LLM #5705)
  • Update cuda_graph_config usage part in DS R1 docs (doc: update cuda_graph_config usage part in DS R1 docs  #5796)
  • Add llama4 Maverick eagle3 and max-throughput and low_latency benchmark guide (Doc: Add llama4 Maverick eagle3 and max-throughput and low_latency benchmark guide #5810)
  • Fix link in llama4 Maverick example (Doc: fix link in llama4 Maverick example #5864)
  • Add instructions for running gemma in disaggregated serving (doc: Add instructions for running gemma in disaggregated serving #5922)
  • Add qwen3 disagg perf metrics (blog: add qwen3 disagg perf metrics #5822)
  • Update the disagg doc ([TRTLLM-5673] Doc: ensure the disagg doc is up to date #5938)
  • Update the link of the diagram (doc: update the link of the diagram #5953)
• Known Issues

What's Changed

• feat: Add support for MXFP8xMXFP4 in pytorch by @djns99 in feat: Add support for MXFP8xMXFP4 in pytorch #5535
• [Doc] update the document of qwen3 and cuda_graph usage by @byshiue in [Doc] update the document of qwen3 and cuda_graph usage #5703
• [Infra] - Fix a syntax issue in the image check by @chzblych in [Infra] - Fix a syntax issue in the image check #5775
• chore: log stack trace on error in openai server by @zhengd-nv in chore: log stack trace on error in openai server #5749
• fix: [nvbug/5368507] Fix test_generate_with_seed CI failure. by @bobboli in fix: [nvbug/5368507] Fix test_generate_with_seed CI failure. #5772
• Refactor the topk parallelization part for the routing kernels by @ChristinaZ in Refactor the topk parallelization part for the routing kernels #5567
• test: [CI] remove closed bugs by @xinhe-nv in test: [CI] remove closed bugs #5770
• [TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler by @Superjomn in [TRTLLM-5530][BREAKING CHANGE] refactor: LLM arglist rename mixed_sampler to enable_mixed_sampler #5751
• fix: Adjust free GPU memory fraction in KvCacheConfig for DeepSeek R1 tests by @yizhang-nv in fix: Adjust free GPU memory fraction in KvCacheConfig for DeepSeek R1 tests #5774
• [TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow by @DylanChen-NV in [TRTLLM-5812][feat] support FP8 row-wise dense GEMM in torch flow #5615
• feat: Optimize TRTLLM Sampler perf single beam single step by @dcampora in feat: Optimize TRTLLM Sampler perf single beam single step #5550
• Refactor: move DeepEP from Docker images to wheel building by @yuantailing in Refactor: move DeepEP from Docker images to wheel building #5534
• [TRTLLM-6291] feat: Add user-provided speculative decoding support by @Funatiq in [TRTLLM-6291] feat: Add user-provided speculative decoding support #5204
• [ci] speedup fused moe tests by @omera-nv in [ci] speedup fused moe tests #5726
• [feat] Adds optional module cache for TRT-LLM Gen Gemm interfaces by @davidclark-nv in [feat] Adds optional module cache for TRT-LLM Gen Gemm interfaces #5743
• chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… by @nv-guomingz in chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie… #5795
• feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL by @yechank-nvidia in feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL #5522
• Revert "chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie…" by @nv-guomingz in Revert "chore: [Breaking Change] Rename cuda_graph_config padding_enabled fie…" #5818
• [fix] https://nvbugs/5333654 Unwaive to check ci status and improve torch compile multi-gpu coverage by @liji-nv in [fix] https://nvbugs/5333654 Unwaive to check ci status and improve torch compile multi-gpu coverage #5700
• [fix] improve fp4_block_scale_moe_runner type check by @Alcanderian in [fix] improve fp4_block_scale_moe_runner type check #5681
• feat(scaffolding): add streaming scaffolding_llm.generate_async support by @dc3671 in feat(scaffolding): add streaming scaffolding_llm.generate_async support #5345
• [None][infra] Set the label community action to only run on upstream TRTLLM by @poweiw in [None][infra] Set the label community action to only run on upstream TRTLLM #5806
• Waive some test_llama_eagle3 unittests by @venkywonka in Waive some test_llama_eagle3 unittests #5811
• [NvBug 5362426] fix: Fix prompt adapter TP2 case by @syuoni in [NvBug 5362426] fix: Fix prompt adapter TP2 case #5782
• chore: bump version to 1.0.0rc3 by @yiqingy0 in chore: bump version to 1.0.0rc3 #5819
• doc: update cuda_graph_config usage part in DS R1 docs by @nv-guomingz in doc: update cuda_graph_config usage part in DS R1 docs  #5796
• fix: Disaggregate serving with attention DP by @VALLIS-NERIA in fix: Disaggregate serving with attention DP #4993
• Fix: ignore nvshmem_src_*.txz from confidentiality-scan by @yuantailing in Fix: ignore nvshmem_src_*.txz from confidentiality-scan #5831
• tests: waive failed cases on main by @xinhe-nv in tests: waive failed cases on main #5781
• [Infra] - Waive L0 test by @yiqingy0 in [Infra] - Waive L0 test #5837
• update namelist in blossom-ci by @niukuo in update namelist in blossom-ci #5838
• Fix a quote error introduced in Refactor: move DeepEP from Docker images to wheel building #5534 by @yuantailing in Fix a quote error introduced in #5534 #5816
• [feat]: Detokenize option in /v1/completions request by @Wokzy in [feat]: Detokenize option in /v1/completions request #5382
• [5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. by @hyukn in [5305318] fix: Fix the accuracy issue when reduce_fusion is enabled for GEMMA model. #5801
• [TRTLLM-5847][feat] Support n-gram speculative decoding with disagg by @raayandhar in [TRTLLM-5847][feat] Support n-gram speculative decoding with disagg #5732
• [TRTLLM-5878] update nspect version by @niukuo in [TRTLLM-5878] update nspect version #5832
• feat: Return context response immediately when stream_interval > 1 by @kaiyux in feat: Return context response immediately when stream_interval > 1 #5836
• test: reduce redundant test cases for TRTLLM Gen FP8 MoE by @DomBrown in test: reduce redundant test cases for TRTLLM Gen FP8 MoE #5845
• [nvbug/5308432] fix: triton_backend test_llava timeout by @chang-l in [nvbug/5308432] fix: triton_backend test_llava timeout #5814
• [TRTLLM-5366][feat]Add support for sm121 by @pamelap-nvidia in [TRTLLM-5366][feat]Add support for sm121 #5524
• chore [TRTLLM-6161]: add LLM speculative decoding example by @Superjomn in chore [TRTLLM-6161]: add LLM speculative decoding example #5706
• Fix lost requests for disaggregated serving by @Tabrizian in Fix lost requests for disaggregated serving #5815
• fix: [5376140] [AutoDeploy] Update unit tests: skip all_close assert for dropout in attention, increase tolerance for rope op test by @Fridah-nv in fix: [5376140] [AutoDeploy] Update unit tests: skip all_close assert for dropout in attention, increase tolerance for rope op test #5855
• Fix GEMM+AR fusion on blackwell by @xavier-nvidia in Fix GEMM+AR fusion on blackwell #5563
• [fix] Catch inference failures in trtllm-bench by @omera-nv in [fix] Catch inference failures in trtllm-bench #5841
• Doc: Add llama4 Maverick eagle3 and max-throughput and low_latency benchmark guide by @jiahanc in Doc: Add llama4 Maverick eagle3 and max-throughput and low_latency benchmark guide #5810
• test: Validate and add accuracy& perf tests for Ministral-8B-Instruct[-FP8](pytorch only) by @venkywonka in test: Validate and add accuracy& perf tests for Ministral-8B-Instruct[-FP8](pytorch only) #5654
• Add is_fp8_output key to XQA kernel cubin hashing (solves Eagle3-one-engine Hopper fp8 bug) by @jhaotingc in Add is_fp8_output key to XQA kernel cubin hashing (solves Eagle3-one-engine Hopper fp8 bug) #5813
• feat: TRTLLM-6224 update xgrammar version to 0.1.19 by @Wanli-Jiang in feat: TRTLLM-6224 update xgrammar version to 0.1.19 #5830
• Doc: fix link in llama4 Maverick example by @jiahanc in Doc: fix link in llama4 Maverick example #5864
• fix: Skip rope scaling for local layers in Gemma3 VLM by @brb-nv in fix: Skip rope scaling for local layers in Gemma3 VLM #5857
• fix: [https://nvbugspro.nvidia.com/bug/5375656] Unwaive for bug 5375656. by @bobboli in fix: [https://nvbugspro.nvidia.com/bug/5375656] Unwaive for bug 5375656. #5842
• [AutoDeploy] re-enable waive for flaky AD test by @lucaslie in [AutoDeploy] re-enable waive for flaky AD test #5867
• Remove unnecessary benchmarking results by @qiaoxj07 in Remove unnecessary benchmarking results #5852
• chores: merge examples for v1.0 doc by @hchings in chores: merge examples for v1.0 doc #5736
• [Bugfix] LLama4: fix for llama4 multimodal support by @chang-l in [Bugfix] LLama4: fix for llama4 multimodal support #5809
• [TRTLLM-6262] Fix Llama4 Scout FP4 crash issue by @chenfeiz0326 in [TRTLLM-6262] Fix Llama4 Scout FP4 crash issue #5834
• chore: some refactor on WideEP by @dongxuy04 in chore: some refactor on WideEP #5727
• [TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner by @DomBrown in [TRTLLM-5881] feat: Integrate TRT-LLM Gen FP4 block scale MoE with Pytorch workflow kernel autotuner #5764
• [ci] parallelize torch unittests by @omera-nv in [ci] parallelize torch unittests #5714
• fix: use current_image_tags.properties in rename_docker_images.py by @ixlmar in fix: use current_image_tags.properties in rename_docker_images.py #5846
• [TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H by @tomeras91 in [TRTLLM-5838][fix] fix max batch size and max tokens in kv cache estimations for Nemotron-H #5371
• Fix : fix moe regression for sm120 by @peaceh-nv in Fix : fix moe regression for sm120 #5823
• [NVBUG-5304516/5319741]Qwen2.5VL FP8 support by @DylanChen-NV in [NVBUG-5304516/5319741]Qwen2.5VL FP8 support #5029
• Update transformers to 4.53.0 by @Wanli-Jiang in Update transformers to 4.53.0 #5747
• [1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes by @chang-l in [1/N][TRTLLM-5195][feat] Share PyTorch tensor between processes #5396
• feat(models): Mistral3.1 VLM pytorch backend support by @2ez4bz in feat(models): Mistral3.1 VLM pytorch backend support #5529
• feat: Custom masking utils for Gemma3 VLM by @brb-nv in feat: Custom masking utils for Gemma3 VLM #5853
• [fix] WAR to fix the illegal memory access issue in moe gemm on SM120 by @peaceh-nv in [fix] WAR to fix the illegal memory access issue in moe gemm on SM120 #5636
• Waive unittest failures introduced by PR#5345 (removal of ScaffoldingOutput class) by @venkywonka in Waive unittest failures introduced by PR#5345 (removal of ScaffoldingOutput class) #5886
• [feat] Add TensorRT-Engine Qwen3 (dense) model support by @gkswns0531 in [feat] Add TensorRT-Engine Qwen3 (dense) model support #5650
• [TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner by @Superjomn in [TRTLLM-5530] chore: rename LLM.autotuner_enabled to enable_autotuner #5876
• avoid nesting NCCL group in allgather and reduce scatter OPs by @QiJune in avoid nesting NCCL group in allgather and reduce scatter OPs #5866
• chore: remove support for llmapi + TRT backend in Triton by @achartier in chore: remove support for llmapi + TRT backend in Triton #5856
• [feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE by @rosenrodt in [feat] Add TRTLLM MoE nvfp4 cubins for mid-high concurrency; attention_dp for TRTLLM MoE #5723
• [fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm by @CarstyYou in [fix] fix tileN cannot % 16==0 & support sm89 deepgemm bmm #5531
• [NvBug 5370718, 5371538] fix: Fix incremental detokenization by @syuoni in [NvBug 5370718, 5371538] fix: Fix incremental detokenization #5825
• [None] - Waive L0 tests by @yiqingy0 in [None] - Waive L0 tests #5915
• [fix] Fix MoE workspace info by storing Torch tensor itself instead of data_ptr by @jinyangyuan-nvidia in [fix] Fix MoE workspace info by storing Torch tensor itself instead of data_ptr #5900
• fix: Make the bench serving script compatible with different usages by @kaiyux in fix: Make the bench serving script compatible with different usages #5905
• feat:enable kvcache to be reused during request generation by @narutolhy in feat:enable kvcache to be reused during request generation #4028
• infra: [TRTLLM-6054][TRTLLM-5804] Fix two known NSPECT high vulnerability issues and reduce image size by @ZhanruiSunCh in infra: [TRTLLM-6054][TRTLLM-5804] Fix two known NSPECT high vulnerability issues and reduce image size #5434
• [refactor] Simplification of Speculative decoding configs by @wili-65535 in [refactor] Simplification of Speculative decoding configs #5639
• feat: binding type build argument (pybind, nanobind) by @Linda-Stadter in feat: binding type build argument (pybind, nanobind) #5802
• doc: Add instructions for running gemma in disaggregated serving by @Tabrizian in doc: Add instructions for running gemma in disaggregated serving #5922
• [fix] Fix mistral unit tests due to transformers upgrade by @2ez4bz in [fix] Fix mistral unit tests due to transformers upgrade #5904
• [nvbugs/5321981] Cherrypick fix: Fix the Llama3.1 405B hanging issue. ([5321981] fix: Fix the Llama3.1 405B hanging issue. #5698) by @nvzhihanj in [nvbugs/5321981] Cherrypick fix: Fix the Llama3.1 405B hanging issue. (#5698) #5925
• [enhance] Add the ability to write a request timeline. by @FrankD412 in [enhance] Add the ability to write a request timeline. #5258
• deepEP fp4 post quant all2all dispatch by @yilin-void in deepEP fp4 post quant all2all dispatch #5881
• test: Fix Gemma3 unit tests due to transformers upgrade by @brb-nv in test: Fix Gemma3 unit tests due to transformers upgrade #5921
• [TRTLLM-4770][feat] Enhance cpp executor cmake to listen to ENABLE_MU… by @WilliamTambellini in [TRTLLM-4770][feat] Enhance cpp executor cmake to listen to ENABLE_MU… #5104
• blog: add qwen3 disagg perf metrics by @Shixiaowei02 in blog: add qwen3 disagg perf metrics #5822
• Refactor the rest routing part for the routing kernels in the MoE TRT-LLM backend by @ChristinaZ in Refactor the rest routing part for the routing kernels in the MoE TRT-LLM backend #5771
• [TRTLLM-5673] Doc: ensure the disagg doc is up to date by @Shixiaowei02 in [TRTLLM-5673] Doc: ensure the disagg doc is up to date #5938
• doc: update the link of the diagram by @Shixiaowei02 in doc: update the link of the diagram #5953
• tests: update sanity tests & fix tests by @xinhe-nv in tests: update sanity tests & fix tests #5906
• [refactor] Move vision parts from processor to model for Gemma3 by @2ez4bz in [refactor] Move vision parts from processor to model for Gemma3 #5888
• [TRTLLM-6264] Fix flaky test_e2e.py::test_openai_lora by @thorjohnsen in [TRTLLM-6264] Fix flaky test_e2e.py::test_openai_lora #5885
• Added code owners for LLM API by @juney-nvidia in Added code owners for LLM API #5960
• [nvbug/5308432] fix: extend triton exit time for test_llava by @chang-l in [nvbug/5308432] fix: extend triton exit time for test_llava #5971
• [NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) by @syuoni in [NvBug 5378370] fix: Fix alltoall for llama4 (apply_router_weight_on_input=True) #5902
• [fix] Remove SpecConfig and fix thread leak issues by @mikeiovine in [fix] Remove SpecConfig and fix thread leak issues #5931
• [BUG5374319][fix] WAR for draft-target-model unit tests error by @wili-65535 in [BUG5374319][fix] WAR for draft-target-model unit tests error #5958
• fix: fast redux detection in trtllm gen routing kernel by @tongyuantongyu in fix: fast redux detection in trtllm gen routing kernel #5941
• fix cancel request logic by @QiJune in fix cancel request logic #5800
• Fix errors in wide-ep scripts by @qiaoxj07 in Fix errors in wide-ep scripts #5992
• [BUG5388075][fix] Fix error in post-merge-tests by @wili-65535 in [BUG5388075][fix] Fix error in post-merge-tests #5949

New Contributors

• @davidclark-nv made their first contribution in [feat] Adds optional module cache for TRT-LLM Gen Gemm interfaces #5743
• @Alcanderian made their first contribution in [fix] improve fp4_block_scale_moe_runner type check #5681
• @Wokzy made their first contribution in [feat]: Detokenize option in /v1/completions request #5382
• @raayandhar made their first contribution in [TRTLLM-5847][feat] Support n-gram speculative decoding with disagg #5732
• @gkswns0531 made their first contribution in [feat] Add TensorRT-Engine Qwen3 (dense) model support #5650

Full Changelog: v1.0.0rc2...v1.0.0rc3

---

This discussion was created from the release v1.0.0rc3.