doc: update cuda_graph_config usage part in DS R1 docs (#5796)

Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com>

Author: nv-guomingz

docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md

@@ -195,20 +195,20 @@ We are seeing meaningful speedup using FP8 KV cache, thus refreshing the numbers
 #### Benchmark
 ```bash
 cat >./extra-llm-api-config.yml <<EOF
-use_cuda_graph: true
-cuda_graph_padding_enabled: true
-cuda_graph_batch_sizes:
-- 896
-- 512
-- 256
-- 128
-- 64
-- 32
-- 16
-- 8
-- 4
-- 2
-- 1
+cuda_graph_config:
+ padding_enabled: true
+ batch_sizes:
+ - 896
+ - 512
+ - 256
+ - 128
+ - 64
+ - 32
+ - 16
+ - 8
+ - 4
+ - 2
+ - 1
 print_iter_log: true
 kv_cache_dtype: fp8
 enable_attention_dp: true
@@ -262,19 +262,19 @@ python ${YOUR_WORK_PATH}/benchmarks/cpp/prepare_dataset.py \
 YOUR_DATA_PATH=./dataset.txt
 
 cat >./extra-llm-api-config.yml <<EOF
-use_cuda_graph: true
-cuda_graph_padding_enabled: true
-cuda_graph_batch_sizes:
-- 1
-- 2
-- 4
-- 8
-- 16
-- 32
-- 64
-- 128
-- 256
-- 384
+cuda_graph_config:
+ padding_enabled: true
+ batch_sizes:
+ - 1
+ - 2
+ - 4
+ - 8
+ - 16
+ - 32
+ - 64
+ - 128
+ - 256
+ - 384
 print_iter_log: ${PRINT_ITER_LOG}
 enable_attention_dp: true
 EOF

docs/source/blogs/tech_blog/blog3_Optimizing_DeepSeek_R1_Throughput_on_NVIDIA_Blackwell_GPUs.md

@@ -151,7 +151,13 @@ These optimizations target the overall execution flow, scheduling, and resource
 
 * CUDA Graph
 
- This had a significant **22% E2E performance impact** for throughput scenarios. CUDA Graphs allow capturing a sequence of CUDA operations and launching them as a single unit, drastically reducing kernel launch overheads. This is particularly beneficial for models with many small kernels, and particularly on the PyTorch flow, because the python host code normally executes slower than C++. Since the CUDA Graph freezes the kernel launch parameters, which is normally associated with the tensor shapes, it can only be safely used with static shape, meaning that different CUDA graphs need to be captured for different batch sizes. Each graph will have some cost of memory usage, and capturing time, thus we cannot capture every possible CUDA graph for all possible batches. For the non-captured batch sizes, PyTorch eager mode code will be executed. There is a feature called CUDA Graph padding in TensorRT-LLM, which is a good trade-off between the number of CUDA Graphs and the CUDA Graph hit ratio; it tries to pad a batch to the nearest one with a captured CUDA Graph. Normally you should enable the CUDA Graph padding feature to increase the CUDA Graph hit rate, but the padding itself has some overhead due to wasted tokens computation. Users can opt-out the CUDA Graph padding feature to see the perf benefits, by setting the `cuda_graph_padding_enabled` to false, see API here [Pytorch backend config](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/config.py#L41)
+ This had a significant **22% E2E performance impact** for throughput scenarios.
+
+ CUDA Graphs allow capturing a sequence of CUDA operations and launching them as a single unit, drastically reducing kernel launch overheads. This is particularly beneficial for models with many small kernels, and particularly on the PyTorch flow, because the python host code normally executes slower than C++. Since the CUDA Graph freezes the kernel launch parameters, which is normally associated with the tensor shapes, it can only be safely used with static shape, meaning that different CUDA graphs need to be captured for different batch sizes. Each graph will have some cost of memory usage, and capturing time, thus we cannot capture every possible CUDA graph for all possible batches. For the non-captured batch sizes, PyTorch eager mode code will be executed.
+
+ There is a feature called CUDA Graph padding in TensorRT-LLM, which is a good trade-off between the number of CUDA Graphs and the CUDA Graph hit ratio; it tries to pad a batch to the nearest one with a captured CUDA Graph. Normally you should enable the CUDA Graph padding feature to increase the CUDA Graph hit rate, but the padding itself has some overhead due to wasted tokens computation.
+
+ Users can opt-out the CUDA Graph padding feature to see the perf benefits, by setting the `cuda_graph_config:\n padding_enabled: False`, see API here [Pytorch backend config](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/pyexecutor/config.py#L41)
 
 * Overlap Scheduler:
 

tests/integration/defs/perf/pytorch_model_config.py

@@ -65,9 +65,10 @@ def get_model_yaml_config(model_label: str,
  ],
  'config': {
  'enable_attention_dp': True,
- 'cuda_graph_padding_enabled': True,
- 'cuda_graph_batch_sizes':
- [1, 2, 4, 8, 16, 32, 64, 128, 256, 384]
+ 'cuda_graph_config': {
+ 'padding_enabled': True,
+ 'batch_sizes': [1, 2, 4, 8, 16, 32, 64, 128, 256, 384]
+ }
  }
  },
  # DeepSeek R1 model with specific batch size 128
@@ -76,7 +77,9 @@ def get_model_yaml_config(model_label: str,
  'deepseek_r1-bench-pytorch-float16-maxbs:128-maxnt:1127-input_output_len:1000,2000-quant:fp8-reqs:5120-con:1024-ep:8-gpus:8',
  'config': {
  'enable_attention_dp': True,
- 'cuda_graph_batch_sizes': [128]
+ 'cuda_graph_config': {
+ 'batch_sizes': [128]
+ }
  }
  },
  # Deepseek_v3_lite_cases