Performance#
This section shows the latency and throughput numbers for Llama models powered by NVIDIA NIM. Please see Using AIPerf to Benchmark for the benchmark process.
For specifications of the hardware on which these measurements were collected, see the Hardware Specifications section.
Llama-3.3-70b-instruct Results#
NIM Container: Llama-3.3-70b-Instruct
Version: 1.8.0#
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 406.77 | 18.58 | 51.67 |
5 | 546.7 | 21.83 | 218.48 |
25 | 688.69 | 38.75 | 623.4 |
50 | 834.51 | 59.73 | 814.37 |
100 | 7996.25 | 92.91 | 917.26 |
150 | 34794.22 | 92.86 | 920.9 |
200 | 61680.22 | 93.06 | 920.54 |
250 | 88605.32 | 93.2 | 920.12 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 47.77 | 18.91 | 52.85 |
5 | 126.53 | 19.04 | 261.83 |
25 | 147.63 | 21.69 | 1149.05 |
50 | 181.64 | 23.97 | 2078.43 |
100 | 229.77 | 29.98 | 3322.87 |
150 | 296.74 | 37.85 | 3948.26 |
200 | 4867.89 | 40.72 | 4624.14 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 31.22 | 18.8 | 53 |
5 | 88.7 | 18.83 | 260.67 |
25 | 138.98 | 20.61 | 1178.68 |
50 | 156.28 | 23.33 | 2081.24 |
100 | 176.8 | 29.87 | 3263.31 |
150 | 195.51 | 37.75 | 3883.1 |
200 | 253.36 | 42.95 | 4534.54 |
250 | 264.22 | 48.16 | 5067.05 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 82.46 | 18.67 | 53.37 |
5 | 145.62 | 19.07 | 260.48 |
25 | 250.49 | 22.45 | 1102.23 |
50 | 278.62 | 25.87 | 1912.93 |
100 | 334.23 | 34.21 | 2895.49 |
150 | 428.54 | 44.44 | 3342.67 |
200 | 558.23 | 50.66 | 3901.74 |
250 | 3499.83 | 56.3 | 4167.39 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 1833.01 | 19.58 | 48.8 |
5 | 2910.33 | 25.24 | 186.8 |
25 | 18781.26 | 48.26 | 415.33 |
50 | 123969.71 | 49.22 | 418.65 |
100 | 333564.85 | 49.81 | 425.1 |
150 | 543226.4 | 50.01 | 423.95 |
200 | 753417.03 | 50.06 | 425.74 |
250 | 963222.96 | 50.13 | 427.06 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 418.61 | 16.72 | 57.05 |
5 | 567.86 | 19.15 | 246.86 |
25 | 708 | 35.86 | 671.12 |
50 | 841.35 | 56.65 | 857.45 |
100 | 5701.4 | 91.4 | 972.2 |
150 | 31108.04 | 91.64 | 972.92 |
200 | 56556.61 | 91.81 | 972.7 |
250 | 82015.69 | 91.91 | 972.59 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 46.38 | 16.79 | 59.51 |
5 | 121.67 | 16.36 | 304.63 |
25 | 169.85 | 18.64 | 1335.79 |
50 | 190.84 | 19.79 | 2515.16 |
100 | 238.18 | 26.06 | 3820.26 |
150 | 302.88 | 33.98 | 4395.18 |
200 | 357.1 | 37.06 | 5370.23 |
250 | 576.85 | 39.99 | 6202.02 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 29.58 | 16.85 | 59.12 |
5 | 96.02 | 16.41 | 297.51 |
25 | 144.09 | 18.03 | 1339.22 |
50 | 157.84 | 20.17 | 2391.63 |
100 | 187.07 | 27.57 | 3520.02 |
150 | 201.75 | 36.76 | 3983.17 |
200 | 258.88 | 42.13 | 4617.01 |
250 | 294.21 | 47 | 5172.59 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 81.45 | 16.72 | 59.56 |
5 | 147.1 | 16.48 | 301.06 |
25 | 252.61 | 19.38 | 1274.05 |
50 | 278.5 | 21.84 | 2261.37 |
100 | 332.18 | 30.22 | 3273.32 |
150 | 423.88 | 41.08 | 3614.04 |
200 | 582.43 | 46.43 | 4251.5 |
250 | 666.7 | 52.75 | 4676.16 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 1892.93 | 17.44 | 54.38 |
5 | 2981.43 | 21.61 | 215.84 |
25 | 7173.02 | 44.36 | 516.05 |
50 | 12866.29 | 72.84 | 621.2 |
100 | 145188.49 | 81.99 | 615.1 |
150 | 288488.94 | 82.99 | 621.73 |
200 | 432638.5 | 83.58 | 620.62 |
250 | 576444.85 | 84.03 | 620.09 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 81.66 | 16.12 | 61.78 |
5 | 306.56 | 15.63 | 314.03 |
25 | 894.46 | 18.35 | 1299.94 |
50 | 1067.36 | 20.75 | 2293.14 |
100 | 1158.71 | 29.61 | 3252.32 |
150 | 1341.05 | 41.04 | 3541.71 |
200 | 1283.61 | 46.78 | 4164.44 |
250 | 1808.22 | 51.77 | 4669.86 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 214.63 | 13.89 | 69.96 |
5 | 640.52 | 14.34 | 320.72 |
25 | 592.2 | 24.78 | 964.48 |
50 | 661.04 | 36.53 | 1322.8 |
100 | 857.04 | 61.36 | 1587.86 |
150 | 1098.38 | 88 | 1665.56 |
200 | 1149.56 | 111.46 | 1760.61 |
250 | 6176.07 | 127.61 | 1785.89 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 32.41 | 13.85 | 72.17 |
5 | 93.74 | 12.93 | 385.51 |
25 | 356.42 | 14.19 | 1740.17 |
50 | 492.05 | 15.62 | 3153.24 |
100 | 566.05 | 19.44 | 5071.9 |
150 | 707 | 24.7 | 5988.23 |
200 | 799.72 | 26.52 | 7430.1 |
250 | 1152.75 | 28.57 | 8579.87 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 22.73 | 13.8 | 72.22 |
5 | 56.59 | 13.44 | 366.12 |
25 | 160.64 | 13.83 | 1716.65 |
50 | 266.81 | 14.83 | 3104.67 |
100 | 370.92 | 18.82 | 4849.98 |
150 | 423.34 | 24.16 | 5718.54 |
200 | 639.02 | 25.32 | 7031.28 |
250 | 890.53 | 27.6 | 7803.07 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 51.01 | 13.99 | 71.26 |
5 | 170.81 | 13.06 | 378.3 |
25 | 503.84 | 14.76 | 1639.43 |
50 | 608.55 | 16.86 | 2864.92 |
100 | 613.39 | 22.02 | 4421.55 |
150 | 743.73 | 28.79 | 5082.9 |
200 | 955.06 | 31.74 | 6120.83 |
250 | 1405.48 | 34.75 | 6918.63 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 941.17 | 14.1 | 68.68 |
5 | 2191.86 | 16.22 | 288.76 |
25 | 4157.12 | 31.48 | 744.05 |
50 | 6205.6 | 50.33 | 933.97 |
100 | 84784.81 | 57.29 | 958.23 |
150 | 176553.42 | 58.23 | 966.23 |
200 | 269826.81 | 58.42 | 957.89 |
250 | 362546.32 | 58.52 | 963.45 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 78.59 | 15.49 | 64.3 |
5 | 292.54 | 16.74 | 293.91 |
25 | 893.19 | 20.19 | 1186.59 |
50 | 996.37 | 24.49 | 1963.32 |
100 | 1091.68 | 33.11 | 2926.44 |
150 | 1204.74 | 41.27 | 3534.23 |
200 | 1249.37 | 48.75 | 4002.72 |
250 | 5689.49 | 54.54 | 4133.74 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 554.81 | 18.54 | 50.99 |
5 | 1005.5 | 23.9 | 193.29 |
25 | 1362.44 | 52 | 457.61 |
50 | 1723.18 | 82.19 | 584.77 |
100 | 2257.5 | 146.67 | 662.51 |
150 | 2644.06 | 213.3 | 687.31 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 79.98 | 18.57 | 53.75 |
5 | 259.87 | 19.88 | 249.99 |
25 | 947.31 | 24.89 | 985.94 |
50 | 1167.07 | 28.75 | 1705.2 |
100 | 1458.45 | 39.83 | 2465.98 |
150 | 1513.77 | 53.21 | 2780.56 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 47.73 | 18.11 | 54.77 |
5 | 135.15 | 19.63 | 247.48 |
25 | 528.55 | 23.47 | 961.89 |
50 | 848.5 | 26.59 | 1629.45 |
100 | 1208.62 | 38.47 | 2257.04 |
150 | 1381.12 | 53.42 | 2498.15 |
200 | 1415.91 | 61.44 | 2932.47 |
250 | 1491.94 | 69.03 | 3283.87 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 131.21 | 18.44 | 53.91 |
5 | 460.71 | 19.83 | 246.66 |
25 | 1345.99 | 26.36 | 903 |
50 | 1498.8 | 31.6 | 1512.02 |
100 | 1625.47 | 46.47 | 2080.95 |
150 | 1729.4 | 63.28 | 2309.02 |
200 | 1680.2 | 72.79 | 2687.61 |
250 | 1884.04 | 82.1 | 2979.05 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 2521.02 | 19.46 | 48.27 |
5 | 5331.09 | 26.89 | 169.17 |
25 | 10457.52 | 64.46 | 358.42 |
50 | 16794.16 | 107.78 | 429.76 |
100 | 178225.46 | 127.17 | 441.63 |
150 | 379534.42 | 128.79 | 444.7 |
200 | 578423.34 | 129.63 | 446.76 |
250 | 778536.81 | 130.36 | 447.46 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 57.63 | 10.25 | 97.06 |
5 | 175.85 | 11.3 | 436.08 |
25 | 536.06 | 14.51 | 1663.36 |
50 | 602.94 | 18.33 | 2642.87 |
100 | 655.12 | 25.67 | 3801.4 |
150 | 687.9 | 31.68 | 4637.51 |
200 | 688.09 | 36.23 | 5421.19 |
250 | 795.95 | 40.31 | 6085.75 |
Version: 1.5.0#
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 368.11 | 19.94 | 48.44 |
5 | 446.68 | 23.89 | 199.17 |
25 | 751.74 | 42.12 | 564.5 |
50 | 1083.13 | 63.72 | 738.78 |
100 | 17754.93 | 87.55 | 777.31 |
150 | 46381.93 | 87.39 | 777.94 |
200 | 73326.6 | 87.64 | 777.02 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 71.82 | 19.26 | 51.84 |
5 | 109.62 | 20.04 | 237.72 |
25 | 159.62 | 23.28 | 1032.56 |
50 | 232.46 | 25.58 | 1880.75 |
100 | 293.19 | 32.83 | 2914.64 |
150 | 448.16 | 45.57 | 3096.87 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 32.78 | 19.11 | 52.15 |
5 | 81.17 | 19.9 | 247.26 |
25 | 247.81 | 21.99 | 1080.83 |
50 | 377.42 | 23.94 | 1936.88 |
100 | 583.54 | 31.79 | 2886.57 |
150 | 831.88 | 44.81 | 3067.43 |
200 | 925.3 | 45.92 | 3947.82 |
250 | 1105.17 | 49.83 | 4521.68 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 103.2 | 19.31 | 51.54 |
5 | 125.56 | 20.52 | 234.76 |
25 | 195.68 | 24.76 | 979.58 |
50 | 243.19 | 28.55 | 1706.26 |
100 | 344.86 | 38.65 | 2507.94 |
150 | 568.88 | 54.06 | 2663.03 |
200 | 1384.9 | 58.11 | 3238.57 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 190.36 | 19.6 | 49.57 |
5 | 430.84 | 21.64 | 216.96 |
25 | 870.9 | 35 | 654.15 |
50 | 1066.1 | 51.56 | 903.17 |
100 | 1332.94 | 88.91 | 1062.3 |
150 | 3853.57 | 124.93 | 1070.78 |
200 | 17105.6 | 125.13 | 1064.79 |
250 | 30309.49 | 125.27 | 1073.5 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 1637.36 | 20.5 | 46.92 |
5 | 3126.01 | 27 | 175.65 |
25 | 33539.02 | 52.4 | 323.8 |
50 | 127678.39 | 51.08 | 335.75 |
100 | 277353.39 | 48.51 | 345.84 |
150 | 339462.09 | 48.51 | 342.77 |
200 | 336890.58 | 51.03 | 338.3 |
250 | 515613.11 | 49.4 | 349.04 |
Llama-3.1-8b-instruct Results#
NIM Container: Llama-3.1-8b-Instruct
Version: 1.8.0#
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 10.26 | 4.63 | 214.56 |
5 | 16.81 | 4.58 | 1076.88 |
25 | 36.34 | 5.19 | 4678.88 |
50 | 67.41 | 5.91 | 8018.23 |
100 | 119.97 | 7.57 | 12214.97 |
150 | 346.53 | 9.81 | 12988.89 |
200 | 465.18 | 12.65 | 13348.4 |
250 | 605.6 | 16.14 | 12964.72 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 13.84 | 4.56 | 218.86 |
5 | 30.88 | 4.81 | 1035.85 |
25 | 97.07 | 5.82 | 4263.49 |
50 | 135.26 | 7.06 | 7009.01 |
100 | 239.92 | 9.51 | 10379.22 |
150 | 376.95 | 13.34 | 11053.75 |
200 | 1664.34 | 14.96 | 12549.74 |
250 | 6550.46 | 16.97 | 12268.82 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 19.03 | 4.53 | 220.1 |
5 | 55.87 | 4.8 | 1031.3 |
25 | 268.42 | 5.84 | 4098.01 |
50 | 414.87 | 7.26 | 6514.66 |
100 | 615.16 | 10.04 | 9392.43 |
150 | 885.53 | 14.14 | 9982.74 |
200 | 1258.79 | 16.02 | 11553.34 |
250 | 1885.04 | 19.68 | 11527.61 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 77.45 | 4.62 | 209.96 |
5 | 279.92 | 5.42 | 838.03 |
25 | 724.14 | 9.67 | 2251.44 |
50 | 817.73 | 16.47 | 2765.71 |
100 | 915.15 | 29.67 | 3180.16 |
150 | 1080 | 43.35 | 3301.68 |
200 | 4361.35 | 50.41 | 3378.48 |
250 | 11717.42 | 50.53 | 3372 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 403.2 | 5.06 | 189.96 |
5 | 1141.22 | 7.39 | 627.76 |
25 | 2371.46 | 19.35 | 1217.04 |
50 | 11071.41 | 30.73 | 1330.57 |
100 | 78187.07 | 31.08 | 1336.09 |
150 | 145290.54 | 31.31 | 1337 |
200 | 211735.71 | 31.46 | 1338.24 |
250 | 278580.85 | 31.62 | 1337.66 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 27.36 | 6.52 | 152.84 |
5 | 95.23 | 7.55 | 654.5 |
25 | 332.13 | 9.34 | 2585.01 |
50 | 589.89 | 11.46 | 4152.55 |
100 | 922.41 | 15.99 | 5914.93 |
150 | 1151.54 | 20.92 | 6800.02 |
200 | 1273.42 | 25.68 | 7424.87 |
250 | 5362.51 | 28.09 | 7435.16 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 9.97 | 4.35 | 228.55 |
5 | 19.02 | 4.11 | 1194.26 |
25 | 48.56 | 4.61 | 5170.73 |
50 | 69.42 | 5.32 | 8823.15 |
100 | 124.16 | 7.08 | 12949.5 |
150 | 318.5 | 9.67 | 13298.39 |
200 | 432.64 | 12.69 | 13456.29 |
250 | 553.89 | 15.82 | 13392.14 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 12.88 | 4.25 | 235.17 |
5 | 30.26 | 4.3 | 1160.07 |
25 | 106.08 | 5.12 | 4832.39 |
50 | 146.47 | 6.04 | 8175.67 |
100 | 305.31 | 8.3 | 11824.76 |
150 | 474.13 | 11.71 | 12530.29 |
200 | 3858.17 | 12.7 | 13534.56 |
250 | 8119.64 | 15.01 | 12973.67 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 19.79 | 4.24 | 234.95 |
5 | 57.78 | 4.31 | 1144.52 |
25 | 194.5 | 5.25 | 4598.71 |
50 | 254.85 | 6.4 | 7513.85 |
100 | 389.08 | 9.11 | 10533.71 |
150 | 713.87 | 13.01 | 10930.77 |
200 | 1362.27 | 14.27 | 12625.72 |
250 | 2313.26 | 16.82 | 12826.05 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 80.16 | 4.36 | 221.63 |
5 | 249.53 | 4.92 | 924.18 |
25 | 443.73 | 9.3 | 2458.55 |
50 | 431.31 | 15.69 | 3024.47 |
100 | 455.52 | 28.05 | 3458.25 |
150 | 643.97 | 40.83 | 3566.03 |
200 | 730.31 | 51.75 | 3763.37 |
250 | 804.18 | 63.22 | 3860.48 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 408.66 | 4.65 | 205.97 |
5 | 1118.18 | 6.38 | 720.3 |
25 | 1890.26 | 16.21 | 1455.56 |
50 | 2911.51 | 30.64 | 1555.38 |
100 | 19767.13 | 46.32 | 1736.48 |
150 | 71795.92 | 46.88 | 1735.44 |
200 | 123202.74 | 47.16 | 1734.91 |
250 | 175197.57 | 47.31 | 1735.35 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 28.82 | 7.11 | 140.14 |
5 | 97.14 | 7.27 | 679.46 |
25 | 420.1 | 8.81 | 2711.49 |
50 | 738.5 | 9.57 | 4854.67 |
100 | 1125.65 | 13.45 | 6864.22 |
150 | 1075.56 | 22.17 | 6456.17 |
200 | 1399.1 | 24.22 | 7811.53 |
250 | 2004.62 | 26.37 | 8813.31 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 19.42 | 13.75 | 72.58 |
5 | 49.22 | 14.12 | 349.76 |
25 | 160.89 | 15.56 | 1535.29 |
50 | 348.94 | 16.93 | 2688.88 |
100 | 486 | 21.19 | 4250.73 |
150 | 627.64 | 25.65 | 5232.81 |
200 | 678.66 | 31.16 | 5812.48 |
250 | 706.39 | 36.66 | 6245.55 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 26.98 | 14.06 | 71.1 |
5 | 94.1 | 14.87 | 335.23 |
25 | 381.83 | 18.2 | 1360.15 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 44.61 | 13.97 | 71.42 |
5 | 173.24 | 14.91 | 331.7 |
25 | 617.75 | 18.59 | 1302.3 |
50 | 704.53 | 23.59 | 2060.1 |
100 | 777.75 | 34.1 | 2869.49 |
150 | 770.03 | 44.38 | 3324.96 |
200 | 894.28 | 55.33 | 3559.97 |
250 | 5047.2 | 62.7 | 3676.16 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 243.92 | 14.72 | 65.86 |
5 | 455.05 | 18.14 | 262.89 |
25 | 644.27 | 36.93 | 655.22 |
50 | 830.47 | 60.47 | 806 |
100 | 8988.57 | 94.33 | 889.32 |
150 | 36691.18 | 94.4 | 890.18 |
200 | 64263.2 | 94.29 | 893.23 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 1299.33 | 17 | 56.67 |
5 | 2691.78 | 26.37 | 180.37 |
25 | 27548.98 | 70.21 | 289.26 |
50 | 182179.84 | 70.65 | 289.79 |
100 | 492175.92 | 70.71 | 291.05 |
150 | 798991.51 | 70.76 | 291.4 |
200 | 1106990.55 | 70.85 | 291.4 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 80.75 | 22.05 | 45.23 |
5 | 152.54 | 23.58 | 210.91 |
25 | 269.84 | 31.04 | 799 |
50 | 312.86 | 40.48 | 1226.46 |
100 | 2752.07 | 58.84 | 1613.38 |
150 | 33137.67 | 58.88 | 1616.93 |
200 | 63646.49 | 58.9 | 1618.87 |
Version: 1.3.0#
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 8.98 | 4.37 | 227.44 |
5 | 15.31 | 5.1 | 971.34 |
25 | 21.08 | 6.02 | 4100.33 |
50 | 50.68 | 7.19 | 6745.78 |
100 | 209.66 | 8.62 | 10369.56 |
150 | 398.54 | 12.49 | 10383.16 |
200 | 501.44 | 17.76 | 9884.77 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 13.89 | 4.77 | 209.52 |
5 | 27.22 | 5.45 | 915.96 |
25 | 110.61 | 6.32 | 3924.39 |
50 | 112.02 | 8.3 | 5979.38 |
100 | 245.7 | 11.12 | 8878.96 |
150 | 5217.57 | 13.78 | 8955.7 |
200 | 9112.64 | 18.27 | 8443.89 |
250 | 21097.73 | 19.26 | 8256.58 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 19.97 | 4.77 | 208.88 |
5 | 57.87 | 5.45 | 908.9 |
25 | 170.91 | 6.57 | 3710.08 |
50 | 214.79 | 8.74 | 5582.41 |
100 | 373.73 | 12.41 | 7819.41 |
150 | 890.01 | 15.84 | 8935.06 |
200 | 3231.57 | 18.31 | 9151.55 |
250 | 6469.67 | 21.75 | 8725.87 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 82.87 | 6.04 | 161.51 |
5 | 249.06 | 7.04 | 664.39 |
25 | 372.59 | 11.61 | 2024.67 |
50 | 415.72 | 19.99 | 2398.11 |
100 | 501.51 | 33.71 | 2844.33 |
150 | 646.08 | 47.58 | 3047.56 |
200 | 6830.5 | 51.83 | 3008.29 |
250 | 14795.72 | 51.74 | 3030.76 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 443.88 | 11.3 | 86.79 |
5 | 1011.39 | 12.71 | 377.88 |
25 | 1432.91 | 21.41 | 1099.79 |
50 | 16505.76 | 32.16 | 1116.23 |
100 | 89292.14 | 32.21 | 1190.64 |
150 | 153371.08 | 32.17 | 1195.86 |
200 | 208680.03 | 32.14 | 1200.69 |
250 | 281213.01 | 32.19 | 1213.79 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 11.96 | 6.54 | 152.24 |
5 | 27.18 | 7.39 | 667.60 |
25 | 34.67 | 8.60 | 2858.90 |
50 | 56.65 | 10.04 | 4858.86 |
100 | 168.60 | 12.57 | 7478.74 |
150 | 479.48 | 14.66 | 8811.28 |
200 | 755.88 | 18.81 | 8864.97 |
250 | 957.59 | 24.77 | 8439.91 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 19.89 | 6.94 | 143.93 |
5 | 51.19 | 7.70 | 647.12 |
25 | 202.87 | 9.42 | 2625.41 |
50 | 329.35 | 11.62 | 4242.00 |
100 | 552.68 | 15.67 | 6268.04 |
150 | 782.65 | 20.32 | 7232.18 |
200 | 7996.68 | 21.76 | 7666.88 |
250 | 20815.08 | 21.96 | 7610.47 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 29.73 | 6.93 | 143.87 |
5 | 96.74 | 7.72 | 640.26 |
25 | 386.45 | 9.50 | 2530.05 |
50 | 625.88 | 12.03 | 3951.34 |
100 | 724.55 | 17.79 | 5326.15 |
150 | 1126.07 | 22.88 | 6241.50 |
200 | 1691.82 | 27.89 | 6548.23 |
250 | 5664.63 | 29.12 | 7150.32 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 128.32 | 8.14 | 119.30 |
5 | 503.39 | 8.88 | 506.79 |
25 | 1243.85 | 17.36 | 1259.32 |
50 | 1273.57 | 29.38 | 1561.19 |
100 | 6941.19 | 44.67 | 1689.99 |
150 | 21179.86 | 44.66 | 1690.45 |
200 | 35001.67 | 44.54 | 1694.93 |
250 | 49204.54 | 44.48 | 1703.87 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 637.84 | 13.16 | 74.21 |
5 | 1881.45 | 13.50 | 346.33 |
25 | 19476.78 | 31.76 | 579.50 |
50 | 92969.97 | 31.81 | 558.51 |
100 | 216580.75 | 31.83 | 582.54 |
150 | 301171.32 | 31.79 | 583.14 |
200 | 348383.77 | 31.81 | 582.88 |
250 | 484376.95 | 31.85 | 584.58 |
Llama-3.1-70b-instruct Results#
NIM Container: llama-3.1-70b-instruct
Version: 1.3.0#
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 25.05 | 13.21 | 75.36 |
5 | 64.82 | 12.64 | 387.55 |
25 | 146.11 | 15.31 | 1564.9 |
50 | 169.61 | 17.42 | 2743.6 |
100 | 235.17 | 23.28 | 4092.34 |
150 | 511.21 | 27.73 | 4962.78 |
200 | 582.07 | 31.22 | 5855.9 |
250 | 718.98 | 35.25 | 6429.95 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 38.64 | 14.24 | 70.14 |
5 | 109.06 | 13.68 | 364.24 |
25 | 393.58 | 15.86 | 1557.21 |
50 | 564.17 | 17.14 | 2868.22 |
100 | 640.42 | 22.64 | 4350.2 |
150 | 792.84 | 28.49 | 5183.08 |
200 | 1417.35 | 31.62 | 6168.81 |
250 | 2045.69 | 35.75 | 6511.88 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 59.89 | 14.24 | 69.98 |
5 | 180.35 | 13.72 | 360.12 |
25 | 571.16 | 16.38 | 1475.28 |
50 | 843.45 | 18.15 | 2631.13 |
100 | 740 | 25.55 | 3794.76 |
150 | 952.07 | 32.31 | 4493.57 |
200 | 980.89 | 37.92 | 5114.09 |
250 | 2632.73 | 41.92 | 5497.21 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 236.58 | 17.35 | 56.2 |
5 | 432.23 | 18.16 | 263.04 |
25 | 657.73 | 28.69 | 830.42 |
50 | 952.43 | 40.13 | 1169.61 |
100 | 1312.92 | 68.86 | 1378 |
150 | 1973.14 | 98.88 | 1428.94 |
200 | 2850.08 | 126.51 | 1455.09 |
250 | 11283.63 | 139.94 | 1488.41 |
Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) |
|---|---|---|---|
1 | 1018.5 | 30.32 | 32.45 |
5 | 2116.84 | 31.29 | 153.71 |
25 | 3987.46 | 43.16 | 541.65 |
50 | 7012.87 | 56.95 | 793.95 |
100 | 93862.88 | 61.72 | 815.75 |
150 | 175864.67 | 61.73 | 815.38 |
200 | 240432.76 | 61.73 | 815.52 |
250 | 335844.98 | 61.75 | 828.23 |
Hardware Specifications#
NVIDIA H100#
Motherboard Model | NVIDIA DGX H100 |
Server Model | NVIDIA DGX H100 |
Number of Nodes | 1 |
CPU Information | Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On |
Number of CPU sockets enabled | 2 |
Number of CPU threads enabled | 224 |
GPU Information | H100 80GB HBM3(GH100) |
Driver Information | 570.124.06 (r570_00) |
GPU Core Clock (MHz) | 1980 |
GPU Boost Clock (MHz) | 1980 |
GPU Memory Clock (MHz) | 2619 |
NVIDIA H200#
Motherboard Model | NVIDIA DGX H200 |
Server Model | NVIDIA DGX H200 |
Number of Nodes | 1 |
CPU Information | Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On |
Number of CPU sockets enabled | 2 |
Number of CPU threads enabled | 224 |
GPU Information | H200 141GB HBM3e(GH100) |
Driver Information | 570.124.06 (r570_00) |
GPU Core Clock (MHz) | 1980 |
GPU Boost Clock (MHz) | 1980 |
GPU Memory Clock (MHz) | 2619 |
NVIDIA A100#
Motherboard Model | NVIDIA DGX A100 |
Server Model | NVIDIA DGX A100 |
Number of Nodes | 1 |
CPU Information | AMD EPYC 7742 @ 2.2GHz 3.4GHz Turbo (Rome) HT On |
Number of CPU sockets enabled | 2 |
Number of CPU threads enabled | 256 |
GPU Information | A100 SXM4 80GB(GA100) |
Driver Information | 570.124.06 (r570_00) |
GPU Core Clock (MHz) | 1155 |
GPU Boost Clock (MHz) | 1401 |
GPU Memory Clock (MHz) | 1593 |
NVIDIA L40s#
Motherboard Model | Supermicro X13DEG-OA |
Server Model | Supermicro SYS-521GE-TNRT |
Number of Nodes | 1 |
CPU Information | Platinum 8570 @ 2.1GHz 4.0GHz Turbo (Emerald Rapids) HT On |
Number of CPU sockets enabled | 2 |
Number of CPU threads enabled | 224 |
GPU Information | L40S(AD102) |
Driver Information | 570.124.06 (r570_00) |
GPU Core Clock (MHz) | 2520 |
GPU Boost Clock (MHz) | 2520 |
GPU Memory Clock (MHz) | 9001 |