Skip to main content
Roviral

DGX Station GB300: How vLLM Concurrency and CPU Offload Changed Five Model Benchmarks

Raising vLLM’s request limit enabled higher throughput tests, while added CPU offload cut per request speed for two models.

DGX Station GB300: How vLLM Concurrency and CPU Offload Changed Five Model Benchmarks

On the DGX Station GB300, raising vLLM’s concurrency limit made higher load levels measurable, while CPU offload coincided with slower per request results for two models. A two day comparison of five large mixture of experts models shows why benchmark figures describe a specific setup: changing memory placement or request handling can affect results even when the hardware stays the same.

Pareto curve, RedHatAI/GLM-5.3-Flash-NVFP4, run 1

Test setup and changed settings

The benchmarks ran on September 17 and 18 using aiperf_tool, a harness built around NVIDIA’s AIPerf client and the vLLM inference server. Each model ran in its own container. Requests used a 1,024 token prompt, slightly longer after applying the model’s chat template, and exactly 512 generated tokens. Early stopping was disabled, keeping the amount of work per request consistent.

Pareto curve, RedHatAI/GLM-5.3-Flash-NVFP4, run 2

Each concurrency level ran for five minutes, with two repeats per run. Concurrent requests operated continuously: a slot sent another request as soon as its previous response finished. The measurements cover total tokens per second, per response streaming speed and time to first token. They don’t measure answer quality, and speculative decoding was not used.

Pareto curve, MiniMaxAI/MiniMax-M2.7, run 1

Both runs used the same DGX Station GB300 and the same five models. Run 1 tested concurrency levels of 1, 4, 16 and 32; MiniMax-M2.7 also ran at 64. Run 2 extended testing to 64 and 128 for all models. The vLLM setting --max-num-seqs, which limits the number of requests processed simultaneously, rose from 64 to 128. vLLM reserves working memory for that maximum at startup, so increasing it reduces memory available for other uses.

Pareto curve, MiniMaxAI/MiniMax-M2.7, run 2

Model memory placement also changed in two cases. --cpu-offload-gb moves a configured quantity of model weights from GPU memory to the Grace CPU’s memory, regardless of how much GPU memory is free. Those weights cross the CPU–GPU link during generation. With --cpu-offload-params experts, only expert weights move; each token uses a subset of them.

Pareto curve, nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4, run 1

MiniMax-M2.7 retained 25 GB of CPU memory offload, and GLM-5.3-Flash-NVFP4 had none in either run. Nemotron-3-Ultra kept 150 GB of expert weights in CPU memory. DeepSeek-V4-Flash changed from no CPU offload to 15 GB of expert weights offloaded, with tool call parsing also enabled. BF16 GLM-5.3-Flash increased its CPU memory allocation from 110 GB to 130 GB.

Pareto curve, nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4, run 2

Results at shared load levels

For MiniMax-M2.7, GLM-5.3-Flash-NVFP4 and Nemotron-3-Ultra, the settings that changed were limited to the higher request cap. At shared concurrency levels, total throughput matched within about 2 percent. That fits the expected behavior: a cap of 128 has no effect when the workload stays at or below 64 concurrent requests.

Pareto curve, nvidia/DeepSeek-V4-Flash-nvfp4-DSpark, run 1

At 32 concurrent requests, GLM-5.3-Flash-NVFP4 delivered about 1,507 tokens per second in both runs. Its run 2 peak reached 3,045 tokens per second at 128 requests, showing how the extra load levels exposed throughput that the first run could not measure.

Pareto curve, nvidia/DeepSeek-V4-Flash-nvfp4-DSpark, run 2

BF16 GLM-5.3-Flash was slower per request in run 2 at every shared concurrency level from 1 through 32. The increase from 110 GB to 130 GB of weights in CPU memory coincided with 10 to 12 percent fewer tokens per second per request. Within each run, repeated measurements varied by no more than 0.05 tokens per second per request, so this shift was substantially larger than the observed repeat variation. At 16 concurrent requests, the aggregate totals are affected by AIPerf’s measurement window; per request speed was lower in run 2 at every shared level.

Pareto curve, zai-org/GLM-5.3-Flash, run 1

DeepSeek-V4-Flash had the largest change at shared load levels: run 2 total throughput was 13 to 58 percent lower. Its repeated measurements also varied more than those of the other models. At 32 concurrent requests, run 1 produced 1,398 and 2,019 tokens per second, while run 2 produced 600 and 828. Even the higher run 2 result was below the lower run 1 result at every shared level. Because CPU offload and tool call parsing both changed, these measurements cannot identify which setting caused the decline.

Pareto curve, zai-org/GLM-5.3-Flash, run 2

Higher concurrency and throughput peaks

Four models reached their highest measured total throughput at 128 requests in run 2. GLM-5.3-Flash-NVFP4 reached 3,045 tokens per second, DeepSeek-V4-Flash reached 2,088, MiniMax-M2.7 reached 1,263, and Nemotron-3-Ultra reached 192. BF16 GLM-5.3-Flash peaked at 214 tokens per second at 64 requests according to AIPerf.

That last figure needs context: vLLM’s own token counter recorded 307 tokens per second at 128 requests, above the 212 recorded at 64. At 128, however, one request took about three and a half minutes to complete, and AIPerf counts only requests that finish within its measurement window. The two counters therefore reflect different completion behavior at that load.

Peak figures from the runs aren’t directly comparable because run 1 tested fewer concurrency levels. The clearest like for like comparison is GLM-5.3-Flash-NVFP4 at 32 requests, where both runs measured about 1,507 tokens per second; run 2 then showed roughly twice that total at 128.

The accompanying Pareto charts plot total throughput against the streaming speed of an individual answer, with each point representing a load level. Their axis scales differ between runs, so visual distances across charts shouldn’t be read as direct quantitative comparisons. The chart set covers both runs for all five models, including the DeepSeek configurations with and without CPU offload and the two BF16 GLM offload amounts.

What the comparison supports

The results suggest that increasing --max-num-seqs to match a higher target load did not change performance at the shared concurrency levels for the three models whose other settings remained stable. CPU offload coincided with slower per request throughput for BF16 GLM-5.3-Flash and DeepSeek-V4-Flash, but the DeepSeek run also enabled tool call parsing. The measurements support identifying those configuration changes as relevant factors, but not assigning a separate effect to each.

This is a limited comparison: there were two configurations, two repeats per load level, and a synthetic workload with fixed prompt and output lengths. Both runs used the same vLLM versions for each model: v0.29.0, nightly builds for the two GLM-5.3 models, and v0.22.0 for Nemotron-3-Ultra. The exact nightly build was not recorded and may have differed between the dates. Real prompts and responses vary in length, so deployment testing should use the workload and concurrency range that matter for the intended service.

Raw measurements, configuration files and charts are available in the September 17 data repository and the September 19 data repository. The practical takeaway is to treat a throughput number as a property of the model, runtime settings, memory placement and request load together.

Source: Michael Wolf · pi3g.com

Filed alongside