How can an LLM respond faster while serving more requests on a single GPU?
Qwen3.8-27B is one of the LLMs commonly used in enterprise private deployments. To optimize its performance, we selected FP8 quantization and carried out inference optimization on a single RTX PRO 6000. Compared with the community-recommended vLLM configuration, single-stream generation speed rose from 79.6 to 137.7 tokens/s, a gain of about 73%.
In high-concurrency scenarios, a single GPU now handles about 33–34 concurrent requests at ≥50 tokens/s (up from ~14, a 2.4× gain) and about 95–97 at ≥20 tokens/s (up from ~57, a 1.7× gain), compared with the community-recommended configuration for each tier.
These gains come from several layers: tuning configuration to the workload, reducing the extra overhead of speculative decoding, and fixing performance issues in the inference engine and the underlying CUDA compute path. The following will describe the key findings, the optimization approaches, and the applicable conditions.
Test Background and Optimization Goals
After deployment, enterprise users typically care about two things: how fast one request is completed and how many requests the system can handle at once. These goals involve different bottlenecks and require different optimizations.
We used the following configurations for the tests:
Item | Configuration or requirement |
|---|---|
GPU | A single RTX PRO 6000 |
Model | FP8 quantization of Qwen3.8-27B |
Inference engine | vLLM v0.32.0 |
Baseline comparison | Community-recommended vLLM configuration |
Test data | ShareGPT |
Quality acceptance | Keep the main model's quantization; GSM8K accuracy drops by less than 1 percentage point |
The single-stream test focused on generation speed. The high-concurrency comparison targeted per-session generation speeds of at least 50 and 20 tokens/s and observed the number of requests a single GPU can handle simultaneously.
Here, the number of concurrent requests means the number of requests being processed simultaneously on the same GPU; it is not equal to the total number of online users. When evaluating service capacity, also consider time-to-first-token (TTFT) latency and actual traffic characteristics.
To ensure its reliability, we calibrated the measurement tools, and used repeated launches and tests to separate genuine gains from compilation and run-to-run variance. The gains from each optimization below come from their corresponding experimental conditions, and cannot simply be added together.
Single-Stream Optimization: Increasing Effective Output per Step
By default, the community-recommended configuration uses MTP speculative decoding with a draft length of 3. The model first drafts several candidate tokens, then the main model verifies them together, aiming to accept several valid tokens in a single verification pass.
The speedup depends on how many tokens each step accepts and how much work the drafter and verifier add.
Baseline measurements showed that each step produced about 2.26 tokens on average and required reading and writing about 36.9 GB of data, among which the target model accounted for about 26.9 GB, and generating the three-step MTP draft read an extra 9.1 GB. The draft head read the full vocabulary on every step, which is a major source of this extra data movement.
We optimized along two directions:
Replace MTP with DFlash2 to generate a block of draft tokens in a single parallel forward pass, and adjust the draft sampling method and length to increase effective output per step. Under this single-stream workload, a draft length of 7 performed best; beyond that, the additional accepted tokens no longer offset the extra verification time.
Reduce the data-read overhead of draft generation and candidate scoring. This included running the drafter in FP8 and using a two-stage scheme: FP8 for coarse candidate filtering, then BF16 for precise scoring. The latter reduced data reads while limiting the impact of low-precision scoring on the draft acceptance rate.
For the execution backend, the single-stream configuration used Marlin to skip activation quantization and eliminated about 176 kernel launches per step. In the corresponding test, this shortened each step by about 0.26 ms and improved throughput by about 2.9%. This gain is scenario-dependent; we will show the high-concurrency comparison later in this article.
In the end, the key metrics were enhanced as follows:
Metric | Community-recommended configuration | Enhanced configuration |
|---|---|---|
Tokens produced per step | ~2.26 | ~3.16 |
Data read/written per step | 36.9 GB | 30.2 GB |
Bandwidth utilization | 81.5% | 83.0% |
Single-stream generation speed | 79.6 tokens/s | 137.7 tokens/s |
Bandwidth utilization barely changed. The main gains come from producing more tokens per step while moving less data.
Operator Optimization: Fixing Abnormal Latency in Narrow Matrix Multiplications
When locating bottlenecks, we found that the most time-consuming task is not necessarily the one with the most room for improvement.
In one test, weight-reading matrix multiplication took about 89% of each step, but was already close to the bandwidth limit. Even further improvements would yield only limited benefit to per-step latency.
By contrast, a narrow matrix multiplication in GDN showed a clear anomaly: with MTP disabled, a single call took about 3 µs; with MTP enabled, for certain small matrix shapes, it took about 26.5 µs.
Tracing the problem into the NVIDIA CUDA library, we found that it selects an unsuitable kernel in the M=2–7 range. After switching to Triton GEMV for that shape, the cumulative time for this part dropped from about 1.27 ms to 0.17 ms per step, which improved throughput by about 4.9% under the corresponding configuration.
This optimization went from the model operator down into the execution path of the CUDA compute library, demonstrating the value of locating and fixing underlying performance issues. Besides tuning model and engine parameters, it also needs to check whether specific compute shapes match the underlying implementation. An operator with modest total compute can still become a substantial extra overhead if executed in an unsuitable way.
High-Concurrency Optimization: Rebalancing Compute and Memory Capacity
At high concurrency, multiple requests share GPU resources, so an approach that works well for a single stream needs to be re-evaluated here.
For example, a longer draft raises output per step, but rejected candidate tokens also consume verification compute. As concurrency grows, this overhead takes compute resources away from other requests. In these tests, k=7, the best setting for single-stream, performed worse than k=2 under the high-concurrency test.
Likewise, in the single-stream test, Marlin gained 2.9% by reducing activation quantization and kernel launches, but at concurrency-64 it was about 24% slower. The high-concurrency configuration therefore reverted to the default CUTLASS FP8 backend.
Memory capacity also affects total throughput. The community-recommended configuration supports a 262K long context by default and enables prefix caching. Under this workload, the associated state and cache allocations capped the number of requests that could run at once, and the rest had to queue.
For this test workload, we changed the configurations below. Shortening the maximum context and disabling prefix caching are meant to show these parameters’ effect on performance. Notably, they are not general recommendations for every application.
Configuration changes | Purpose | When it applies |
|---|---|---|
Reduce maximum context from 262K to 8K | Observe the impact of the context limit on concurrency capacity | The application can accept an 8K context limit |
Disable prefix caching | Observe the impact of the cache switch on state usage and concurrency capacity | Decide based on prefix-reuse benefits |
Store GDN state in BF16 | Reduce state usage and read/write volume | Verify via quality regression |
Set draft length to k=2 | Limit extra verification compute | Target this high-concurrency workload |
Increase CUDA graph capture size | Reduce execution overhead beyond the captured range | Match the batch size |
With only the official image and adjusting the configuration, total throughput at the 50 tokens/s tier increased from about 1.56k to 2.56k tokens/s.
This gain comes from matching the configuration to business requirements. Both the context limit and the cache switch affect application capability and should be chosen according to actual traffic.
Engine Optimization: Reducing State Storage and Memory Waste
We then examined the overhead that speculative decoding added inside the engine.
In the tested version, GDN speculative decoding had to save multiple intermediate states, and verifying each token produces a state write. The longer the draft, the greater the state usage and read/write overhead, which both affect execution time and limit concurrency capacity.
Using replay, we reduced the state slots reserved per request to a single copy while recording the input of each token for the current step. At the beginning of the next step, we replayed based on the tokens actually accepted to restore the state needed to continue.
This trades replay computation for less state storage and reads/writes. In the concurrency-64 and k=7 test, per-layer GDN time dropped from about 0.66 ms to 0.19 ms. The benefit varies with concurrency and draft length, thus cannot be directly equated to an overall throughput improvement.
We also found that when the KV block sizes of the drafter and the target model do not match, there is noticeable page-padding waste. Setting --block-size 64 works around the issue in this combination. Under the corresponding k=7 configuration, maximum concurrency rose from 31 to 95, consistent with the capacity-analysis prediction.
Although this enhancement only required changing a parameter, finding the effective value required a deep understanding of how the engine organizes memory.
Optimization Results and Deployment Recommendations
From a deployment-capacity perspective, under the two per-session generation-speed targets, the number of concurrent requests a single GPU can support improves as follows:
Per-session generation speed target | Community-recommended configuration, concurrent requests | Optimized configuration, concurrent requests | Change in concurrency throughput |
|---|---|---|---|
≥ 50 tokens/s | ~14 streams | 33–34 streams | ~2.4× the original |
≥ 20 tokens/s | ~57 streams | 95–97 streams | ~1.7× the original |
Note: For the 50 tokens/s tier, the community-recommended configuration used MTP with a draft length of 3, while the optimized configuration used a draft length of 2; for the 20 tokens/s tier, both configurations disabled speculative decoding. The multipliers in the table are computed from approximate concurrency counts.
This work shows that inference optimization involves configuration, engine implementation, and kernel-level computation. For deployments planning to adopt these approaches, one can first determine the actual context length, traffic characteristics, and response requirements, adjust the configuration, and then evaluate the additional gains from engine and operator patches.
All configurations should be recorded together with the hardware, model, and engine versions, and validated on the application's own quality-regression set. Single-stream and high-concurrency cases should be tested separately to avoid directly applying the optimal combination from one scenario to a different one.
Going forward, we will continue to optimize performance for commonly used models, organizing configuration choices, performance bottlenecks and implementation improvements across different hardware and business use cases to provide practical guidance for deployment.