serving/raw-vllm: a concurrency sweep measuring latency
(TTFT / ITL / end-to-end) and throughput, with GPU and KV-cache utilisation read from the
same Prometheus the dashboards use. Harness: vllm bench serve (the repo’s benchmarks/ Job).
Run context
Results: concurrency sweep
Latencies in milliseconds. TTFT = time to first token (prefill), ITL = inter-token latency (decode), E2E = end-to-end request latency.
Resource peaks over the run (Prometheus / DCGM):
What the numbers say
- Compute-bound, not memory-bound. GPU compute pins at 100 % under load while KV-cache
never exceeds ~1 % and nothing ever queues (
waiting = 0). For a 0.5B model the KV footprint per request is tiny, so on one L4 you run out of compute long before memory. Scaling pods/replicas wouldn’t help on a single GPU: the GPU is already the bottleneck. - TTFT degrades first. As concurrency rises 1 → 32, TTFT p50 grows ~5× (27 → 135 ms): more requests competing for prefill slots queue at the front of the request. This is the first SLI to watch under load.
- Decode (ITL) stays cheap and flat. ITL holds ~5-6 ms until concurrency 16, reaching 12 ms p95 only at 32. Continuous batching keeps per-token decode efficient; the small model means decode is never the constraint here.
- Throughput scales near-linearly to saturation. Output throughput rises 168 → 3773 tok/s (≈22×) from concurrency 1 → 32, tracking GPU utilisation toward 100 %. Past saturation, more concurrency buys throughput only by trading latency (TTFT/E2E climb).
- E2E is decode-dominated. End-to-end is ≈ TTFT + 127×ITL; with 128 output tokens the ~5-7 ms ITL accounts for most of the ~0.7-1.1 s total, so E2E tracks ITL more than TTFT.
Live view
The vLLM Serving Grafana dashboard (dashboards/vllm-serving-dashboard.json) shows the
same signals in real time: TTFT/ITL/E2E percentiles, prompt vs generation throughput,
running vs waiting requests, KV-cache usage, and GPU util/mem (DCGM).