Sovereign LLM Serving on H200 — a 4-Model Concurrency Curve (70B, gpt-oss-120b, DeepSeek-V3-671B)

Summary

Measured the concurrent-request ceiling, time-to-first-token, and throughput of four open models served under vLLM 0.22 on NVIDIA H200 (Hopper) hardware, fp8 weights + fp8 KV cache, to characterize how cheaply a sovereign actor can serve frontier-class models privately. Headline finding: serving concurrency is governed by ACTIVE parameter count, not total size. gpt-oss-120b (MoE, ~5B active) sustains the highest concurrency of all (214x max-concurrency, chat ceiling >300 and long-context >100), beating the dense Llama-3.3-70B despite 120B total params; DeepSeek-V3-671B (MoE, ~37B active) serves privately across one 8xH200 node (~200 concurrent chat/node tuned). The earlier ~5-concurrent ceiling measured on an unoptimized Blackwell stack was a serving-configuration artifact, not a hardware limit -- on mature H200 with an optimized stack, per-GPU concurrency reaches 46-214x depending on the model.

Claims

• —Llama 3.3 70B (dense, fp8) on a single H200 (TP=1, fp8 KV) reaches vLLM max-concurrency ~46x; the chat workload (512 in / 128 out) does not cross a 5s p99 TTFT through 300 concurrent (p99 ~1s at 300), peaking ~3,668 tok/s; the long-context workload (4096 in / 512 out) is memory-bound with a ceiling around 10-20 concurrent.
• —gpt-oss-120b (mixture-of-experts, ~5B active of ~117B total, MXFP4) on a single H200 reaches vLLM max-concurrency ~214x; BOTH the chat workload (>300 concurrent) and the long-context workload (>100 concurrent) stay below the 5s p99 TTFT threshold, peaking ~6,859 tok/s (~2x the dense 70B). It beats the dense 70B on long-context concurrency by roughly 5-10x despite ~1.7x the total parameters.
• —DeepSeek-V3 671B (mixture-of-experts, ~37B active, fp8) serves privately across one 8xH200 node (TP=8) with coherent generation. With CUDA-graph capture enabled it sustains ~200 concurrent chat requests per node at p99 TTFT <1s (~4,593 tok/s) -- roughly 8x the throughput of the enforce-eager floor (~50-100 concurrent), at the cost of a longer (~655s) startup.
• —Across the four models the ordering of per-GPU light-user concurrency is gpt-oss-120b (5B active) >= Llama-70B (dense) >> DeepSeek-V3-671B (37B active + tensor-parallel communication overhead). Serving cost is therefore driven by ACTIVE parameter count, not total model size -- a sparse low-active MoE is the most concurrency-efficient way to serve frontier-class quality.
• —The ~5 concurrent-request ceiling previously measured (Llama 70B GGUF on an unoptimized Blackwell stack) is confirmed to be a serving-configuration / software-immaturity artifact, not a hardware limit: the same model class on mature H200 with an optimized fp8 stack reaches an order of magnitude (or more) higher concurrency.

Assumptions

• —Load generated with `vllm bench serve` using synthetic random-token prompts; real traffic with shared persona/RAG prefixes may achieve higher effective concurrency via prefix-cache reuse, so these figures are conservative for production.
• —vLLM 0.22 on H200 (Hopper, sm_90), fp8 weights + fp8 KV cache, FlashAttention; native PyTorch sampler (flashinfer sampler disabled). The DeepSeek-V3 'floor' used --enforce-eager (no CUDA graphs); the 'tuned' run enabled CUDA graphs. Llama-70B and gpt-oss-120b ran on a single GPU (TP=1); DeepSeek-V3 used a full 8-GPU node (TP=8).
• —'Max-concurrency' is vLLM's KV-cache-capacity figure at the served context length; the practically meaningful number is the sustained ceiling where p99 TTFT first crosses 5s, reported per workload shape. Single-run, no deterministic seed; full per-configuration JSONs accompany the reproduction bundle.

Context

Reproducibility

Structural Metrics