The private-AI model benchmark for work that can't leave your building.

One datacenter Blackwell card handles ~4.5× the concurrent load of a GB10 appliance — and keeps scaling to 128 streams.

Serving many users at once is where the hardware tiers separate. On an identical model (Gemma-4-26B-A4B, NVFP4), a single RTX PRO 6000 Blackwell serves 2,480 tok/s across 32 concurrent streams to the NVIDIA GB10 appliance's 554 — about 4.5×. And the datacenter card keeps scaling where the bandwidth-bound appliance can't: on a dense 27B it reaches ~2,790 tok/s at 128 concurrent sessions with zero preemptions (memory-bound, not compute-bound), while the GB10 plateaus by ~32. Reach for the Blackwell card for multi-user and bursty serving; the GB10 for single-stream appliance and edge.

We swept the whole ladder on one model: appliance to 4× NVLink H200. More GPUs buy model size and users, not single-user speed.

Same model (Qwen3.6-35B-A3B, FP8), same engine and settings, swept to saturation on every hardware tier we deploy. Single-user speed barely moves across the top four tiers (~200–216 t/s). What scales is capacity: aggregate throughput, comfortable concurrent users, and the memory pool that determines the largest private model you can run.

Two lessons for buyers. First: tensor-parallel scaling is sublinear for efficient MoE models (1× → 1.42× → 1.81× going 1→2→4 cards): you add cards to fit bigger models (an 80B coder on 2-way, a 397B frontier-class model on 4-way) and to raise the ceiling, not to make one user faster. Second: the interconnect is everything. The identical 4-GPU setup with just one of the four links crossing PCIe instead of NVLink collapsed from 14,071 to 1,693 tok/s, an 8× penalty. This is why consumer platforms top out at one big GPU, workstation platforms (Threadripper PRO) at ~4, and why NVLink-bridged server platforms own the multi-GPU tier. The platform is as much the product as the GPU.

Start with one GB10. Add a second for a bigger model. Change tiers for more users.

The ladder above turns into a simple upgrade guide. One GB10 (128 GB) runs a 26–35B-class private model for a small team (roughly 32 comfortable concurrent streams), and disciplined agent loops raise output quality on the same hardware before you spend anything. A second GB10 pools memory to ~256 GB: it does not make any single user faster, but it unlocks models that simply don't fit one unit, including a 500K-token-context coder that scored within ~2 points of its cloud-hosted version on SWE-bench Verified. That's the honest pitch: the pair is a model-size and context play for compliance-bound teams, not a speed upgrade. Need many simultaneous users? Change tiers instead: one RTX PRO 6000 workstation serves ~5× a GB10 pair's aggregate, and a single H200 nearly 3× that again. Need a frontier-size model on-premises? That's the NVLink multi-H200 tier (287–575 GB pools), where the largest open models run entirely inside your compliance boundary.

For single-user work, a laptop iGPU now trades blows with a Grace-Blackwell box.

Gemma-4-26B-A4B (4B active) on an AMD Strix Halo laptop — Radeon 8060S iGPU, unified LPDDR5X — decodes 63 t/s single-stream on llama.cpp Vulkan (43 t/s via Ollama), clearly ahead of the NVIDIA GB10's best vLLM single-stream (44.4 t/s). It is model-dependent — Qwen3.6's MTP path keeps the GB10 ahead there (95 vs 65 t/s) — and the same 96 GB unified pool loads gpt-oss-120b whole: a 120B-class MoE at 37 t/s on a laptop. The GB10 keeps the serving crown: vLLM batching scales it to 554 t/s at 32 concurrent streams, where llama.cpp cannot follow.

A near-cloud, CUI-capable coder already runs on our fleet.

On the official SWE-bench Verified harness (stratified-150, no LM judge), DeepSeek-V4-Flash local scored 67.3% — essentially tying cloud DeepSeek-V4-pro (69.6%), beating the fleet's Qwen3.6 (55.3%), about 15 points under Claude Opus-4.8 (82.0%). We own a private coder within reach of the frontier.

NVFP4 4-bit is lossless — for some models, not all.

Unsloth's NVFP4 quants hold full quality for Gemma-4 (coding and research within noise of BF16) at half the VRAM on Blackwell. But Qwen3.6-35B develops a long-form runaway at NVFP4 that FP8 doesn't — so quant choice is per-model, not universal.

Serving config is a first-class correctness variable.

Gemma-4 with the wrong vLLM tool parser emitted zero tool calls — it looked agentically incapable when it was a one-line misconfiguration. With --tool-call-parser gemma4 it became the best tool-caller in the set. Both models then passed real end-to-end runs through our live agent gateway.

Never let a model grade its own family.

Self-judging inflates scores 20–37%. Our cross-judge gate forbids same-family grading, and headline studies now use a three-model frontier panel (median of GPT-5.4, Opus-4.8, GLM-5.2) — which repeatedly caught a single judge under-rating one contestant by ~0.3.