We evaluate open-weight and cloud LLMs the way a compliance-bound firm actually uses them — coding, hard debugging, cited research, agentic tool-use and long-form writing — judged by a three-model frontier panel and run on Petronella's own GPU fleet, so CUI and privileged data never touch a third party.
Under a three-model frontier panel (GPT-5.4 + Claude Opus-4.8 + GLM-5.2, median score on identical generations, N=5), the answer splits by task difficulty. Qwen wins routine coding; Gemma wins the hard coding set decisively and is the stronger agent — verified end-to-end through our live agent gateway.
Serving notes that mattered: Gemma-4 needs vLLM --tool-call-parser gemma4
(the wrong parser makes zero tool calls) and --max-model-len 131072 to fit the agent
gateway's ~58k-token system prompt. NVFP4 is near-lossless for Gemma-4 but not for Qwen3.6-35B (a
long-form runaway appears — use FP8). Full findings →
The single most decision-useful view: the strongest model we found for each fleet configuration and role, with the trade-offs that decided it.
--tool-call-parser mistral.Best score per model, cross-judged (generator never grades itself). Cloud models are shown as reference lines; everything marked LOCAL runs entirely on the PTG fleet.
Full tables with per-rep ranges, latency, temperature and dates →
The headline results driving PTG's private-AI deployment decisions — every number measured on our own hardware with a real evaluation harness, not vendor marketing.
Gemma-4-26B-A4B (4B active) on an AMD Strix Halo laptop — Radeon 8060S iGPU, unified LPDDR5X — decodes 63 t/s single-stream on llama.cpp Vulkan (43 t/s via Ollama), clearly ahead of the NVIDIA GB10's best vLLM single-stream (44.4 t/s). It is model-dependent — Qwen3.6's MTP path keeps the GB10 ahead there (95 vs 65 t/s) — and the same 96 GB unified pool loads gpt-oss-120b whole: a 120B-class MoE at 37 t/s on a laptop. The GB10 keeps the serving crown: vLLM batching scales it to 554 t/s at 32 concurrent streams, where llama.cpp cannot follow.
On the official SWE-bench Verified harness (stratified-150, no LM judge), DeepSeek-V4-Flash local scored 67.3% — essentially tying cloud DeepSeek-V4-pro (69.6%), beating the fleet's Qwen3.6 (55.3%), about 15 points under Claude Opus-4.8 (82.0%). We own a private coder within reach of the frontier.
Unsloth's NVFP4 quants hold full quality for Gemma-4 (coding and research within noise of BF16) at half the VRAM on Blackwell. But Qwen3.6-35B develops a long-form runaway at NVFP4 that FP8 doesn't — so quant choice is per-model, not universal.
Gemma-4 with the wrong vLLM tool parser emitted zero tool calls — it looked agentically incapable when it was a one-line misconfiguration. With --tool-call-parser gemma4 it became the best tool-caller in the set. Both models then passed real end-to-end runs through our live agent gateway.
Self-judging inflates scores 20–37%. Our cross-judge gate forbids same-family grading, and headline studies now use a three-model frontier panel (median of GPT-5.4, Opus-4.8, GLM-5.2) — which repeatedly caught a single judge under-rating one contestant by ~0.3.
Task-appropriate, cross-judged. Each model runs a fixed corpus per capability — 22 core + 20 hard coding tasks, 27 research cases, an 88-case tool decline-safety corpus, agentic multi-turn tasks, and a 3,000-word blog with a 10-criteria rubric. Coding and research combine deterministic auto-checks (does the code run, are the citations right) with an LM judge; blog combines structural gates with a judge. The generator is never the judge, and never shares its model family with the judge.
Frontier judge panel. Headline comparisons are graded by three independent frontier models — GPT-5.4 (OpenAI), Claude Opus-4.8 (Anthropic) and GLM-5.2 (Zhipu) — scoring identical saved generations, with the median taken per case so no single judge's bias can decide a result. DeepSeek-V4-Flash serves as a validated free grader for bulk work.
Measured on-fleet. Everything is timed and scored on Petronella's own GPUs — the point is that a private, compliance-bound firm can run capable AI on privileged and CUI data without sending a single token to a third party.