Petronella.ai

DeepSeek CPU Inference Runs 284B Model Without GPUs

November 8, 2026 · Cybersecurity
DeepSeek CPU Inference Runs 284B Model Without GPUs

DeepSeek CPU inference just crossed a threshold: a 284B-parameter model runs on a 64 GB RAM workstation with no GPU, and that changes what private AI can look like for regulated businesses.

The project, called llama-modchip, was published on GitHub by the developer @synchopate, who documents the full method, benchmarks and reproducibility instructions in the repository. The headline claim is that DeepSeek-V4-Flash, a 284.33B-parameter mixture-of-experts model, fits and runs entirely in system memory on a 64 GB machine, with zero GPUs required. Anyone who handles controlled unclassified information, protected health information, or other regulated data should pay attention, because the practical barrier to running a frontier model entirely on premises has always been hardware cost, not software.

Key Takeaways

What happened: the facts in detail

The claim

The repository's own summary states the goal plainly: run "Real-time Frontier & Agentic MoE inference on the CPU workstation you already own," with 100% of the model in RAM and zero NVMe swap. The flagship result is DeepSeek-V4-Flash, a 284.33B-parameter mixture-of-experts model with 13.17B active parameters, using 43 MLA layers and a 256+1 expert routing structure. The author reports it fits in under 59 GiB of RAM at full 256K context on a 64 GB machine.

The hardware

The headline benchmark was run on an AMD Ryzen 9 9955HX, a Zen 5 laptop-class processor, with zero GPUs. The project also documents results on a wide range of other platforms, including a Threadripper PRO 7985WX with 64 cores and 256 MB of L3 cache, an AWS Graviton3 instance, an Apple M4 Pro, Intel consumer chips paired with NVIDIA RTX cards on Windows, and a compact Ryzen 7 8745HS mini-PC. The DeepSeek result itself, however, is tied to the 64 GB Zen 5 configuration.

How the model was shrunk

The project describes three compression mechanisms working together. First, what the author calls Analytical SwiGLU-H2 Second-Order Hessian Compression, credited with shaving 20.31 GiB from the model's footprint. Second, a custom quantization scheme called Fractal-64, running at 3.75 bits per weight, applied to the model's multi-head latent attention tensors. Third, a four-level tiling scheme that maps the whole model across AVX-512 registers, 32 KB of L1 data cache, 1 MB of L2, and 64 MB of L3, designed to saturate 8-channel DDR5 memory bandwidth.

The result: the model occupies 56.56 GiB on disk, down from 76.87 GiB in the standard IQ2_XXS format, which the author notes crashes with an out-of-memory error on 64 GB Linux. At runtime the resident memory, measured as VmRSS, sits between 57.61 and 58.9 GiB with the full 256K context loaded.

How it was measured

The repository reports operator-level profiling and, in the author's words, mathematical proofs in dedicated documentation files within the project. The reported figures for DeepSeek-V4-Flash on the 9955HX are:

The project also publishes a 100-point quality, tool-calling, coding and needle-retrieval gate, and states that the Fractal-64 and MXFP4 formats for the Nemotron model score 100/100 on that gate. A community leaderboard collects scorecards submitted through GitHub issues and discussions, and the repository offers a two-second, zero-download CPU readiness test so users can check whether their own machine could run the 284B model.

The smaller-model results

The second pillar of the project is arguably more immediately practical. NVIDIA-Nemotron-3.5-30B-A3B, a 30.2B-parameter model with 3.0B active parameters, runs in 14.35 to 17.59 GiB at 25.0 to 60.4 tok/s on pure CPU with zero GPUs. On a Threadripper the author reports 111.6 tok/s, and on an AMD Vulkan hybrid configuration 80.8 tok/s. Even an AWS Graviton3 ARM instance reaches 44.97 tok/s decode. The speed comes partly from a 1.64 MB speculative drafter described as a Stockfish-PVS v2 plus RouteCrack system, with a reported 96.4% acceptance rate, and from cache isolation techniques that keep the drafter from evicting the main model's L3 working set.

Agentic workloads

A third component targets agent-style use. The project includes a launcher that wires a Claude Code-style CLI, with all 40 tools active, to the local engine, and eliminates what the author identifies as a 45-second prompt re-evaluation bottleneck on CPU agents by restoring cached context from shared memory in 18.9 ms on the Nemotron model and 13.1 to 32.6 ms on DeepSeek. For businesses experimenting with local AI agents, that warm-restore figure is one of the more consequential numbers in the repository.

DeepSeek CPU inference: how it works and why it matters technically

To understand why this result is unusual, it helps to know where the memory bottleneck actually sits. Large mixture-of-experts models keep most of their parameters in the expert layers, but only activate a small subset of those experts per token. DeepSeek-V4-Flash has 284.33B total parameters but only 13.17B active per token. That means the compute per token is modest, but the storage requirement is enormous. Traditionally, that combination pushed people toward GPUs with very large VRAM, or toward splitting the model across multiple machines.

DeepSeek CPU inference inverts that. If the whole model fits in system RAM, and the active parameters per token are small enough that memory bandwidth, not compute, becomes the limit, then a high-core-count CPU with fast DDR5 and large caches becomes a viable inference engine. The project's engineering is aimed squarely at that memory-bandwidth limit: register tiling with AVX-512, careful L3 management, and a quantization format dense enough to squeeze the model under the 64 GB ceiling in the first place.

Two other design choices matter for business readers. The model uses a hybrid architecture with Mamba-2 state-space layers alongside the attention trunk, which the author credits with keeping the 256K context key-value cache down to 6 GiB. And the mixture-of-experts routing is pruned and re-tuned at inference time, trading a small amount of routing precision for speed, which is why the repository reports several configurations at different points on the speed-versus-fidelity curve rather than a single number.

For a regulated business, the significance is directional rather than immediate. The long-standing argument for keeping AI workloads on premises, covered in our Private LLM Deployment: Enterprise Self-Hosted AI (2026) guide and our On-Premise AI: Keep Your Data Off the Cloud overview, has been data control. The counterargument has been cost: frontier-class models demanded GPU hardware priced far beyond a departmental budget. If a 284B model genuinely runs on a 64 GB CPU box, the hardware side of that tradeoff shifts. Our earlier analysis in Why CPU Matters More Than GPU for Local LLMs argued that CPU-first inference was underrated for exactly this reason; this project is the strongest public demonstration of that thesis we have seen to date.

It is also worth noting what this does not change. A 284B model decoding at 9 to 14 tokens per second is usable for interactive chat and background analysis, and workable for single-user or small-team scenarios. It is not a replacement for a GPU cluster serving hundreds of concurrent users. The 30B-class results, at 25 to 111 tok/s, are the ones that look like everyday production speeds.

What the evidence does and does not show

The results come from the project author's own repository and documentation. The author publishes reproducibility instructions, a benchmark scorecard, a community leaderboard, and links to a preprint PDF in the repository's paper directory, which is more verifiability than most performance claims in this space offer. Even so, these are self-reported figures, and independent replication is not documented in the sources available to us. Businesses should treat the numbers as the author's measurements on specific hardware, not as vendor-validated benchmarks.

Several caveats are stated in the repository itself and deserve emphasis. The compressed models use a custom tensor type, GGML_TYPE_Q2_0 with type ID 40, and the author explicitly warns that opening them in unpatched llama.cpp, Ollama, or LM Studio will fail with an "unknown tensor type 40" error. Evaluation requires the project's own launcher or its standalone pre-compiled binary. That has governance implications: running a nonstandard inference binary is a supply-chain decision that should go through your normal software review, not a side experiment on a system that touches regulated data.

The quality picture is partially documented. The author reports a 100/100 score on the project's own 100-point quality gate for the Nemotron model in Fractal-64 and MXFP4 formats, and describes the default DeepSeek configuration as lossless. But a project-authored quality gate is not the same as independent evaluation on your workloads, and the faster DeepSeek configurations explicitly trade routing precision for speed. What the compression does to long-context reasoning, tool-calling reliability, and domain-specific accuracy beyond the published gate is not stated in the sources.

There are also open questions the sources do not answer. The repository does not document sustained thermal behavior on a laptop-class 9955HX under hours-long inference load, power consumption, behavior on Windows for the 284B model specifically (the Windows pack is described for hybrid GPU plus CPU use), or licensing terms for the compressed model artifacts beyond what the underlying model licenses impose. The DeepSeek-V4-Flash result depends on a specific memory ceiling; a machine with slightly less free RAM, or background services competing for memory, may not reproduce the fit. The project's own readiness test exists precisely because of that sensitivity.

Finally, the project is young and fast-moving, at version 2.4.0 at the time of writing. Benchmarks this early in a project's life tend to change, in both directions. The right posture for a regulated business is informed watching and controlled lab evaluation on isolated hardware, not production adoption. In Part 2 we will look at what this means for compliance and security posture, and how to evaluate it against frameworks like the CISA Cross-Sector Cybersecurity Performance Goals and NIST SP 800-61 Rev. 3.

The Compliance and Risk Implications for Regulated Organizations

The technical achievement in llama-modchip is interesting on its own. For a regulated business, the interesting part is what it changes about where AI workloads can physically live. If a 284B-parameter model can run entirely in RAM on a single CPU workstation with no GPU, the argument that frontier-class local AI requires expensive data center hardware gets weaker. That matters for any organization whose contracts, regulators, or risk assessments restrict where data may be processed.

But a smaller hardware footprint does not shrink a compliance obligation. Running a model locally changes the data flow; it does not change the duty to protect the data, log access, respond to incidents, and demonstrate control. CISA's Cross-Sector Cybersecurity Performance Goals 2.0, which align to the NIST Cybersecurity Framework 2.0 functions including the newer GOVERN function, frame this well: protections are a baseline of practices, not a purchase. A workstation that runs a large model locally still needs the same governance around it as any other system that touches sensitive information.

There are also risks specific to this project that a security leader should weigh before touching it. The repository distributes custom model formats and a custom binary. The project's own documentation warns that its Fractal-64 tensors will fail in unpatched llama.cpp, Ollama, or LM Studio, which means organizations would be running a bespoke executable rather than a widely audited upstream toolchain. The benchmarks are self-reported by the project, and while the repo points to reproducible benchmark scripts and a community leaderboard, independent validation is not documented in the sources we reviewed. Treat the performance claims as promising but unverified until you reproduce them on your own hardware.

Finally, note what the project does not claim. It does not claim to be a security product, does not mention encryption of model weights at rest, and does not address access control, audit logging, or multi-user isolation. Those gaps are yours to fill, and they are exactly the gaps that frameworks like NIST SP 800-171 and the HIPAA Security Rule care about.

What This Means for Regulated Industries

Defense Contractors

For defense industrial base contractors, the appeal is obvious: a capable model that never leaves your network is one way to keep controlled unclassified information away from public cloud AI services. The project's hybrid mode, which keeps the attention trunk on a GPU while routing MoE experts through CPU DDR5, and its pure-CPU mode both keep inference on premises. The Windows pack even runs a local web chat interface on the machine itself.

But CMMC obligations do not care how impressive the model is. If the system processes CUI, it lands inside your assessment boundary, and the NIST SP 800-171 controls that apply to any other system apply to it: access control, audit logging, configuration management, incident response, and FIPS-validated cryptography for protecting CUI confidentiality. A custom binary with no published security hardening guidance is a new system to inventory, baseline, and document. The practical move is to enclave it: keep the AI workstation inside a narrowly scoped boundary rather than letting it touch your general network, which reduces audit surface and keeps the assessment manageable.

Also remember the incident reporting clock. DFARS 252.204-7012 requires reporting a cyber incident affecting covered defense information to DoD within 72 hours of discovery, and requires preserving affected system images and monitoring data for at least 90 days afterward. A locally deployed AI system is another endpoint that must be covered by that capability. If you cannot monitor it, you cannot meet the requirement.

Healthcare

Healthcare organizations face the same structural opportunity with PHI. The HIPAA Security Rule requires a risk analysis and administrative, physical, and technical safeguards under 45 CFR Part 164; a self-hosted model can support those safeguards by eliminating transmission of PHI to a third-party AI service. Given that business associates accounted for about 66% of breached healthcare records in 2024, and that IBM's 2026 research puts the average healthcare breach at $6.64M, reducing third-party data flows is a defensible risk decision.

The caution is symmetric. A workstation running a 284B model is a system that creates, receives, or maintains PHI, so it inherits the full safeguard obligations: access controls, audit controls, and a place in your breach response plan under 45 CFR 164.404, 164.406, and 164.408. There is no HHS-issued HIPAA certification for an AI deployment; documentation aligned to NIST SP 800-66 Rev 2 and the HHS Office for Civil Rights audit protocol is how you demonstrate diligence. Before deploying, ask the vendor question the project does not answer: who at your organization can access the machine, and is every access logged?

Legal

Law firms sit in an awkward middle: they are often contractually bound to client confidentiality terms stricter than regulation, and they increasingly face client demands to warrant that AI tools never expose client data. A pure-CPU deployment on a workstation the firm already owns, with zero API calls, is a clean answer to that demand. The project's zero-API Claude Code mode, which wires an agentic coding tool to a fully local engine, illustrates the pattern: the intelligence is local, the data is local, and nothing transits the internet.

The diligence burden shifts rather than disappears. Firms should document the deployment the way they would document any vendor: what the software is, where it came from, what it does with data, and who reviewed it. The custom binary distribution model described in the repository means your review should include the executable itself, not just the model weights. For firms handling matters governed by client-imposed security requirements, put the AI workstation inside the same boundary as the matter data and apply the same matter-level access controls.

Financial Services

Financial institutions operate under overlapping obligations to protect customer information, maintain audit trails, and manage third-party risk. Local inference reduces third-party exposure, which is a genuine risk-reduction argument. The 256K context capability is also practically relevant: long documents, transaction histories, and regulatory filings are exactly the workloads where context length matters.

The governance questions are the same as everywhere else. Examiners and auditors will ask what the system does, who can use it, what it logs, and how you would know if it were compromised. None of those answers come from the model. They come from the surrounding controls, and a self-deployed custom binary needs the same configuration baseline, logging, and monitoring as any production system. If your institution reports to a board on AI risk, an on-premises deployment is a decision that belongs in that governance record, not a shadow IT project on a spare workstation.

A Practical Action Plan for This Month

You do not need to deploy anything to benefit from this story. Here is what a security or IT leader can do in the next few weeks:

  1. Inventory your current AI exposure. Identify every place staff use public AI tools with work data, including browser extensions and unapproved SaaS signups. IBM's 2025 research found 20% of breaches involved shadow AI, so assume you have some.
  2. Classify the data involved. Determine which of those uses touch CUI, PHI, cardholder data, client confidential data, or regulated personal information. That classification drives everything downstream.
  3. Write or update an AI acceptable use policy. State clearly which tools are approved, which data classes may be used with them, and who approves exceptions. Tie it to your existing acceptable use policy rather than creating a parallel document.
  4. Run the project's hardware readiness check. The repository offers a zero-download CPU readiness test. Two minutes on a candidate machine tells you whether this path is even plausible for your hardware before anyone invests time.
  5. If you evaluate the software, do it in an isolated test environment. Use a machine outside your production network, with no sensitive data, and evaluate the custom binary on its merits before any production decision.
  6. Reproduce the benchmarks yourself. The project publishes benchmark scripts and invites scorecard submissions. Your own numbers on your own hardware are the only ones that count for a capacity plan.
  7. Define the control boundary before deployment. Decide now where the system would sit, who can access it, how access is logged, and how it is monitored. If those answers do not exist, the deployment is not ready.
  8. Check your incident response coverage. Confirm your detection and response capability would see compromise of an AI workstation, and that your reporting obligations, including the DFARS 72-hour requirement where applicable, are covered for that asset.
  9. Update your risk register and, if applicable, your SSP. A new system handling sensitive data belongs in your system security plan, your POA&M if it introduces gaps, and your risk analysis.
  10. Get governance sign-off. For defense contractors this means the CMMC senior official or designated security officer; for healthcare, the security officer under 45 CFR 164.308(a)(2). Document the decision either way.

How Petronella Technology Group, Inc. Helps

Petronella Technology Group, Inc. works with defense industrial base and healthcare clients that cannot send CUI or PHI to a public cloud, which is precisely the constraint this story speaks to. Our private AI cluster and 24/7 AI-plus-human hybrid threat analysis stack exist because local AI only reduces risk when the surrounding controls are real. Our managed detection and response service provides the monitoring, AI-powered analytics, and human-led investigation that a locally deployed AI system needs, with guardrails our clients can show auditors: the AI never closes a ticket on its own, never touches production systems without human authorization, and every action is logged for CMMC and HIPAA audit.

On the compliance side, our compliance practice covers CMMC readiness, HIPAA, SOC 2, PCI DSS, and NIST SP 800-171, and every practitioner on the team holds the CMMC-RP designation. We are a Cyber AB Registered Provider Organization, RPO #1449, and we prepare contractors for assessment; only a C3PAO can certify. If a local AI deployment is on your roadmap, the right sequence is a scoped boundary, a documented control set, and monitoring in place before the first prompt. Our broader cybersecurity services and our guide to on-premise AI cover the infrastructure decisions that come first. For the underlying hardware question, see our analysis of why CPU matters more than GPU for local LLMs and our enterprise private LLM deployment guide.

Sources

Frequently Asked Questions

Can you really run a 284B model on a 64 GB workstation without a GPU?

According to the project's documentation, yes: DeepSeek-V4-Flash at 284.33B parameters runs at 57.61 to 58.9 GiB of resident memory at full 256K context on a 64 GB machine, with pure CPU decode of 9.06 to 10.74 tok/s and up to 11.5 to 14.7 tok/s with its speculative drafting. These are the project's own reported figures, so verify them on your hardware before planning around them.

Is DeepSeek CPU inference fast enough for real work?

It depends on the workload. The reported 9 to 15 tok/s decode range on the 284B model is usable for interactive drafting and analysis but not for high-volume automation. The smaller 30B MoE models are a different story, with reported speeds of 25.0 to 60.4 tok/s on a 16-core Zen 5 CPU and 111.6 tok/s on a Threadripper, which is fast enough for many production tasks.

Does running AI locally make us compliant?

No. Local deployment changes where data flows, not whether your controls satisfy your obligations. CMMC, HIPAA, and similar frameworks require access control, audit logging, incident response, and documented risk analysis regardless of where the model runs. Local AI removes one risk, third-party data exposure, and introduces a new system you must secure and document.

Is it safe to run the llama-modchip binary in a production environment?

Not without your own review. The project distributes a custom pre-compiled binary and custom tensor formats that do not work in standard llama.cpp, Ollama, or LM Studio, and the sources do not document independent security review of that binary. Evaluate it in an isolated environment first, and treat production adoption as a software supply chain decision.

What hardware do we need to try this?

The project targets modern CPUs with AVX-512 support and fast multi-channel memory, listing specific optimizations for AMD Zen 4 and Zen 5 chips, Intel processors, Apple M4 Pro, and AWS Graviton3. It also offers a hybrid mode that splits work between a GPU and system RAM. The repository includes a zero-download readiness test you can run in seconds on a candidate machine.

How do we decide between a local model and a cloud AI service?

Start with the data. If the workload involves CUI, PHI, cardholder data, or client confidential material and your contracts or regulations restrict third-party processing, local inference deserves serious consideration despite the operational overhead. If the workload involves public or low-sensitivity data, a vetted cloud service with contractual protections may be simpler. Document the decision either way, because that documentation is what an auditor or client will ask for.

If you want help thinking through whether local AI fits your compliance posture, call Penny, our AI assistant, at 919-348-4912, or reach Petronella Technology Group, Inc. through our contact form. The first conversation is a free 30-minute scoping consultation with a practitioner, not a sales script.

Get the 2026 Cybersecurity Survival Guide

Free, practical, and specific to regulated environments. We will email it to you.

No spam. Unsubscribe anytime.

Talk to Petronella Technology Group, Inc.
Private, on-premises AI and compliance for regulated data. Call 919-348-4912, get a free AI assessment, or explore our AI, cybersecurity, and compliance services.