Updates·October 9, 2026, 02:00

Guides: One flag can speed up large AI models on midrange graphics cards

AI-generated from the sources listed, with automated source checks.

The --n-cpu-moe flag in llama.cpp moves parts of an AI model into the computer's regular memory. According to two guides, it can deliver faster responses, but only if you measure your way to the right setting yourself.

If you run a large AI model locally on your own computer, your graphics card's memory can quickly become too small. The llama.cpp program has a flag that can help, and two guides describe how it works. The guides are based on users' own measurements and are not tests in themselves.

What the flag does

The flag is called --n-cpu-moe (short form -ncmoe) and is aimed at so-called MoE models. They activate only a fraction of their parameters for each word they write. That means the rarely used parts, known as experts, can sit in slower regular memory (RAM) without slowing things down much.

With the flag, the expert weights in the first N layers are placed in RAM, while the attention layers stay on the graphics card. The related flag --cpu-moe puts all experts in RAM, while --n-cpu-moe gives you layer-by-layer control.

Measure for yourself

The guide from OpenClawDC advises against copying other people's values. Instead, you should measure tokens per second with the llama-bench tool for several values of N, find the point where speed drops, and step back by one. According to the guide, if the graphics card's memory spills over into system memory, the result is a silent slowdown of 2-3 times. No error message appears.

As an example, it cites a measurement on an RTX 5090: 54 tokens/s at N=20, 64.5 at N=16 and 69.4 at N=12. At the next lower value, speed fell to 27.5. Users also report roughly 51-53 tokens/s for Qwen 3.6 35B on a 12 GB RTX 3060 and about 60 on an RTX 4080. Results depend on quantization, RAM speed, context length and processor.

New cache in llama.cpp

A new option, --moe-cache-mib, is a cache on the graphics card for experts that sit in RAM. It is off by default. The author of the associated pull request measured 2.20 times faster on an RTX 5090 and 1.62 times on an RTX 4090. These are the developer's own tests.

Independent reports showed smaller gains of 1.01-1.29 times on cards such as the RTX 3090 and RTX 5060 Ti 16 GB. The cache only applies to batches of up to 32 tokens and can degrade processing of the prompt itself. One report on a GLM model showed speed cut in half. For small cards, Developers Digest recommends tuning --n-cpu-moe by hand.

The measurements mentioned apply to cards with 12 GB and up. The sources contain no independent measurements on cards with 8 GB.