CPU Bottleneck in LLM Inference: Why Your GPU Sits Idle and How to Fix It

You installed a powerful GPU, loaded a local LLM in Ollama or llama.cpp, hit generate — and then opened nvidia-smi to find GPU utilization sitting at 20 or 30 percent while tokens trickle out. Your first instinct is that something is broken. Sometimes it is. But often, low GPU utilization during LLM inference is the system telling you exactly where the real limit is — and it usually isn’t the GPU’s compute power.

A CPU bottleneck in LLM inference looks different from the GPU bottleneck gamers talk about. In games, the CPU can’t feed frames fast enough. In LLM inference, the “bottleneck” is usually memory bandwidth, batching, or layers stranded on the CPU — and the fix depends entirely on which one you have. This guide explains why your GPU sits idle, how to diagnose the real limiter in about a minute, and seven fixes that actually work.

What “Low GPU Utilization” Actually Means

When nvidia-smi reports GPU utilization, it measures what fraction of time the GPU’s compute cores (the streaming multiprocessors) were active — not how hard the memory system is working. A GPU can show low compute utilization while its memory bandwidth is completely saturated. That distinction is the key to this entire topic.

LLM inference also has two very different phases. Prefill processes your whole prompt at once — big matrix multiplications, high arithmetic intensity, and the GPU’s compute cores stay busy. Decode generates tokens one at a time — tiny operations that mostly involve reading data from memory. Most of your waiting happens in decode, and decode is where utilization looks disappointingly low.

So the first rule: if your GPU utilization is low while generating tokens, that is expected behavior up to a point. The question is whether it’s healthy-low or problem-low — and the sections below show you how to tell.

Why LLM Decoding Is Memory-Bound, Not Compute-Bound

To generate a single token, the GPU must read the model’s entire weights from VRAM, plus the growing KV cache (the stored attention keys and values for your conversation). The actual math per token is small; the data movement is enormous. The GPU finishes the computation quickly, then waits for the next weights to arrive from memory.

This is why memory bandwidth matters far more than raw compute power (TFLOPs) for token generation. During decode, the governing principle is simple: optimize bytes per token, not FLOPs. Anything that increases bytes moved per token — a larger model, higher precision, a longer conversation filling the KV cache — slows generation without raising compute utilization.

The 5 Real Reasons Your GPU Utilization Is Low

1. Memory-Bandwidth-Bound Decoding (Often Normal)

This is the default state of single-user LLM inference and the most misunderstood. With one conversation generating one token at a time, there simply isn’t enough parallel work to keep thousands of GPU cores busy. The memory system is the limit, the compute cores wait, and utilization reads low. If your token speed feels reasonable for your hardware and nothing else looks wrong, this is likely all it is — not a problem to fix, but physics to accept.

2. Partial CPU Offload in llama.cpp (Layers Split Between CPU and GPU)

When a model doesn’t fully fit in VRAM, llama.cpp lets you keep some layers on the GPU (-ngl) and run the rest on the CPU. CPU-processed layers run dramatically slower than GPU layers, so the GPU finishes its portion and then sits idle waiting for the CPU to catch up. The telltale sign is utilization oscillating — spiking high, dropping to near zero, spiking again — rather than sitting steadily low. Every layer you move from CPU to GPU removes one of these waiting periods.

3. Batch Size of 1 — No Continuous Batching

GPUs are built for parallelism. Processing one request at a time leaves most of that parallelism unused — the same weights get loaded from memory to produce a single token. Serving engines like vLLM use continuous batching to group many requests into each forward pass: the weights are loaded once and serve many tokens, which raises both utilization and total throughput. Ollama, by contrast, is designed around simple single-user interaction and doesn’t batch this way, which is one reason it shows lower utilization than vLLM on the same hardware.

4. CPU-Side Work: Tokenization, Preprocessing, and Draft Tokens

Before the GPU can do anything, the CPU must tokenize your prompt, schedule work, and manage the serving pipeline. Under concurrent load, tokenization alone can consume a large share of total latency — published analyses of serving stacks have found CPU-side scheduling and tokenization accounting for a substantial fraction of end-to-end time, leaving the GPU starved. This shows up as a genuine CPU bottleneck: CPU cores pinned near maximum while the GPU waits. It’s especially visible with multimodal models (the CPU decodes images before the GPU ever sees them) and with speculative decoding, where the CPU proposes draft tokens the GPU then verifies.

5. PCIe and Interconnect Bottlenecks

Data constantly moves between system RAM and VRAM — offloaded layers, KV cache, prompt data. That traffic crosses the PCIe bus, and an older or narrower PCIe link throttles it. The motherboard and PCIe limits that affect gaming GPUs apply here too. Multi-GPU setups have their own version: tensor parallelism across GPUs connected only by PCIe (no NVLink) introduces a silent communication bottleneck that shows up as mysteriously low per-GPU utilization.

Diagnose Your Bottleneck in 60 Seconds

Don’t change settings blindly. Run through this flowchart while your model is generating — open nvidia-smi dmon (or watch -n 1 nvidia-smi) in one terminal and a CPU monitor like htop in another, then follow the table:

Step	What to check	What you see	Verdict
1	GPU-Util pattern in nvidia-smi	Oscillating wildly (high → near-zero → high)	CPU offload bottleneck — the GPU is waiting on CPU-side layers. Go to Fix 2.
2	GPU-Util pattern	Steadily low, no oscillation	Continue to step 3.
3	Memory-Util / VRAM in nvidia-smi	VRAM nearly full, compute util low	Memory pressure — KV cache or model barely fits. Go to Fix 3.
4	Memory-Util / VRAM	Plenty of free VRAM, compute util low	Continue to step 5.
5	CPU usage in htop during generation	One or more CPU cores pinned near 100%	CPU-side bottleneck — tokenization, preprocessing, or offloaded layers. Go to Fixes 2 and 5.
6	CPU usage	CPU relaxed, GPU steadily low, single user	Batching / memory-bandwidth limit — normal for single-stream decode. Go to Fixes 1 and 4; also read Fix 7 before “fixing” anything.
7	Multi-GPU: nvidia-smi topo -m	GPUs linked by PCIe only, no NVLink	Interconnect bottleneck. Go to Fix 6.

This single pass tells you which of the five causes you’re dealing with. Now apply only the fixes that match your verdict.

7 Fixes That Actually Raise GPU Utilization

1. Raise Batch Size — or Switch to Continuous Batching

If your diagnosis points to single-stream underutilization and you serve multiple users or requests, batching is the highest-leverage fix available. In vLLM, raising --max-num-seqs and --max-num-batched-tokens lets each forward pass do more useful work per memory read. The general principle behind fixing CPU bottlenecks applies here too: the fix is about feeding the fast component enough parallel work, not about making the fast component faster. Note the tradeoff — bigger batches raise throughput but also per-request latency, so tune for your use case.

2. Move More Layers Onto the GPU

If utilization oscillates and the CPU is saturated, you’re paying the split-execution tax. In llama.cpp, raise -ngl to put more layers on the GPU; the goal is the entire model on GPU if VRAM allows. If it doesn’t fit, that’s what Fix 3 is for — shrink the model rather than stranding layers on the CPU. Each layer moved to the GPU removes a synchronization point where the GPU idles.

3. Quantize Weights — and the KV Cache

Quantization attacks the memory-bandwidth problem directly: fewer bytes per weight means fewer bytes moved per token. Dropping from 16-bit to 4-bit weights roughly quarters the memory traffic of the model itself. For long conversations, quantizing the KV cache matters just as much, since the cache grows with every token and competes with weights for VRAM and bandwidth. Both vLLM and llama.cpp support KV cache quantization — for long-context work it’s frequently the single most effective change.

4. Pick the Right Engine for the Job

Engine choice sets your utilization ceiling. Ollama is the simplest path — one command and you’re chatting — but it serves requests essentially one at a time, so utilization stays modest. llama.cpp (via llama-server) gives you fine control over offloading, threading, and batching for single-machine use. vLLM is built for throughput: continuous batching, PagedAttention for KV cache efficiency, and prefix caching — the right choice when you serve concurrent users and want the GPU saturated. Match the engine to the workload instead of fighting the wrong one’s architecture.

5. Feed the CPU: Threads, Instruction Sets, and Faster RAM

When the CPU is the limiter, small configuration details matter enormously. Set llama.cpp’s thread count (-t) to your physical core count — hyperthreaded logical cores can actually slow inference down. Check that your CPU supports AVX2 or AVX-512 (lscpu | grep -i avx); vector instructions make a large difference to CPU-side layer speed. And remember that CPU inference and CPU-side work are memory-bandwidth-bound on the host side too — faster system RAM (DDR5 over DDR4, more memory channels populated) directly raises the ceiling for anything the CPU touches.

6. Check PCIe Generation and Multi-GPU Topology

Run nvidia-smi topo -m and confirm what your GPU is actually connected with. A GPU sitting in an older PCIe slot, or a multi-GPU setup doing tensor parallelism over PCIe without NVLink, creates a transfer bottleneck that no software tuning will fix. For multi-GPU inference without fast interconnects, data parallelism (separate model copies per GPU) usually beats tensor parallelism (splitting layers across GPUs). This is one of those “check once, benefit forever” items.

7. Know When to Stop — Healthy Low Utilization Is Real

The most underrated fix is recognizing when there is nothing to fix. Single-user chat with a model that fits comfortably in VRAM, a quiet CPU, and steady (if low) GPU utilization is the normal operating point of memory-bound decoding — the GPU is waiting on memory bandwidth, which is already the fundamental limit. Chasing 90%+ compute utilization on a single interactive session usually means adding batching or concurrency you don’t need, at the cost of latency you will feel. Optimize until the experience is good, then stop.

CPU vs Memory Bandwidth vs GPU: Which Is Your Limit?

These three limiters feel identical from the user’s chair — slow tokens — but they point at completely different upgrades. Use these qualitative rules:

• It’s the CPU if host cores are pinned during generation, utilization oscillates (split execution), or speed collapses specifically on multimodal inputs and long prompts that need heavy preprocessing. The answer is a faster CPU, more cores for preprocessing, or moving work off the CPU (full GPU offload).
• It’s memory bandwidth if the CPU is relaxed, VRAM isn’t full, utilization is steadily low, and generation speed scales with quantization level or context length. The answer is reducing bytes per token (quantization, shorter context, smaller model) or a GPU with higher memory bandwidth — not a GPU with more TFLOPs.
• It’s the GPU if compute utilization is genuinely high during prefill, or you’re running out of VRAM entirely and can’t fit the model no matter the quantization. The answer is more VRAM or more bandwidth — matched to which of the two is actually exhausted.

Notice how different this is from the classic CPU vs GPU bottleneck framing in gaming. In games, the question is which component caps your frame rate. In LLM inference, the question is usually which data path caps your token rate — and the data path is memory bandwidth far more often than either processor.

Frequently Asked Questions

Why is my GPU utilization low during LLM inference?

Almost always one of five things: token decoding is memory-bandwidth-bound (normal), some model layers run on the CPU while the GPU waits, you’re generating one token stream at a time with no batching, the CPU is saturated with tokenization or preprocessing, or a slow PCIe link throttles CPU-GPU transfers. Run the 60-second diagnosis above to find which one applies to you.

Is low GPU utilization normal during LLM decoding?

Yes — to a degree. Decoding generates one token at a time, and each token requires re-reading the model weights from memory. That makes decode memory-bound: compute cores idle while waiting on memory bandwidth. Steadily low utilization with a relaxed CPU and reasonable speed is healthy. Oscillating utilization or a maxed-out CPU alongside it signals a real problem.

How do I increase GPU utilization in llama.cpp?

First, put as many layers on the GPU as VRAM allows (raise -ngl) — partial CPU offload is the most common utilization killer. Second, set threads to your physical core count. Third, quantize the model and KV cache to reduce memory traffic. Fourth, enable Flash Attention if your build supports it. Each step removes a different reason the GPU sits idle.

Does batch size affect GPU utilization in LLM inference?

Enormously. A batch size of one leaves most GPU parallelism unused because the same weights are loaded from memory for a single token. Batching (especially the continuous batching in vLLM) amortizes each memory read across many tokens, raising both utilization and total throughput. This matters most when serving multiple users; for single-user chat, the latency tradeoff usually isn’t worth it.

Is my CPU or my memory bandwidth the bottleneck?

Check both monitors during generation. CPU pinned near maximum while the GPU oscillates or idles points to the CPU (offloaded layers, tokenization, preprocessing). A relaxed CPU with steadily low GPU compute utilization points to memory bandwidth — the classic decode limit. The fixes are completely different, so diagnose before you spend money on hardware.

Should I use vLLM or Ollama for higher GPU utilization?

It depends on your workload. vLLM will show higher utilization because continuous batching, PagedAttention, and prefix caching are designed to saturate the GPU under concurrent load. Ollama prioritizes simplicity for single-user interaction and doesn’t batch the same way. For one person chatting, Ollama’s lower utilization is fine; for serving many users, vLLM’s architecture wins.

Low GPU utilization during LLM inference is a symptom, not a diagnosis. Work through the 60-second check, match your symptom to one of the five causes, and apply only the fixes that fit — most “slow local LLM” problems turn out to be batching, offloading, or memory bandwidth, not a weak GPU.