Running a Frontier-Class Local LLM on One GPU

A local LLM on one GPU used to be a party trick. In 2026 it’s a real option, and that shift matters more than another closed-model launch. You can now run a genuinely useful open-weight model on a single consumer card, offline, with nobody metering your tokens.

I set one up expecting a toy. I got something I actually use.

What fits now

The unlock was the open-weight labs shipping small-but-serious variants. Alibaba’s Qwen line put out a version that runs on a single 16-24GB GPU under Apache-2.0, and quantized DeepSeek checkpoints fit on consumer hardware too. Four-bit quantization is the trick that makes it work: you shrink the weights enough to fit in VRAM and lose very little on ordinary tasks. Coding help, extraction, classification, summarization, all fine. You won’t beat the biggest frontier model on the hardest reasoning, but most of my day isn’t the hardest reasoning.

The licenses are the other half. Apache and MIT mean you can run these offline, fine-tune them, and keep whatever you build. No rate limit, no TTL games, no “your framework is bypassing our pricing” email.

The engine decides your speed

Here’s the part that surprised me. The model matters less than the serving engine once you care about throughput.

Ollama is the gentlest on-ramp, and for one user poking at a model it’s genuinely fine. But the moment you have concurrent requests, vLLM pulls away hard, because it does continuous batching and Ollama historically didn’t. 2026 benchmarks put vLLM at large multiples over Ollama under load. On Apple silicon the calculus shifts again, with the MLX backend now a first-class option. So the honest answer to “how fast is a local LLM GPU setup” is “depends what you run it with,” and that’s a knob most people don’t know they’re leaving on the floor.

My setup: Ollama for quick interactive use on the laptop, vLLM on the box with the real GPU when I’m serving anything that overlaps.

Why I bothered

Cost is the obvious reason, but it’s not the main one for me. It’s that the thing runs on my hardware and I can see what it’s doing. No request leaves the machine. No provider decides mid-quarter that my tokens should cost more. When the frontier labs spent 2026 letting their own models break out of sandboxes, “the model runs entirely on a box I control” started to sound less like hobbyism and more like sense.

It’s not all upside. You manage your own VRAM, you pick your own quantization, and you eat the setup time. The largest closed models are still ahead on the genuinely hard stuff. But the gap that used to justify sending every token to an API has narrowed to a specific set of tasks, and for everything else the answer now lives on one GPU under my desk.

Try it before you renew a plan. You may find, like I did, that a surprising share of what you were paying for runs fine at home.