--- title: Running a Frontier-Class Local LLM on One GPU description: A local LLM on one GPU stopped being a toy in 2026. Open weights from Qwen and DeepSeek now run on a single consumer card, and the throughput surprised me. canonical: https://lukeocodes.dev/local-llm-gpu type: article locale: en_GB published: 2026-09-06T00:00:00.000Z modified: 2026-09-06T00:00:00.000Z authors: - Luke Oliff section: AI tags: - AI - Models - open weights - qwen - deepseek - ollama - vllm feed: https://lukeocodes.dev/rss.xml search: https://lukeocodes.dev/search.json?q={query} schema: - "@context": https://schema.org "@graph": - "@type": WebSite "@id": https://lukeocodes.dev/#website url: https://lukeocodes.dev name: Luke Oliff alternateName: lukeocodes inLanguage: en publisher: "@id": https://lukeocodes.dev/authors/luke#person description: Personal blog and writing by Luke Oliff — developer relations, voice AI, developer experience. potentialAction: - "@type": SearchAction target: "@type": EntryPoint urlTemplate: https://lukeocodes.dev/blog/search?q={search_term_string} query-input: required name=search_term_string - "@type": Person "@id": https://lukeocodes.dev/authors/luke#person name: Luke Oliff url: https://lukeocodes.dev/authors/luke description: Engineer and developer relations leader. Writing about developer relations, AI, and the voice industry, from the UK. mainEntityOfPage: https://lukeocodes.dev/authors/luke givenName: Luke familyName: Oliff image: https://lukeocodes.dev/authors/luke.jpg jobTitle: Engineer and developer relations leader sameAs: - https://www.linkedin.com/in/lukeocodes - https://github.com/lukeocodes - https://bsky.app/profile/lukeocodes.dev - https://x.com/lukeocodes - https://hashnode.com/@lukeocodes - https://dev.to/lukeocodes knowsAbout: - Developer Relations - Developer Experience - Voice AI - Text-to-Speech - Answer Engine Optimisation - Search Engine Optimisation - Open Source - Developer Tooling - API Design - Speech-to-Text - Voice Agents - Community Building knowsLanguage: - English nationality: British alumniOf: - "@type": Organization name: The Open University - "@context": https://schema.org "@graph": - "@type": Person "@id": https://lukeocodes.dev/authors/luke#person name: Luke Oliff url: https://lukeocodes.dev/authors/luke description: Engineer and developer relations leader. Writing about developer relations, AI, and the voice industry, from the UK. mainEntityOfPage: https://lukeocodes.dev/authors/luke givenName: Luke familyName: Oliff image: https://lukeocodes.dev/authors/luke.jpg jobTitle: Engineer and developer relations leader sameAs: - https://www.linkedin.com/in/lukeocodes - https://github.com/lukeocodes - https://bsky.app/profile/lukeocodes.dev - https://x.com/lukeocodes - https://hashnode.com/@lukeocodes - https://dev.to/lukeocodes knowsAbout: - Developer Relations - Developer Experience - Voice AI - Text-to-Speech - Answer Engine Optimisation - Search Engine Optimisation - Open Source - Developer Tooling - API Design - Speech-to-Text - Voice Agents - Community Building knowsLanguage: - English nationality: British alumniOf: - "@type": Organization name: The Open University - "@type": BlogPosting "@id": https://lukeocodes.dev/local-llm-gpu#article headline: Running a Frontier-Class Local LLM on One GPU alternativeHeadline: Running a Frontier-Class Local LLM on One GPU description: A local LLM on one GPU stopped being a toy in 2026. Open weights from Qwen and DeepSeek now run on a single consumer card, and the throughput surprised me. datePublished: 2026-09-06T00:00:00.000Z dateModified: 2026-09-06T00:00:00.000Z url: https://lukeocodes.dev/local-llm-gpu mainEntityOfPage: https://lukeocodes.dev/local-llm-gpu inLanguage: en author: - "@id": https://lukeocodes.dev/authors/luke#person isPartOf: "@id": https://lukeocodes.dev/#website keywords: AI, Models, open weights, qwen, deepseek, ollama, vllm articleSection: AI about: - "@type": SoftwareApplication name: Qwen - "@type": SoftwareApplication name: DeepSeek abstract: In 2026 you can run a genuinely capable open-weight model on a single 16-24GB consumer GPU. Quantization makes it fit, the engine you pick decides your throughput, and for a lot of real work you no longer need a frontier API at all. speakable: "@type": SpeakableSpecification cssSelector: - .post-tldr - h1 image: https://lukeocodes.dev/og.png?h=Running+a+Frontier-Class+Local+LLM+on+One+GPU&d=A+local+LLM+on+one+GPU+stopped+being+a+toy+in+2026.+Open+weights+from+Qwen+and+DeepSeek+now+run+on+a+single+consumer+card%2C+and+the+throughput+surprised+me.&e=Writing wordCount: 504 timeRequired: PT3M publisher: "@id": https://lukeocodes.dev/authors/luke#person - "@type": BreadcrumbList itemListElement: - "@type": ListItem position: 1 name: Home item: https://lukeocodes.dev - "@type": ListItem position: 2 name: Writing item: https://lukeocodes.dev/blog - "@type": ListItem position: 3 name: Running a Frontier-Class Local LLM on One GPU item: https://lukeocodes.dev/local-llm-gpu - "@type": FAQPage "@id": https://lukeocodes.dev/local-llm-gpu#faq mainEntity: - "@type": Question name: Can you really run a good model on one consumer GPU? acceptedAnswer: "@type": Answer text: Yes, in 2026 you can. Alibaba's Qwen line shipped a smaller variant that runs on a single 16-24GB card under an Apache-2.0 license, and quantized DeepSeek checkpoints fit too. It won't match the largest frontier models on the hardest tasks, but for coding help, extraction and summarization it's more than enough. - "@type": Question name: Ollama or vLLM for a local LLM GPU setup? acceptedAnswer: "@type": Answer text: Ollama is the easiest way in and fine for single-user use. vLLM pulls far ahead under concurrency thanks to continuous batching, with 2026 benchmarks showing large multiples over Ollama when several requests overlap. Pick Ollama to start, move to vLLM when you're serving more than yourself. - "@type": Question name: Does quantization ruin quality? acceptedAnswer: "@type": Answer text: Not meaningfully at 4-bit for most tasks. You trade a little accuracy for a model that fits in consumer VRAM and runs at usable speed. Go too aggressive (2-3 bit) and it degrades, so 4-bit is the sweet spot most people land on. --- [Luke Oliff.](/) - [Blog](/blog) - [CV](/cv) - [Uses](/uses) - [User manual](/user-manual) - [Living with ADHD](/living-and-working-with-adhd) - [RSS](/rss.xml) [Luke Oliff · Writing](/blog) # Running a Frontier-Class Local LLM on One GPU Sep 2026 · [AI](/blog/category/ai) · 3 min read · [Luke Oliff](/authors/luke) **TL;DR:** In 2026 you can run a genuinely capable open-weight model on a single 16-24GB consumer GPU. Quantization makes it fit, the engine you pick decides your throughput, and for a lot of real work you no longer need a frontier API at all. A local LLM on one GPU used to be a party trick. In 2026 it's a real option, and that shift matters more than another closed-model launch. You can now run a genuinely useful open-weight model on a single consumer card, offline, with nobody metering your tokens. I set one up expecting a toy. I got something I actually use. ## What fits now The unlock was the open-weight labs shipping small-but-serious variants. Alibaba's Qwen line put out a version that runs on a single 16-24GB GPU under Apache-2.0, and quantized DeepSeek checkpoints fit on consumer hardware too. Four-bit quantization is the trick that makes it work: you shrink the weights enough to fit in VRAM and lose very little on ordinary tasks. Coding help, extraction, classification, summarization, all fine. You won't beat the biggest frontier model on the hardest reasoning, but most of my day isn't the hardest reasoning. The licenses are the other half. Apache and MIT mean you can run these offline, fine-tune them, and keep whatever you build. No rate limit, no TTL games, no "your framework is bypassing our pricing" email. ## The engine decides your speed Here's the part that surprised me. The model matters less than the serving engine once you care about throughput. Ollama is the gentlest on-ramp, and for one user poking at a model it's genuinely fine. But the moment you have concurrent requests, vLLM pulls away hard, because it does continuous batching and Ollama historically didn't. 2026 benchmarks put vLLM at large multiples over Ollama under load. On Apple silicon the calculus shifts again, with the MLX backend now a first-class option. So the honest answer to "how fast is a local LLM GPU setup" is "depends what you run it with," and that's a knob most people don't know they're leaving on the floor. My setup: Ollama for quick interactive use on the laptop, vLLM on the box with the real GPU when I'm serving anything that overlaps. ## Why I bothered Cost is the obvious reason, but it's not the main one for me. It's that the thing runs on my hardware and I can see what it's doing. No request leaves the machine. No provider decides mid-quarter that my tokens should cost more. When the frontier labs spent 2026 letting their own models break out of sandboxes, "the model runs entirely on a box I control" started to sound less like hobbyism and more like sense. It's not all upside. You manage your own VRAM, you pick your own quantization, and you eat the setup time. The largest closed models are still ahead on the genuinely hard stuff. But the gap that used to justify sending every token to an API has narrowed to a specific set of tasks, and for everything else the answer now lives on one GPU under my desk. Try it before you renew a plan. You may find, like I did, that a surprising share of what you were paying for runs fine at home. [← Newer: Scripting LLMs From the Terminal With the llm CLI](/llm-cli-terminal) [All posts](/blog) → ## Search this site `GET https://lukeocodes.dev/search.json?q=` returns matching posts as JSON, each with its page URL and its markdown URL. Add `&limit=<1-50>` for more than 10. How it works: [/blog/search](https://lukeocodes.dev/blog/search.md). ```sh curl "https://lukeocodes.dev/search.json?q=developer+relations" ``` --- © 2026 Luke Oliff. Posts live at the root of this site. - [GitHub](https://github.com/lukeocodes) - [LinkedIn](https://www.linkedin.com/in/lukeocodes) - [X](https://x.com/lukeocodes) - [Bluesky](https://bsky.app/profile/lukeocodes.dev) - [Email](mailto:luke@lukeoliff.com) - [RSS](/rss.xml) ## Cookies This site sets one analytics cookie, from Google Analytics, and only in a browser that accepts the cookie banner. A client that does not run scripts sets no cookies, so there is nothing to accept or decline here. [Cookie policy](/cookies)