Which local AI models are actually useful?

In short

• gpt-oss:20b is the best all-rounder for a 16–24 GB card. It fits entirely on the card, runs at about 50 tokens/s, reads long source files quickly and got every standard test right. Its weakness showed only on the hardest work.
• Only gpt-oss:120b and gemma4:26b passed all nine real-work runs. These are real changes to existing code. qwen3.6:27b-coding passed 8 but is slow; gpt-oss:20b passed 6 and failed every attempt to extend the BASIC interpreter. Here the 120B runs mostly from system RAM at 15–20 tokens/s.
• gemma4:26b is the find of these tests: as good as the 120B at real work, in under a third of the memory. It takes about 18.5 GB of graphics memory with a 32K context, so it wants a 24 GB card. On this 20 GB card, which also runs the Windows desktop, that left the card nearly full and the model ran at 9–37 tokens/s instead of 74–88. It also reasons at length, about 11,000 tokens per real-work answer against 3,000 for the 120B.
• Mixture-of-experts models are what make a small card work. Dense 24–32B models either crawl once they spill out of graphics memory, spend so long reasoning that answers take many minutes, or skip reasoning and get it wrong.
• A 24 GB card is the sweet spot. From there up to 72 GB, more graphics memory buys speed, not capability: nothing that fits in 32–64 GB did better than the two models above.
• Holding gpt-oss:120b in fast memory is the next step, but its value is unproven. On these tests it did no better than gemma4:26b. The cheapest way to do it is a Ryzen AI Max+ machine: about £3,500 with 128 GB, at roughly 50 tokens/s. A 72 GB NVIDIA card is two to three times faster, and reads prompts far faster, but costs about three times as much as the whole Ryzen machine.

On this page

1. The test machine
2. How the models were tested
3. Results
4. What the results mean
5. Setting up Ollama on Windows
6. What more money buys
7. Host machine and operating system
8. Buying a used workstation card
9. Local or cloud?
10. Recommendations
11. Caveats, and what went wrong
12. Run the tests yourself

How the models were tested

Every answer is graded automatically. Code is compiled with gcc under AddressSanitizer and UndefinedBehaviorSanitizer, which catch memory errors and undefined behaviour as the code runs, and then compared against a reference implementation on thousands of cases. Each test was itself tested first: a correct reference solution must pass, and deliberately broken versions must fail.

The real-work tasks were added after the first round, because the best models were getting everything else right and the tests had stopped telling them apart. They are closer to what an assistant is actually asked to do: change code it didn't write without breaking what already works. The interpreter task, for example, requires that RETURN discards any FOR loops the subroutine opened, and that a NEXT inside a subroutine can't close the caller's loop.

Most of these models reason before answering, and that reasoning counts towards a token budget. Answers were capped at 12,000 tokens for the standard sets and 16,000–24,000 for the harder ones. A model still reasoning at the cap scores a fail. In normal use there's no cap, but you wait for every one of those tokens.

The models

All except gpt-oss are the standard 4-bit (Q4_K_M) downloads from Ollama's library.

Results

Scores

Real work is 3 tasks × 3 seeds. A dash means not run: the real-work tasks came last, so they went to the leading models and the coding-tuned ones. Times are for the 6 coding tasks, 6 general questions and the PicoMite questions, on this machine.

The real-work tasks in detail

* gemma4:26b ran at 9–22 tokens/s on these tasks because with a 32K context it left the card nearly full. With a little memory to spare it runs at 46–88 tokens/s, which would make these 2–6 minutes; see setup.

gpt-oss:20b failed the interpreter extension three different ways: its GOSUB returned to the wrong place, it broke how existing FOR loops are skipped, and once its code didn't compile because it called a function before declaring it. gpt-oss:120b got all three right, including the rules about loops inside subroutines, and so did gemma4:26b.

qwen3.6:27b-coding came close, with 8 of 9: one allocator run read past the end of an array. But it's a dense model, so it's slow on this card, about 23 tokens/s, and it reasoned for about 12,000 tokens per answer: around eight minutes an answer here.

qwen3.6:35b-a3b-coding passed all three tasks with one seed and none of the six runs with the other two. Three times it was still reasoning at 24,000 tokens and never wrote any code, once its code didn't compile, and twice it failed the tests. devstral-small-2 answered in one to four minutes and failed all nine runs. It made the same mistake in the interpreter every time, not discarding the subroutine's own FOR loop on RETURN, and its allocator miscounted free memory within the first 30 operations.

Speed

Tokens per second at 32K context unless stated. A token is roughly three-quarters of a word; anything above about 10 tokens/s reads comfortably.

What the results mean

Mixture-of-experts models are the key on a small card

A dense model reads all of its parameters for every token. Once a dense 32B model spills out of a 20 GB card it drops to 2–3 tokens/s, and one coding task took 45 minutes. A mixture-of-experts model picks a few experts per token: qwen3.6:35b-a3b reads only about 3B of its 35B parameters, so it runs at 50–60 tokens/s with a fifth of itself in system RAM. That's what lets an 80B or 117B model run at all on this machine.

How efficiently a model reasons matters as much as its speed

gpt-oss reasons in a few hundred to a few thousand tokens. The qwen3.6 models often used 10,000 or more on a single short function, and four times out of six qwen3.6:35b-a3b-coding was still reasoning at 12,000 tokens and never wrote any code. Doubling the budget to 24,000 tokens didn't help: it ran out four times out of six again, and still passed only two. Given 60,000 tokens for the printf task, it stopped at about 20,000 with code that still failed. The problem is how it reasons, not how long it's allowed to. The dense qwen3.6:27b-coding did get there: its one standard-set failure, the ring buffer, passed with the bigger budget, after 19,900 tokens and 20 minutes. At 50 tokens/s, 12,000 tokens is four minutes of waiting for one function.

gemma4:26b reasons at length too, about 11,000 tokens for a typical real-work answer against 3,000 for gpt-oss:120b and 7,000 for gpt-oss:20b, but it finishes, and it gets the answer right. On a card that holds it entirely, that's a few minutes per answer.

The easy tests stopped separating the best models

gpt-oss:20b and gpt-oss:120b both scored full marks on the standard coding set, the general questions and the PicoMite questions. Only the real-work tasks showed the difference, and it was in the task that most resembles everyday assistant work: changing code you didn't write, with rules that interact. That's where model size showed.

Long context is cheap with these models

Every model answered all five PicoMite questions correctly after reading 22,000 tokens of Commands.c, including exact error strings and why cmd_for removes stale stack entries. The better MoE models read that much source in 10–25 seconds on this card.

Coding specialists that don't reason are fast but sloppy

qwen3-coder-next answers in seconds without reasoning first, and its code shows it: it treated a lowercase rem as an ordinary word, divided by zero in the square root, and gave -7 MOD 2 as 1 rather than C's −1. devstral-small-2 is the same kind of model and did no better: three of six standard coding tasks, neither hard task, and none of the nine real-work runs. Being dense, it's also slow on this card, at 12–16 tokens/s.

RAM speed matters only for models that spill

Anything that fits on the card doesn't care how fast system RAM is. For the big models it's capacity that counts: 96 GB holds the 65 GB gpt-oss:120b with room for everything else. Here the 48 GB modules run at 4400 MT/s with two fitted and 3600 with four, so filling all four slots for 192 GB would make every spilling model about 18% slower. It's only worth it for models over about 80 GB.

Setting up Ollama on Windows

Ollama's defaults are conservative. These user environment variables, followed by restarting Ollama, roughly doubled the speed of a model that spills into RAM (qwen3:32b at 8K context went from 4.5 to 8.1 tokens/s):

OLLAMA_FLASH_ATTENTION=1        faster attention, less memory
OLLAMA_KV_CACHE_TYPE=q8_0       halves the memory used by the context
OLLAMA_NUM_PARALLEL=1           one request at a time, no wasted context slots
OLLAMA_KEEP_ALIVE=30m           keep the model loaded between questions
OLLAMA_MODELS=D:\Ollama\models  optional: keep models off the system drive (and out of Dropbox)

Leave the card some headroom

On Windows the desktop and ordinary apps hold graphics memory: here the desktop compositor took 1.4–1.7 GB, Phone Link up to 2 GB, and Explorer, Firefox and EasyEDA about 0.3 GB each. Ollama didn't allow for that. On every load it assumed the same 19.3 GB of the 20 GB card was free, so a model that fits by Ollama's sums can leave the card all but full, and when that happens it slows right down. Ollama still reports it as 100% on the card.

gemma4:26b shows it clearly. With a 32K context it took about 18.5 GB of the card, leaving roughly half a gigabyte. It generated at 9–37 tokens/s, and the very same answer ran at 13 tokens/s on one load and 37 on the next. With a 16K context it ran at 74–88 tokens/s. With the full 32K context and 2 of its 30 layers deliberately moved to the processor, it ran at 46–56 tokens/s: faster with part of it off the card than with all of it on. The likely cause is Windows moving memory between the card and system RAM to make room, though that isn't proven here.

• If a model that should fit runs slower than expected, try a smaller context, or move a couple of layers to the processor with Ollama's num_gpu option (here num_gpu 28), and compare speeds.
• Task Manager's Shared GPU memory isn't a reliable sign of trouble. llama.cpp deliberately keeps some buffers, such as the vocabulary table, in pinned system memory, and they show up there: 1.1–1.4 GB for gemma4:26b even when it was running at full speed.
• If the PC has integrated graphics, plugging the monitor into the motherboard should leave the whole card for models. That wasn't tested here.
• NVIDIA's driver lets CUDA programs overflow into system RAM rather than fail, controlled by CUDA – Sysmem Fallback Policy in the NVIDIA Control Panel. Changing it wasn't tested here.

Processor threads

For a model mostly on the card, 8 threads gave 94% of the speed of 16 and kept the PC responsive. For gpt-oss:120b, which runs three-quarters from system RAM, 16 threads were clearly faster: 19.6 tokens/s against 15.3 with 8.

Quantisation and what the card accelerates

The standard 4-bit downloads (Q4_K_M) are the sweet spot: a small loss against full precision, usually 1–3% on benchmarks, at a quarter of the size. Coding suffers first from heavier compression, because one wrong token breaks a build, so don't go below 4-bit for code. Generation speed depends on how many bytes are read per token, so an 8-bit model runs at about half the speed of a 4-bit one.

The Ada card's tensor cores support FP16, BF16, FP8, INT8 and INT4. Ollama's llama.cpp engine unpacks its quantised weights to INT8, which is where the card's prompt-reading speed comes from. FP8 is used by server engines such as vLLM, not by Ollama, and FP4 needs NVIDIA's newer Blackwell cards. gpt-oss's native MXFP4 format runs fine on Ada; it just doesn't get Blackwell's extra speed.

Working from WSL

Run Ollama on Windows and let tools inside WSL reach it over localhost. With networkingMode=mirrored under [wsl2] in %UserProfile%\.wslconfig, WSL sees Windows' localhost directly, so WSL never needs access to the GPU at all.

What more money buys

The models you can run are set by how much fast memory you have, and the price tiers don't line up neatly with that. Read against the results above, going from 20 to 24 GB is worth having, because 24 GB holds gemma4:26b entirely with room for the desktop. From there up to 72 GB, more memory mostly buys speed. The next step that adds capability is holding gpt-oss:120b in fast memory, and these tests couldn't show that it does better than gemma4:26b.

Speeds for configurations not tested here are estimates from memory bandwidth and published measurements. Prices were rising through 2026 because of the memory shortage; check before buying.

How much memory gpt-oss:120b needs

It's a mixture-of-experts model, but that doesn't mean only the active experts need to be on the card. Each layer picks 4 of its 128 experts for every token, and the choice changes from token to token, so every expert has to be reachable. What the design does buy is that each token reads only about 5B of the 117B parameters, so the part left in system RAM costs much less than it would for a dense model.

So a 72 GB card is the smallest that holds it, and 80–96 GB is comfortable. Below that, speed is set mostly by how much has to come from system RAM, which is why the gains only become large in the last step.

Ryzen AI Max+: 395 or 495?

These chips share one pool of fast memory between processor and graphics, so a 128 GB machine can give 96 GB or more to models. Both run gpt-oss:120b at about 50–55 tokens/s on current llama.cpp builds (Vulkan, MXFP4); early-2025 software managed only 30–34, so check how any benchmark was run. The 495 uses the same 256-bit memory bus with faster memory, 273 GB/s against 256, so it's only about 7% faster at generating text. Its extra 64 GB matters only for models in the 200–300B class, which would run noticeably slower and weren't tested here.

The weak spot is reading prompts: about 340–700 tokens/s on gpt-oss:120b depending on the graphics driver. A coding assistant that feeds in a 20,000-token file would wait 30–60 seconds each time, much as this machine does today. When comparing benchmarks, ask for generation speed with a long context and for prompt-reading speed, not just the usual short test.

The 32 GB cards

The Radeon AI PRO R9700 and Intel Arc Pro B70 make 32 GB cheap, but they read memory at about a third of an RTX 5090's speed. The R9700 has the more mature software: AMD supports it in ROCm, including under WSL2, and Ollama and LM Studio run it on Windows. The B70 is cheaper and has ECC memory, but its software is the weak point; under WSL2 Intel's compute backend often can't see the card, so plan on native Linux. Neither runs NVIDIA-only software, which matters for image and video generation and for training.

An RTX 4090 or 3090?

The RTX 4090 is the same Ada generation as the test card, so software support is identical. It has 24 GB, about 3.6 × the memory bandwidth and 4 × the compute: roughly 150 tokens/s on gpt-oss:20b instead of 51, and long prompts read in 2–3 seconds instead of 9. Its extra 4 GB is just what gemma4:26b needs to run entirely on the card alongside the desktop, which makes it the best general-purpose upgrade these results point to. Against that, it draws 450 W and is a 3–3.5-slot card with a 16-pin power connector. A used 3090 has the same 24 GB and generates text almost as fast on these models for about half the price; it reads prompts at about half a 4090's speed.

For a small machine like the MS-02, which takes only low-profile cards, the RTX PRO 4000 Blackwell SFF is the drop-in upgrade: 24 GB, 432 GB/s (1.5 × the Ada SFF card), still 70 W, and with Blackwell's FP4 support.

Host machine and operating system

A graphics card needs a host with the slots, power and cooling for it. Two second-hand workstations compared for a one-card-now, maybe-two-later plan with a 300 W R9700:

The P520 wins for two cards; its fast, plentiful RAM also helps models that spill. Check power connectors: many current cards use a 16-pin 12V-2x6 plug, and older workstation supplies have 8-pin leads, so you may need adapters, two 8-pin leads per card. The MS-02 used here takes only low-profile cards and has a 350 W supply, so its upgrade path is a low-profile card or an external GPU over USB4.

Do you need Linux?

No, not for one card. If your coding assistant uses a cloud model, the operating system doesn't matter at all. For local models, NVIDIA's CUDA works well natively on Windows and inside WSL2, and AMD's R9700 is supported under both. Linux is worth it if you buy Intel's B70, run two or more cards together, or want server engines such as vLLM. Even then, consider a separate headless Linux box reached over the network rather than moving your main desktop. If you do, use Ubuntu 24.04 LTS: it's what AMD and Intel document and test against first.

Buying a used workstation card

eBay has many listings for cards like the RTX A6000 from sellers with a few hundred feedback and 100% positive. There's no single scam tied to these cards, but several patterns are common:

• Hijacked accounts. Old, dormant accounts with clean feedback suddenly list expensive cards below market price, then disappear. Years of silence followed by a run of GPU listings is the signature.
• Farmed feedback. Feedback earned on cheap items or as a buyer, then a switch to £3,000 cards. Filter the feedback to "as a seller" and look at what was actually sold.
• Fake tracking or an empty box, and requests to pay off eBay. The latter is always a scam.

Before buying, compare the price with eBay's sold listings, and ask for a photo of the card's serial label next to a note with your username and the date. Pay only through eBay checkout. Film the unboxing in one take, and test straight away within the return window: check that nvidia-smi reports the right card and memory, run a VRAM test that fills all of it, and give it an hour under load. Where you can, collect in person and see it running first. An established IT refurbisher costs more but usually gives a warranty, which at these prices is often worth it.

Local or cloud?

Even £10,000 of hardware doesn't buy a top cloud model; the largest open models need hundreds of gigabytes. The same money pays for the most expensive individual coding plans, about $200 a month, for over four years. Local models win on privacy, working offline and having no running costs, and they're good for routine work. Many people use both: a local model for everyday jobs and a cloud model for the hard problems.

You can try models before buying hardware. Services such as OpenRouter serve the same open models for cents per million tokens, so a week of real work costs very little, and an hour on a rented 96 GB card shows how fast a big card would feel.

Recommendations

Poor value for this work: 48 GB cards and the RTX PRO 4500, which cost much more than 32 GB cards without running better models; and the 192 GB Max+ 495 unless you specifically want 200B-class models.

Before spending anything, try gemma4:26b and gpt-oss:120b on your own work. gemma4:26b runs at full speed on a 20 GB card with a 16K context. gpt-oss:120b runs on any PC with a 16–24 GB card and 96 GB of RAM at 15–20 tokens/s: slow, but enough to judge whether faster hardware is worth it.

Caveats, and what went wrong

This is a practical test on one machine: 11 coding tasks, 6 questions and 5 code-reading questions, with three seeds only on the real-work tasks. Single runs can swing by an answer, and the same seed doesn't always reproduce the same answer, because the draft-token prediction some models use isn't fully deterministic.

Building a fair harness turned out to be half the work. Three bugs in it were found and fixed during testing, and the affected results were re-graded:

• Stale programs. When a model's code failed to compile, the harness could run the previous model's compiled test instead, which might pass. Six qwen "passes" were answers with no code at all, from models that ran out of tokens while reasoning. The old program is now deleted before each compile.
• Code cut short. One model copied the task text into a comment, including the three backticks that mark a code block, and the harness took that as the end of the code. Code blocks now only count when the backticks start a line.
• A fair interface. The allocator task shows its #defines as an interface; a model that reasonably treated them as coming from a header failed to compile. The harness now provides them.