I have an old Optiplex with a low end GPU (RTX 3050 6GB VRAM), thinking of using it for code completion and generating functions - kind of like the old days of Vibe coding, stitching together functions from Chat, before Agentic AI became the rage. I sort of like that way better for important projects because I read and knew the code…
Drop of research (ChatGPT) suggests Qwen/Qwen3-4B-GGUF at main as the everyday model, using ollama as the interface. [Also suggested ComfyUI with SD 1.5 for small image generation]. I’m working on setting this up now, and will report back.
Anyone with any experience in a similar deployment or have insight or suggestions - specifically how to squeeze the most juice out of the hardware with which models and parameters?
Just from my litte knowledge coming from following SLM news a little too much, I would’ve thought that either one of the Qwen 3.5 or Gemma 4 QAT models would be a better fit.
ALthough the truth is that this space is so fast moving, that one day I think I did enough research to have it all figured out, know which small model from different size tier are best for what, and then I blink and all of a sudden each model line is a few releases later, and things are different already.
Generally Gemma is best for natural conversation and creative writing, either Qwen or Gemma for coding, depending on which size models you’re looking at and who you listen to, Granite and Phi for low hallucination rates, and LFM for speed.
Then there are all the MoE models, which always punch above their weight for their size, but I never payed enough attention to them because MoE somehow performs worse then dense models if you’re running on CPU or integrated GPU.
Thanks for the insight! I’ll definitely give 3.5 and Gemma a shot. Let me see how they compare with 3-4B. Its very possible that ChatGPT is more behind the times than you are, (although Gemini also suggested 3).
Yeah, with this type of stuff you have to specifically prompt it to only look for the latest info online, specifying the dates, and even then, it will still usually be behind.
You basically have to learn about the latest models yourself, and then tell the LLM to research how they compare.
Once you give it the specific model and version they’re a lot better at pulling up accurate up-to-date info.
I wouldn’t bother with Qwen 4B, it’s going to be way too dumb for anything you might consider useful. I haven’t noticed useful intelligence in models lower than ~20B. What I’d recommend for you is running Qwen 3.5 A3B or Gemma 4 26B A4B using one of the hacky projects that streams from system RAM to VRAM. For low difficulty tasks, they might have enough intelligence to get by, and the expert-streaming (might) allow it to run on only 6GB of VRAM. Realistically, it’s probably the best you could possibly run on that hardware.
Both of these models seem pushing very far past the hardware’s capacity.
I tested the following low end models:
gemma 3 4b
phi 4 mini
qwen 3 4b
qwen 2.5 coder 4b
qwen coder outperformed the others by miles, in terms of speed, and accuracy. Going to try to level up slowly, don’t think I’m going to jump directly from 4B to 26B.
Probably not as far as you think. MoE models such as these only run 4B parameters at a time, so it’s essentially the same as any of those small models you’ve tested. But intelligence-wise they act more like their full parameter count, so you get much more bang for your buck. The only caveat is that you still need to host the full model in RAM, but for MoE, often system RAM will do. There are programs designed to allow you to do this.
I’ll give it a try, but when it bleeds over to system RAM it slows it down a lot.
Can you recommend something specific?
Agav, for what it is worth, Gemini says this:
Gemini
1. “Models under ~20B aren’t useful”: False
This was true back in the Llama 1/2 era (2023), but it is completely outdated.
Modern 3B–4B parameter models (such as Qwen 2.5 Coder 3B and Qwen 3 4B) punch vastly above their weight class. They are specifically trained on high-density synthetic reasoning, AST representations, and massive code corpora. For day-to-day coding tasks:
Syntax generation, boilerplate, and regex
Routine refactoring and unit tests
Debugging common exceptions and library usage
Fast inline tab-completion
A dedicated 3B coder handles these with high accuracy. While a 20B+ model has superior architectural planning and deep multi-hop reasoning, claiming sub-20B models are “too dumb to be useful” ignores how effective compact, code-specialized models are today.
2. “Stream 26B/35B MoE models across PCIe on 6GB VRAM”: A Terrible User Experience
Your friend is referring to Mixture of Experts (MoE) models (like Qwen 3.5 35B-A3B or Gemma 4 26B-A4B, where only ~3B to 4B parameters are active per token). They are suggesting tools like MoE-Infinity, FMoE, or specialized llama.cpp CPU-GPU offloading runtimes that keep the inactive experts in system RAM and stream the active experts into VRAM on the fly.
Here is why this fails in practice on your hardware:
The PCIe & System RAM Bottleneck: On an older desktop running DDR4 system RAM over a PCIe 3.0/4.0 x8 or x16 bus, routing expert weights across the bus for every single token causes massive bandwidth contention.
Brutal Latency: Instead of the smooth 45–60 tokens/sec you get by running a 3B–4B dense model entirely inside your GPU’s VRAM, expert-streaming on a 6GB card typically crawls at 1.5 to 5 tokens/second—with jarring stutter and severe Time-to-First-Token (TTFT) delays.
Unusable for Interactive Coding: In a code editor, waiting 15 seconds for a model to even start streaming code ruins the developer flow. Inline tab-completion becomes outright impossible.
Complexity & Stability: Running hacky experimental branches requires specialized runtimes, custom build flags, and strict manual tuning, completely abandoning the plug-and-play offline stability of Ollama.
The Verdict
Setup
Speed
Context Headroom
Coding Utility
Dense 3B–4B in VRAM (qwen2.5-coder:3b / Qwen3-4B)
40–60 tok/s
High (8k–16k without paging)
Excellent daily driver. Instant, reliable offline assistant for scripts, debugging, and functions.
MoE RAM Streaming (26B–35B MoE via system RAM offload)
1–4 tok/s
Poor (heavy RAM & cache pressure)
Novelty / Benchmarking only. Far too slow for interactive coding workflows.
Stick with qwen2.5-coder-3b or Qwen3-4B. Having an instant, responsive assistant generating code right in front of you at 50 tokens/second will always provide a better development experience than waiting on a slow, offloaded behemoth.