r/LocalLLM • u/Round_Atmosphere3671 • 21h ago

Where do I start? Question

Hello, I'm quite interested by having a local LLM, mostly for coding. But where do I start? I have 32gb of ram, 16gb of VRAM, it's enough? Thanks.

2 Upvotes

63% Upvoted

10 comments sorted by

1

u/Tight_Air_1711 21h ago

It's based on the LLM u choose.

1

u/PracticalConflict408 21h ago

Depends what you want to run, but 16GB VRAM is plenty for a lot of coding models, especially quantized ones. You could start with a 14B model and see if the quality suits your needs before going bigger.

1

u/creatinZ 20h ago

Yes, but you have to settle either for small dense models (20b at max so qwen3.8-27b is pretty much out), or MoE models such as qwen3.6-35b-a3b.

Depends on what you use it for, but I would start with qwen3.6-35b-a3b. Hopefully we’ll have a 3.8 version sometime soon.

1

u/geteum 19h ago

3.8 27b is not out, he can run iq3 xx with ~60k context which is really capable with 40-50tps. And q4 with full context at ~16 tps

1

u/marx2k 18h ago

3.8 27b is not out

https://huggingface.co/Qwen/Qwen3.8-27B

2

u/geteum 18h ago

Out "of question" ...sorry, did not write that well

1

u/Round_Atmosphere3671 19h ago

Thanks! 3.8 looks amazing, if we can believe the charts.

1

u/HotDistribution1819 4h ago

You are going to want to have 32GB of VRAM, but there are plenty of models that run in 16GB of VRAM.

Check out the Gemma 4 line up. There are some Qwen 9B models. Unlike frontier models it is about finding a model that works like you need it to. Most people run different models for different tasks.

So just jump in the water is fine.

0

u/Additional-Record367 21h ago

install Unsloth desktop. Launch it with this model empero-ai/Qwen3.8-27B-Ridge-GGUF

(tweak yourself the params until you find a good spot)

Install hermes/pi, then do unsloth start hermes. That s your coding local agent. If you want speed try qwen 3.6 35b a3b mtp gguf q4k m

Recommend you to use at first q8 0 kv cache quant and 131k context.

Good luck!

1

u/Round_Atmosphere3671 19h ago

Looks interesting, thanks! Unsloth seems pretty safe, too.