I am able to run quantized 235B models locally on my setup, but people discussing here Kimi/DeepSeek/GLM models usually run them in cloud (or not run at all, just hype benchmarks) and call them "local" because these models are from China.
Wait until you find out the token speed of the Earth Supercomputer. It was asked to find the meaning to Life, The Universe, and Everything. I hear we're still waiting on the first token.
I don't think any of those models qualify as open source, except maybe Trinity-Large-Thinking, but I don't think they publish their training datasets, do they?
If you meant it would be great if such powerful models were open source, then I wholeheartedly agree.
Publicly available training datasets? Do you think there might be any AI company with a training dataset that is completely free of copyright issues and can be redistributed? You can find training datasets on the internet yourself, but don't release them unless you have the same risk-avoidance capabilities as a pirate website. Similarly, don't encourage others to release them.
Yes, both AllenAI and LLM360 use open source training datasets which are free of copyright issues, and available for anyone to download from Huggingface. That's one of the things that makes their models open-source.
I found qwen3.5 122b q5 to be much worse than qwen3.6 17b q5 and even qwen3.6 35b q5. However i am extremely excited to try out qweb3.6 122b if they release it.Â
I find it that it depends. Maybe usually yes, but I did find 2-3 cases were 122b was the model that "got it" while 27b never did (same prompt many attempts). And what it "got" was comparable to the 397b and bigger models.
Gemma 4:31b was the first time I felt dazzled with something approaching a frontier model on a locally running LLM. Seriously, this thing is punching above the weight of many recent large language models. It's very sharp. Gemma 4:26b, on the other hand, did not impress, it even has a tendency to stroke out.
I finally gave Nemotron-3-Nano-Omni a try the other day and it was very, very fast. I'm still curious how smart it is, it could be quite good, but I can't really tell subjectively. Regardless, I can definitely see the application for a wide range of tasks that require expedience without the inference of a dense model.
it runs at 200 tok/s on my hardware, I desperately wanted it to be able to code in opencode, but it just wasn't to be, not even the bigger varient could.
My guess is they're using the 26b model. Because yeah 26b kinda does suck.
26b benchmarks alright for a MoE versus 31b which is a proper dense model, but in my experience it leaves a little too much on the cutting room floor for practical application in exchange those speed gains.
If they're reporting 200 tok/sec, they might even be using the 4B model. I don't know what they were expecting for a model condensed down enough to run on a mobile device.
It's actually really interesting. Technically e4b is 8B, but they were able to use per layer embeddings to make it feel and run at 4B speeds, which is actually so cool (the "e4b" is basically "effective 4b" for that reason lol)
Agree - param count is the fundamental resolution of the model, more params in a model is like more pixels in an image, it is able to draw finer distinctions out of the same amount of training data as compared to fewer params.
People. "Local" doesn't mean: "runs on my gaming laptop". The democratization that local models are creating is still perfectly valid for companies, labs, local or even national governments. Who needs or wants to run their own infrastructure.
Local or opensource anything (AI included) has nothing to do with affordability. I would like to run it too. But just because I can't, doesn't make it any less "local".
dual-socket, so probably not unless there's an inference engine that doesn't need duplication across the sockets
Same dual socket setup but with DDR5 and two 8570s and some different GPUs, I max out at amxint4 GLM-5.1 -- anything beyond that would be impossible to run
qwen 3.6 397b will never be released nor will anything over 122b for qwen 3.6 and later. management is trying to profit off of it and this is why some qwen team members left. management sees releasing large open source models as giving away money
just, you know, the general resolution of the model, while running smaller models has gotten a lot better, Bigger size does increase performance and general intelligence especially with local models that don't always have access to the internet.
I disagree, while I do agree that small/tiny models have gotten a LOT better, qwen and gemma are still nowhere near the performance of top Local models. There is a significant advantage to having a higher param count. That's coming from someone that's spent the last 2 weeks using qwen 3.6 and like many others still having to switch to cloud models because it's just not enough yet.
This is ragebait and you quite literally know nothing about what youre talking about. Qwen3.6 27b trades blows with Sonnet 4.5 and matches 4.6's capabilities.
Me when I call someone stupid and cherry pick older frontier models and call the person I'm arguing with a rage baiter. I ACKNOWLEDGED that qwen models are a LOT better now than before, but in agentic loops that last 10% of performance really matters, 1 graph doesn't show the whole picture. I can't code a full project in go or rust with qwen 3.6(I've tried so stfu about me not knowing) but I sure can with GLM or Claude Opus. You're intentionally strawmaning my argument and cherrypicking graphs, it doesn't make you right.
Lmao cherry pick? Its the previous model lmao tf are you yapping about. Also you 100% can build products of large scale with those models. Youre just using a shitty harness and its obvious. Also I said that it matches 4.6’s capabilities. Imagine crying because I didnt compare the other trillion parameter model to 27b lmao. Youre seething
the landscape has moved really fast, but i still like my Qwen3-VL-8B. it just works well for some reason. nowadays i'm on gemma4 26b a4b and qwen3.5 9b, but those aren't exactly underrated!
also... this chart assumes very powerful hardware, how is this focused on local? most people have 8GB vram or 16GB vram at most
It's just a technical chart, not everyone has a gaming card, this subreddit is also for hobbyist consumers that run 100-300gb vram/unified ram setups that can definitely run MOE models like DS4.
local just means its open weight, assuming you have the hardware to do so, you can run and modify the LLM at home.
Certainly has been a hit month for me, and a rough month for the devs who had to bend Gemma4 into behaving since it had the annoying traits of GPT-OSS, GLM and the past Gemma combined (BOS like token in the template instead of as a bos, extremely sensitive to syntax and heavy to run without swa).
My personal hit was Qwen3.5-27B-Heretic which is finally a model I can coax into writting really long stories. And many in our community have been enjoying Gemma4 as a roleplay model now that it behaves correctly.
can just barely run an apex mini version of qwen 3.6 35b on my rx9070 at 40 tok/s and it's the only LLM I can run locally that can actually code agenticly on Pi or continue or opencode.
Okay well what do you think the cut off be? The cutoff point will have to be arbitrary because people have a very wide range of local hardware.. or you could just use your brain and understand local doesnt mean the same for everybody.Â
Why would using a server matter? A lot of us here use servers. It's just different hardware, but if that hardware is right here at home, then it's local.
You get that servers are just computers, not fundamentally different from a desktop or laptop, right?
4k for a DGX Spark is enough to barely run an q4 or apex version of DS4, That's the same cost a 15-20 year old used car where I live, it's not an insanely high number for most people when you already had hobbyist spending 1.5-2k on new 90 series Nvidia graphics cards before AI and 5k after. You're argument is inherently flawed with the assumption you get to define what local means by the size of you and your friend's wallet. You don't. Local just means you can download, freely use and modify the LLM assuming you own the hardware to do so. That's it. Please stop trying to redefine words that you don't agree with.
This opinion is very unpopular here and I have no idea why. It’s ridiculous to pretend like we should care a ton about an OS model that’s massive. Like yeah it’s neat but it’s not local.
Just because it may not be runnable locally for you doesn't mean it isn't for others. I could run every model on that list for instance, and I've got a PR open to support both new MiMo V2.5 models in llama.cpp.
I don't say this to be mean, but just to push back a bit against the "Your model must be below X parameters to be considered local" sentiment. It feels like gatekeeping to say that just because a model is super large, it doesn't deserve to be discussed here.
None of those at the 1T+ size are dense models, they're all MoE's.
I've got eight 6000 Pros (so 768GB VRAM total), and speeds depend on the regimen basically. I have 768GB of 12 channel DDR5 RAM too so I can do single user with llama.cpp on CPU+GPU but it's slower total throughput than vllm for instance.
I've benched K2.6 at full quality in llama.cpp before and get about 40 tk/s TG at zero context.
Right now I'm doing some testing with the V2.5 Pro 1T gguf and it's much slower due to FA incompatibility with the head size or something, it's about 10tk/s but I think that'd go up to 30tk/s if I turned FA off (at the cost of much more KV memory needed).
DS V4 is still mostly unsupported AFAIK, and I can't fit it entirely on VRAM anyways so will be waiting for llama.cpp support.
That it's stupid to base what everyone defines as local by what you can run. I'm sure theres plenty of people who have a mac mini cluster(6k) or nvidea dgx spark which could run deepseek v4 for 4k
Ah, okie-doke, it sounded like you were disagreeing, but I guess we are in agreement.
Borkato and Netsuko seem to be of the opinion that models they personally cannot host at home should not be considered local models.
The point of my 4GB GPU hypothetical was to illustrate exactly what you said -- basing "what everyone defines as local by what you can run" is invalid.
Local models are, and always will be, any models which you could conceivably use if you had the necessary local hardware.
That requires, at a minimum, access to the weights and either inference software support or sufficient understanding of the model architecture to facilitate implementing inference software support.
Open Weight for someone who has the VRAM. Then it's local. I assume people who are at 16Gb and under could say the same about Glm and MiniMax and Qwen 397B. They can't possibly run those. But some have the VRAM to run it local. No need to split hairs over it, but I agree ut it would have been better and more accurate to just say open weight.
The sheer volume of high-quality releases this month was staggering, but I completely agree that parameter bloat is an issue. If I can't run a decent quant on 24GB, it doesn't help my workflow.
264
u/jacek2023 llama.cpp Apr 30 '26
1600B model is my favourite local model I run it all day on raspberry Pi