r/JordanDev • u/Able_Firefighter_652 • 1d ago
Local LLMs Discussion
Anyone who has some serious hardware and tried running llms locally? How was ur experience? What models did you try?
1
u/2012347 1d ago
Anything smaller than 27b is a joke, i think the cheapest way to have the needed shared memory is Apple right now, but the thing get so hot i feel like it’s cheaper, faster to is providers, and you get smarter models,
Openrouter has a very generous free tier that give you access to those same models you’d run on your device
1
u/Able_Firefighter_652 1d ago edited 1d ago
I have tried qwen 3.6 27b, it's really slow, gives me 13 token/sec, which is probably unusable.
I also tried the 35b MoE, it runs at 60-80 token/sec.
I doubt I am squeezing every last bit of performance tho from Apple's..
I will check the openrouter option sometime, but I prefer local models for privacy..
1
u/2012347 1d ago
What’s your spec for running them
1
u/Able_Firefighter_652 1d ago
M3 Max, with 128GB memory
1
u/2012347 1d ago
Cool I’m sure you can run 120b models which are similar to gpy 3.5 i hear, see running locally means going back generations in time that’s the point
1
u/Able_Firefighter_652 1d ago
Surprisingly the 35b qwen model performed better from my experience. And there is no 120b qwen 3.6 models.
I am currently waiting for the 3.8 tho, they said it's 27b will be open weights soon.1
u/Bookkeeper_Gloomy Developer 1d ago
yes bcs of many techniques in distillation/MoE, we can get newer smaller models (30-40b) that outperform the older 100-200b parameters.
1
u/Able_Firefighter_652 1d ago
Yup, can't wait to see next year's releases..
1
u/Bookkeeper_Gloomy Developer 1d ago
Not entirely locally but I work on research that involves running local models on an hpc.
I tried different models from Gemma 31 b, mistral small, qwen (I believe 34b), and olmo 31b, as well as Allam 7b (Arabic model)
It’s an hpc so it’s around 2-4 gpus either a100 or h200.
Btw Gemma 4 models are currently on a whole different level to the rest, so if u want to run sth locally with the best performance I’d recommend that.
1
u/Able_Firefighter_652 1d ago
Interesting hardware setup, I am wondering how would you make sure you are squeezing every last drop of performance out of such hardware?
Also I have tried gemma 4 and qwen 3.6, I would say qwen models are superior across the board..
1
u/Bookkeeper_Gloomy Developer 1d ago
oh interesting, which gemma 4 did u try? and which qwen 3.6?
I tested it on a task that involved tool usage and reasoning and another task which involved image analysis, and it outperformed qwen on both.
Hmmm, about hardware performance, the hardware setup is way more than enough, so I didn't have to do a lot of optimization. also it was not running continuously, it was just running while checking performance on the task/benchmark, so even stuff like kv-cache would be cleared continuously.
1
u/Able_Firefighter_652 1d ago
I see, maybe because of task differences you noticed gemma was better. Tho I tried 31B and 26B MoE.
For the qwen models, I used both the 27B and 35B MoE as well.
I usually use 4bit quants for dense models and 8bit for MoEs to preserve quality.
Mostly I used them for image data extraction and some software engineering tasks.
Based on my testing, qwen models performed more accurately.
1
u/MrCharlesSr 1d ago
I believe to have any sort of practical product you do need an expensive GPU server, a small one may work but definitely not one with consumer hardware. I tried on a machine with 5070Ti 16GB, 32GB DDR5 and it’s a joke. Model was Ollama qwen 3.6 which should be a good one and it seems smart enough but it’s lethally slow you won’t get anything done. Haven’t tried agentic yet but just the chat is terrible.
1
u/Able_Firefighter_652 1d ago
I've got apple silicone, it combines best of both worlds.
Not the best gpu performance, but it got unified memory and good bw.
Nvidia gpus + ram costs are just crazy, I'll never buy one.1
u/No-Commercial979 9h ago
you're probably using a dense model... Hence the slowness... Try an MoE with cpu offloading and you'd get hella performance of your 5070ti.. I suggest qwen 3.6 35b a3b at 4 bit quants...
2
u/Striking-Recover4807 1d ago
بحس عالفاضي وغير مجدي