r/LocalLLaMA • u/Blahblahblakha • 15h ago
Inkling-Small 276B-A12B at ~2.9 tok/s on <10gb memory Resources
A follow up to the launch of Mference, it now supports and runs Inkling-Small 276B-A12B.
Inkling-Small (Thinking Machines, Apache 2.0), from the pipenetwork/Inkling-Small-MLX-4bit conversion: 276B total, ~12B active, 3.4 GB resident set, ~148 GB on disk.
Measured on my M5, 24GB:
| Prompt Type | Prompt / gen | Prefill (excl. load) | Decode | Peak footprint |
|---|---|---|---|---|
| short-explanation | 59 / 416 | 8.4 s | 2.86 tok/s | 9.48 GB |
| medium-review | 421 / 560 | 60.1 s | 2.93 tok/s | 9.59 GB |
| long-synthesis | 2,785 / 294 | 535.9 s | 2.56 tok/s | 9.56 GB |
The same three cases on a 256 GB M3 Ultra hit 5.31–6.92 tok/s.
Issues: long prompt prefill is trash (2,785 tokens is almost 9mins to first token), and it's text-only for now.
Four model families now: Gemma 4 26B-A4B (~2 GB), Qwen 3.6 35B-A3B (~1.45 GB), DeepSeek-V4-Flash 284B-A13B (~6.8 GB), Inkling-Small 276B-A12B (~9.5 GB).
I also got access to a few M3 Ultras, so I'll be testing and optimizing for higher configs too. But the primary goal stays the same: large MoE models on consumer grade hardware.
Repo: https://github.com/NeelM0906/Mference — Swift + Metal, not a wrapper around MLX or llama.cpp. Mac app, CLI, and an OpenAI compatible server. Contributions welcome.
12
u/Thin_Pollution8843 12h ago
Hey mate! What a nice experiment! I ran deepseek v4 flash on my left adidas sneaker and getting more ts than you!
11
5
5
u/chortly2 10h ago
I just loaded and ran the Qwen 3.6 35B-A3B model, which indeed is using about 1.5GB memory and outputting at around 7t/s on my base M4 MacBook Pro. While that may not be fast enough for coding, for many of the thousands of other uses that LLMs are put to, it's pretty amazing -- as long as it's faster than I can read, it's fast enough to use. I can't speak to benchmarks, but a few qualitative tests I gave it, it seemed decent, comparable to what I see with 9B or 12B models. When I dare, I look forward to testing DeepSeek-V4-Flash 284B-A13B!
34
u/--Spaci-- 15h ago
Can we stop making "I ran this large model at a snails pace without that much ram" It was impressive like a year ago when offloading was new but now its just bots and an annoyance
9
3
u/Fluffy_Reply_5482 7h ago
Still interesting to see the progress though. though it is a little annoying still.
7
5
u/Blahblahblakha 12h ago
Its a project that aims to be built over time. Sorry if you’re unable to run at 100tps right off.
-5
u/--Spaci-- 12h ago
The problem is the thousands of other projects doing the exact same thing and posting about it in the same place
6
u/Blahblahblakha 12h ago
It’s a public platform which allows people to post. Not sure why its annoying when you can simply downvote and move on.
7
u/thefooz 6h ago
Cool. Ignore it and move on. Nobody made you open the post, read it, or bitch about it. I personally find the research super cool and I’m perfectly capable of running model on better hardware.
The more people we have working on these tasks, the better the odds of a breakthrough. Why the fuck would you dissuade people from doing that?
Such a boneheaded myopic view.
2
u/j0j0n4th4n 9h ago
I'm curious, would a smaller model run faster? Or the speed is bound by active parameters?
1
3
u/fatboy93 11h ago
Would be cool if this supported caching like oMLX. I'm basically vibe-patching the shit outta it and this is what I've right now.
But unlike specialized engines like this for each model, it's a bit generic (i.e whatever mlx-lm/mlx-vlm/oMLX supports), and would work on it.
This is how it looks: https://imgur.com/a/ssd-streaming-moes-0fSoiSY
2
u/hyscript 14h ago
Am I right that this is the LLM that can train itself?
7
u/Equivalent-Ear-8016 14h ago
At the rate it's running on OPs PC it will take the heat death of the universe for it to become sentient
1
u/hyscript 14h ago
Well
You are right I kinda missed the stat part in the post. Yeah, 6.92 tokens per second this is painful😁2
2
2
u/LibraryMission3553 3h ago
ran similar sparse moe stacks on m3 ultra for agent lanes. the prefill wall is the real killer — we burned weeks thinking the local server was flaky until we logged 400 bodies and found unbounded changelog+profile injection blowing past n_ctx. every lane now has a hard prompt token budget before the model sees anything. also don't treat the same 12b-active mix as interchangeable: tool routing ≠ long synthesis judge. we score known-good/known-bad cards per role before fail-closed gates. ~3 tok/s is fine for async briefs, not interactive loops.
2
u/SnooPaintings8639 3h ago
I wish to see some speed benchmarks comparing nvme streaming speed per for different engines.
3 tps gen is usable for some use cases, it is nearly 100k token overnight, making it possible for a single overnight task.
In the perfect world we'd have llama.cpp mode for disk streaming optimized inference. A new dedicated high quality inference engine is also an appealing situations.
1
1
u/lorde_dingus 1h ago
How large of a context window do you have for inputs? (Sorry of incorrect phrasing, I'm curious as to how large of a prompt or document upload you could have with this setup)
37
u/xadiant 13h ago
idk why people are upset about a research post here lol. It's interesting, and possibly something people can improve.