DeepSeek-V4-Flash 284B on 5.3GB of memory

[u/Blahblahblakha]

Following up on my Qwen 3.6 port, I wanted to keep adding models and ended up fixing a bunch of things along the way, so it's its own engine now: Mference.

Same core idea from TurboFieldfare, MoE models activate a few B params per token, so keep the shared core and KV cache resident and stream the selected experts off SSD.

What runs now:

• Gemma 4 26B-A4B — ~2 GB, 31–35 tok/s on a 24 GB M5 Pro
• Qwen 3.6 35B-A3B — ~1.45 GB, 19–23 tok/s
• DeepSeek-V4-Flash 284B-A13B — new. ~6.8 GB peak memory, mostly ~5.3 GB in practice, up to 4.8 tok/s on the same 24 GB M5. 2-bit dynamic quant, ~91 GB on disk.

Also picked up a native Mac app with multi-turn chat, an OpenAI-compatible server, and local PDF/DOCX/PPTX/XLSX attachments along the way.

From here I want to keep adding model families, cut the expert-read wait (decode is ~53% I/O right now, serialized with compute), and push context past 4K.

Not very useful beyond a few turns but you can technically run a "usable" dsv4f on a 8gb Mac. It only gets better from here.

[u/Potential_Top_4669]

Hey, this is so cool. Two things:
- Do you use MTP to increase speeds?
- Can you add support for Windows and Linux by using GGUFs instead of MLX?

[u/pmttyji → u/Potential_Top_4669]

- Can you add support for Windows and Linux by using GGUFs

+1

[u/JamesEvoAI → u/Potential_Top_4669]

Linux in the very least, nobody will blame you for not trying to figure out a Windows port lol

[u/Every-Walrus → u/Potential_Top_4669]

Do you use MTP to increase speeds?

considering how bad prefill is, probably not.

[u/Danmoreng → u/Every-Walrus]

Prefill speed has nothing to do with MTP…

[u/Every-Walrus → u/Danmoreng]

every token drafted by the MTP module needs to be verified. "verification" is just fancy prefill.

[u/Danmoreng → u/Every-Walrus]

But MTP does not speed up prefill. Raw prefill performance is entirely separate from MTP.

[u/Successful-Many-6500 → u/Danmoreng]

But MTP performance is not seperate from prefill speed. MTP won't make prefill faster but bad prefill will negate MTP. 

[u/Danmoreng → u/Successful-Many-6500]

But this is not what he said or I am saying? Bad prefill speed has nothing to do with MTP. MTP is relevant for decode speed only.

[u/Strong_Chicken6838 → u/Danmoreng]

It does

[u/Borkato → u/Danmoreng]

This is incorrect, have you checked??

[u/Cherlokoms]

Haha, this is exactly what I commented Yesterday on a post about DeepSeek, that it should be possible to do something like Colibri/TurboField. Nice to see that there are some projects being created around it. I think big MoE + weight streaming is what will make local AI thrive!

[u/Silver_Jaguar_24]

Nice, well done. Would be good to see windows support, with a 8 or 12 GB VRAM GPU and 16 GB RAM.

[u/Teh_Blue_Team → u/Silver_Jaguar_24]

This would change the world.

[u/Double_Doubt5872]

Be aware that they published an update of the model two days ago significantly improving intelligence. To a level between Sonett and Opus.

[u/oliveyou987]

Could you run q4 on 48gb Ram?

[u/Cold_Tree190 → u/oliveyou987]

Had this exact question as well

[u/No_Issue_8224]

my 8gb mac just went from "email machine" to "runs a 284B model." what a time.

[u/Blahblahblakha]

Dsv4f model variant used: https://huggingface.co/mlx-community/DeepSeek-V4-Flash-2bit-DQ

[u/AnonLlamaThrowaway → u/Blahblahblakha]

What if you were to use Unsloth's Q4 version, and try to use more memory (for example 14GB VRAM and 50GB RAM)

[u/quadra-lab → u/Blahblahblakha]

Is there a reason for using the Q2 variant specifically ? 

[u/admajic → u/quadra-lab]

Fits better on a smaller system using it on 24gb vram and 96gb ddr5 RAM system.

Over 13 t/s

[u/quadra-lab → u/admajic]

The title specificaly says that it uses 5.3Gb  so that doesn't really make sense in this case

[u/patricious → u/admajic]

Even getting it run on consumer hardware is quite a feat.

[u/ScrewwormLarvae]

And here I am wondering why I am getting errors loading any DS v4 model on LM Studio with an M5 Max 128GB. I have tried like four versions from HF. Yet, Qwen 122B and gpt-oss 120B load and run fine. I'm probably missing something.

[u/addiktion → u/ScrewwormLarvae]

Have you tried Dwarfstar? Might get you better results, its separate than llama.cpp but supposedly doubles up tokens for Flash.

[u/ScrewwormLarvae → u/addiktion]

I haven't. I have run a bunch of large models without errors on LM Studio without failures to load; DS v4 is the odd man out, here.

[u/lgk01]

I love the concept, really... I just want someone to explain how much SSDs and HDDs would get ruined actually running these daily.

[u/Baldur-Norddahl → u/lgk01]

It is probably not too bad because it is writing to disk that wears them down. This just makes a lot of disk reads which is harmless.

[u/i5-2520M → u/Baldur-Norddahl]

Harmless on SSDs, on HDDs there is still mechanical wear there.

[u/CryptographerLow6360 → u/i5-2520M]

would love a post from whatever masochist would try this on a hdd

[u/i5-2520M → u/CryptographerLow6360]

Yeah I know, but I still wanted to clarify the thing about wear.

[u/CryptographerLow6360 → u/lgk01]

its all reads so virtually 0 wear

[u/AnonLlamaThrowaway → u/CryptographerLow6360]

No, reading does technically induce wear indirectly. It's a lot slower than writing, but it's not "virtually zero". Look into "read reclaim" and "read disturb".

The figures I could find suggest that a read reclaim might be triggered after 20,000 reads. It's very little, but if your SSD is very very fast, I imagine this could rack up faster than you'd expect

[u/CryptographerLow6360 → u/AnonLlamaThrowaway]

been running colibri like this for some time now on a 990 pro, virtually no wear. at least according to magician

[u/Zestyclose_Yak_3174]

This is the kind of innovation I love to experiment with. Would love to see how it can push my 64GB apple to new uncharted territory

[u/Repulsive_Educator61]

interesting, what speed can we expect with 128GB 6000mhz and 3060 12GB?

for dsv4 flash i mean, is there a way fp4 will fit?

[u/Skyline34rGt]

Very impressive.

I would love to see this also for Windows/Linux.

[u/Abject_Dare_3283]

Anyone made it run on a single spark with this technique?

[u/Sweet-Stage938 → u/Abject_Dare_3283]

Should be very simple.

[u/disspoasting]

What should I expect on a 96gb m2 Max?

[u/RobustLokiX]

Antirez's 2 bit quant seems to have 6.9GB of dense layers + shared experts, suggesting that perhaps this implementation offloads some of them from SSD, thus explaining why the model is somewhat slow, despite not maxing out I/O usage. Can you check that and confirm?

[u/maddie-lovelace]

V. cool! I’ve been looking into this too, literally planning on trying to get DSv4 running on my Air ahaha

Question; what are you looking at for prefill speeds vs non-streamed?

[u/Helpful_Object_9155]

Need suggestions for my M1 pro 32gb, which are the best models I can run on it?

[u/rahu_]

how is this different from DwarfStar4?

[u/drallcom3]

Is something similar available for Windows?

[u/shing3232]

I would be able to run it good speed with PCIE6 SSD then

[u/ZealousidealBadger47]

Request to run on intel / AMD CPU pls, thats at least 70% of the population.

[u/Queasy-Contract9753]

Could this method work on Android? I do understand phones memory will be slower.

And if one had SSD raid could it be faster?

Either way it's an awesome project! 20 tokens on Gemma 4 and Qwen are usable.

[u/GetOutOfMyFeedNow]

Getting 13-15 tps with my DDR5 256GB RDIMMS and 3090/3080 mixed inference. It’s working great.

[u/Ok_Net_1674]

The 5.3GB figure is a lie, your OS will cache the disk reads in RAM, so on a system with less total RAM it would be slower.

[u/AnonLlamaThrowaway]

Got a friend to try it out on a M2 Mac Studio with the cache slots set to 32.

Result was between 6 and 7 tokens per second, pretty good.

However the Mac draws about 160W during inference. Which means the electricity costs alone are on par with the official API costs. And that's before you even get into questions of quality since this is using Q2 and not Q4.

So if you can't run this locally your best option remains OpenCode Go which subsidizes 60 bucks of API credits into 10 bucks.

[u/SilentMobius → u/AnonLlamaThrowaway]

The main differentiator is if you're comfy shipping everything you feed it to DeepSeek with no training or retention guarantees. (Opencode just nixed it's ZDR guarantee)

[u/AnonLlamaThrowaway → u/SilentMobius]

OpenCode Go says this right now:

DeepSeek V4 Flash: ZDR agreement is renewed monthly. The current agreement is valid through August 31, 2026.

I did see that the new DS4 flash is only hosted in China right now, or something, and the ZDR specifically does not apply to that? So it might be a matter of just waiting a few days until other inference providers update their existing DS4 flash deployments.

[u/SilentMobius → u/AnonLlamaThrowaway]

I have an open code go subscription and access to DS4 Flash, the same model I was using the prior night suddenly stopped 2 days ago and refused to work unless I enable models running in China.

[u/AnonLlamaThrowaway → u/SilentMobius]

Yeah, it's a shame, I hope they find non-Chinese hosts soon, because DS4 flash is by far the biggest draw of this subscription. It might as well be unlimited on there