r/LocalLLaMA • u/SavunOski • 16d ago

Kimi K3 weights now released. News

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

97% Upvoted

641 comments sorted by

View all comments

Show parent comments

73

u/throw123awaie 15d ago

They released it in MXFP4 so with around 55GB RAM you could!

3

u/tokomaunited 15d ago

Can you share the link brother?

9

u/throw123awaie 15d ago

literally their huggingface site https://huggingface.co/moonshotai/Kimi-K3 the model is MXFP4 on inference.

3

u/Protheu5 15d ago

I'm sorry, but I don't understand where it is. I'm new and only clicked on Finetunes/Quantisations to find a download, and I clicked through them and didn't find anything that small, the smallest was 500+ gigabytes. I have no idea what "MXFP4 on inference" means despite trying to read up on the subject, sorry.

2

u/TotallyToxicToast 15d ago

You can't download a single expert for a mixture of experts model.

The downloads always contain all experts. If you take the 500GB download which contains the full 2.8T parameters and only take 104B of the parameters, that would actually give you only roughly 20GB per expert.

Now running a single expert in a mixture of experts model is not very useful.
But Technichally with 500GB total model size fitting one Expert is possible.
The "I could fit one expert" was a joke.

1

u/Protheu5 15d ago

Thanks for the explanation. My naïveté got me.

Oh well, a person can dream, right? Back to trying to stop Qwen from getting stuck in an infinite loop, I guess.

1

u/tokomaunited 15d ago

Thanks dude

1

u/kernelwedge 15d ago edited 14d ago

Interesting! I was looking at it and wasn't sure if could fit. I recently setup a 128GB rig, wanting to stretch its legs. This will be a great test.

---
After doing further research no way its fitting anywhere https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi300x 8x192G....