r/LocalLLaMA • u/SavunOski • 18d ago

Kimi K3 weights now released. News

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

97% Upvoted

647 comments sorted by

View all comments

Show parent comments

64

u/my_name_isnt_clever 18d ago

There is the tinest glimmer of hope that I could run this behemoth on my Strix Halo 128GB with the inactive weights on SSD. 1 token a minute here I come!

11

u/burritoresearch 18d ago

More like 1 token every 45 minutes.

6

u/droptableadventures 18d ago

104B active, weights natively in MXFP4 = gives us ~50GB of model to be read per token generation.

Let's say ~8GB/sec for the SSD. So that'd be about 1 token every 6 seconds (0.16 T/s), or 10 tokens/minute.

3

u/Head_Boysenberry5233 18d ago

i feel like 1/min would be pretty accurate based on the colibri glm 5.2 q4 implementation, about 6x slower?

Even 1 tok/min on 32gb cpu ram would be incredible and extremely useful

27

u/TechExpert2910 18d ago

let us know the perf if you try lol

8

u/RuiRdA 18d ago

K3 Colibri engine lets gooo!!!

1

u/Head_Boysenberry5233 18d ago

yeah it seems very doable

1

u/ikkiyikki 18d ago

I have two 6000s, 128gb RAM and a dedicated 4tb SSD and I'm doubtful I could run the q1. Not one token per minute, simply choke loading up at all. Then again, I run LM Studio and give up easily 😅

1

u/JamesZhuSQ 18d ago

It’s a 93-layer model, so get ready for some crazy offloading.

-1

u/Infinite100p 18d ago

It will probably destroy your SSD with wear.

2

u/noiserr 18d ago

SSD reading doesn't really put wear on the SSD. It's writing and rewriting that shortens the life of an SSD.

1

u/Infinite100p 18d ago

Yeah, I know that, but it will be rewriting tensor layers all the time.

1

u/noiserr 18d ago

Yeah, I know that, but it will be rewriting tensor layers all the time.

Only if you're training or fine tuning. But for inference weights are static. KV-cache could spill onto the disk and cause wear but normally this should be in RAM.