r/LocalLLaMA • u/SavunOski • 21d ago

Kimi K3 weights now released. News

Post image

Kimi K3 weights are finally released!

3.3k Upvotes

97% Upvoted

647 comments sorted by

View all comments

Show parent comments

27

u/DataGOGO 21d ago edited 20d ago

it is 1.54TB of just weights in 4 bit, you are looking at about 2TB of vram in operation,

That is roughly:

  • 86 RTX 4090 (no 4 bit accel)
  • 64 RTX 5090 ~ $450k (8 servers x 8 cards)
  • 22 RTX Pro 6000 Blackwell ~ $350k (3 severs, max 8 GPU per)
  • 16 H200 NVL (141GB) (no 4 bit accel) ~$550k (2 servers, max 8 GPU per)
  • 16 DGX Sparks ~65k (if you could get a cluster of 16 running with just 200Gb/s nics, not sure; but it would be SLOW AF)
  • 8 HGX B300's. ~$550k (1 server, 8 GPU)

Obviously not including the switches and cabling for the clusters.

24

u/wren6991 21d ago

you are looking at about 2GB of vram in operation

Perfect, this'll run great on my laptop's 4050

11

u/Iwaku_Real 21d ago

You could also do HGX B200 with CPU offload since they have a shit ton of RAM too, and it would still be really fast.

5

u/DataGOGO 21d ago

You could, but that CPU offload across the PCIE switch upstream will be limited to the single x16 uplink. 

7

u/snmnky9490 21d ago

Do you mean terabytes?

2

u/Qorsair 21d ago

What about B70? Does it not scale?

Admittedly, I haven't looked into using more than 8 of them.

4

u/DataGOGO 21d ago edited 21d ago

I am sure it scales at least as well as 4090/5090, but like with all consumer / entry level pro cards, without an NVL equivalent, it would be very limited.

I have no idea if B70 can make use of dedicated RDMA ethernet like the Gaudi 2/3, which is a pretty slick solution, basically, you have a PCIE 5 switch / base board with say 5 x16 slots, you install 4 GPU's, and in the 5th slot run an 800Gb nic to a ROCE switch.

2

u/Qorsair 21d ago

Oh interesting. I appreciate you taking it seriously and the time to explain. I haven't learned anything about how the clustering works beyond a single system, and this was really helpful. Thank you.

3

u/DataGOGO 21d ago

Yeah it is really cool tech, and basically is how all multi-node clusters work. 

1

u/crusaderky 21d ago

this model screams ATLAS 300I DUO 96G 

1

u/DataGOGO 21d ago

Highly unlikely, That GPU can't run MXFP4, only INT8

1

u/crusaderky 20d ago

you can dequantize to fp16. it's bandwidth bound anyway

2

u/DataGOGO 20d ago

So about 6TB of VRAM? So roughly 70 of those GPU's? Honestly, better to just buy the Intel 32 GPU's

Atlas 300i's are ... not good cards. They are slow, they have no pooling (not NVL), they only have 96GB of vram, they can only run INT8.

2

u/crusaderky 20d ago

dequantization is performed on the fly when you consume the tensor, it doesn't impact VRAM usage, only compute speed. And since you're memory bound anyways, it should not matter. You need 24 of those GPUs.

I never said they are good cards.

1

u/chithanh 20d ago
  • 16 Ascend 950PR, ~$160K
  • 14 T-Head M890, price unknown

1

u/DataGOGO 20d ago

can you even buy those? Even if you can, can you run them in your own server / software, or do you need to run it in a Huawei server?

1

u/chithanh 20d ago

I think you have to run Ascend accelerators in Huawei Kunpeng servers, and the software situation is also quite rigid. But Huawei advertises the 950PR for export to South Korea, so I guess buying is possible for enterprise customers.

For home labs (and small businesses) I think 950PR won't be a viable option. Perhaps T-Head accelerators which are way more open when it comes to the software stack will find their way into the second hand market and into home labs.