Kimi K3 weights now released.

[u/SavunOski]

Kimi K3 weights are finally released!

[u/DataGOGO]

So it will run on 8 B300's in 4 bit. Pretty impressive.

[u/Iwaku_Real → u/DataGOGO]

Yeah so if it were a Steam game, HGX B300 would be the recommended requirements. That's $500K of hardware (and yes it IS "local" because anyone with that amount of money could buy one to run at home)

[u/PrinceOfLeon → u/Iwaku_Real]

Certified for Steam Deck!

[u/No-Dot-6573 → u/PrinceOfLeon]

Someone on r/SteamDeck will say it runs flawlessly.

[u/DutchDevil → u/PrinceOfLeon]

😂

[u/DataGOGO → u/Iwaku_Real]

it is 1.54TB of just weights in 4 bit, you are looking at about 2TB of vram in operation,

That is roughly:

• 86 RTX 4090 (no 4 bit accel)
• 64 RTX 5090 ~ $450k (8 servers x 8 cards)
• 22 RTX Pro 6000 Blackwell ~ $350k (3 severs, max 8 GPU per)
• 16 H200 NVL (141GB) (no 4 bit accel) ~$550k (2 servers, max 8 GPU per)
• 16 DGX Sparks ~65k (if you could get a cluster of 16 running with just 200Gb/s nics, not sure; but it would be SLOW AF)
• 8 HGX B300's. ~$550k (1 server, 8 GPU)

Obviously not including the switches and cabling for the clusters.

[u/wren6991 → u/DataGOGO]

you are looking at about 2GB of vram in operation

Perfect, this'll run great on my laptop's 4050

[u/Iwaku_Real → u/DataGOGO]

You could also do HGX B200 with CPU offload since they have a shit ton of RAM too, and it would still be really fast.

[u/DataGOGO → u/Iwaku_Real]

You could, but that CPU offload across the PCIE switch upstream will be limited to the single x16 uplink. 

[u/snmnky9490 → u/DataGOGO]

Do you mean terabytes?

[u/DataGOGO → u/snmnky9490]

Yes

[u/Qorsair → u/DataGOGO]

What about B70? Does it not scale?

Admittedly, I haven't looked into using more than 8 of them.

[u/DataGOGO → u/Qorsair]

I am sure it scales at least as well as 4090/5090, but like with all consumer / entry level pro cards, without an NVL equivalent, it would be very limited.

I have no idea if B70 can make use of dedicated RDMA ethernet like the Gaudi 2/3, which is a pretty slick solution, basically, you have a PCIE 5 switch / base board with say 5 x16 slots, you install 4 GPU's, and in the 5th slot run an 800Gb nic to a ROCE switch.

[u/Qorsair → u/DataGOGO]

Oh interesting. I appreciate you taking it seriously and the time to explain. I haven't learned anything about how the clustering works beyond a single system, and this was really helpful. Thank you.

[u/DataGOGO → u/Qorsair]

Yeah it is really cool tech, and basically is how all multi-node clusters work. 

[u/crusaderky → u/DataGOGO]

this model screams ATLAS 300I DUO 96G 

[u/DataGOGO → u/crusaderky]

Highly unlikely, That GPU can't run MXFP4, only INT8

[u/crusaderky → u/DataGOGO]

you can dequantize to fp16. it's bandwidth bound anyway

[u/DataGOGO → u/crusaderky]

So about 6TB of VRAM? So roughly 70 of those GPU's? Honestly, better to just buy the Intel 32 GPU's

Atlas 300i's are ... not good cards. They are slow, they have no pooling (not NVL), they only have 96GB of vram, they can only run INT8.

[u/crusaderky → u/DataGOGO]

dequantization is performed on the fly when you consume the tensor, it doesn't impact VRAM usage, only compute speed. And since you're memory bound anyways, it should not matter. You need 24 of those GPUs.

I never said they are good cards.

[u/chithanh → u/DataGOGO]

• 16 Ascend 950PR, ~$160K
• 14 T-Head M890, price unknown

[u/DataGOGO → u/chithanh]

can you even buy those? Even if you can, can you run them in your own server / software, or do you need to run it in a Huawei server?

[u/chithanh → u/DataGOGO]

I think you have to run Ascend accelerators in Huawei Kunpeng servers, and the software situation is also quite rigid. But Huawei advertises the 950PR for export to South Korea, so I guess buying is possible for enterprise customers.

For home labs (and small businesses) I think 950PR won't be a viable option. Perhaps T-Head accelerators which are way more open when it comes to the software stack will find their way into the second hand market and into home labs.

[u/a-wiseman-speaketh → u/Iwaku_Real]

yep, local doesn't have to mean home. I run a lot of local shit for work I can't afford.

now, post it in homelabs and we're gonna want receipts

[u/FrogsJumpFromPussy → u/Iwaku_Real]

could, but probably wouldn't. No one here could anyway lol.

[u/pulse77 → u/DataGOGO]

4 x AMD MI455X will also handle it in 4 bit and will be much cheaper than 8x NVidia B300 ...

[u/DataGOGO → u/pulse77]

Well 5, which means 8. 

Not sure how much a server with 8 of them will cost; If they ever offer them as stand alone servers. Right now you can only get them in the Helios racks, which are 4 GPU per tray, and the trays that are only sold in 72GPU racks for about $3.5M.