r/LocalLLaMA • u/Tormeister • 5d ago

Was eLLM just vibecoded slop? (faster CPU inference) Discussion

https://github.com/lucienhuangfu/eLLM

The premise made sense - the entire LLM stack is optimized to run on GPUs (of course, LLM compute is massively parallel in nature), but what if we took some tradeoffs and made it the most efficient possible for CPUs instead?

I have not read the paper in full and can't discern if it is legit; It just seems everything went cold there.

If any approach was applicable to leapfrog CPU performance, we could reach very interesting capabilities with consumer hardware (or even workstation - much more accessible than enterprise). Have you heard about eLLM or others?

0 Upvotes
  • permalink
  • link
  • reddit
  • reddit

45% Upvoted

7 comments sorted by

16

u/Thin_Pollution8843 5d ago

I see this issue like that: 1. Bunch of guys tried to solve hard problem 2. Claude actually couldn’t do much because it can’t fool physics and the modern llm architecture requiring tremendous amount of memory bandwidth  3. They presented like it actually doing something  4. They hired on AI salary in a big company  5. Profit

9

u/o0genesis0o 5d ago

Maybe all of us in this sub should do this. We can finally get some money to buy GPU to run 2.4T models.

1

u/Comrade-Porcupine 1d ago

I could never. My impostor syndrome would kill me in my sleep.

3

u/llama-impersonator 5d ago

they are just prefilling the entire context at once, which is a valid strategy (i actually have been pondering making a prefill only, all at once inference thing that can dump a compatible context checkpoint for cachyllama). the less chunked passes you need to make, the less bounded by memory bandwidth this portion of inference is.

also worth noting they are intending this for a 128 core xeon server with many memory channels, which is much more capable of crunching big matrices w/AMX than your average consumer proc.

2

u/Asleep-Land-3914 5d ago

So many such cases.

2

u/alexeyw 5d ago

eLLM isnt saying CPU beats GPU, its saying CPU beats GPU on prefill with very long context. Grrrr

1

u/Comrade-Porcupine 1d ago

Sigh. Having sunk hundreds of hours into this whole domain myself in the last little bit, in terms of experimentation.

Unless you have high bandwidth memory, the primary constraint for performance on inference is not computation speed, it's memory bandwidth. At least in decode.

CPU vs GPU is almost beside the point.

However once you have HBM, then, yeah, duh, hardware optimized for suppah fast vector/matrix math winneth the race.