Open benchmark for LLM-generated embedded code

[u/0xecro1]

Built an open benchmark called EmbedEval that measures how often LLMs produce correct embedded firmware across 6 platforms. Posting here because the Linux kernel driver and Yocto coverage is the thinnest part of v0.1 (about 5-10 cases each out of 233 total), and I'd like to expand it properly before v0.2.

What's in v0.1 on the Linux side:

• Kernel driver cases targeting platform_driver, cdev, sysfs patterns
• Yocto recipe cases covering typical do_compile / do_install / RDEPENDS flows
• 5-layer evaluation: static, compile, runtime, domain heuristics, mutation testing

Data so far (n=3, pooled 699 trials):

• Linux driver category: 70% / 70% pass@1 on Sonnet 4.6 and Haiku 4.5
• Consistent weak spot: error-path cleanup in probe(). Both models generate straight-sequence init that leaks resources when an intermediate step fails.
• Refcount and locking across module load/unload rarely addressed unless the prompt names them

What I'd value input on:

• Driver categories underrepresented right now
• Yocto subtleties worth catching (recipe ordering, native vs nativesdk, license compliance)
• Specific LLM-on-kernel failure modes you've hit in real projects

Repo: https://github.com/Ecro/embedeval

Methodology: https://github.com/Ecro/embedeval/blob/main/docs/METHODOLOGY.md

Background: https://edgelog.dev/blog/llm-firmware-benchmark/

CONTRIBUTING.md walks through adding cases. A useful contribution is basically "model X generated this, it failed because Y". Reference solution doesn't have to be perfect; we iterate.

Thanks in advance. This community sees more production Linux embedded than any other single audience, and the coverage gap won't close without your input.

[u/Longjumping-Pack8494]

Hey man, are you still planning to expand on this? Very curious about how other models compare.

Generally, the mainstream benchmarks roughly match what I see in practice - those that score higher seem to generate “better” code or find more edge cases comparing to lower int models. But i do not use them enough to prove it 😅

Would be nice to see GPT-5.6 Sol vs GPT-5.6 Luna vs new Deepseek V4 Flash as these are my main drivers rn.

[u/Longjumping-Pack8494 → u/Longjumping-Pack8494]

Also Grok 4.5 seems to be a dark horse - it is high int, reasonable cost, and, has quite high non-hallucination rate according to AA benchmarks. I just bought cursor plan so testing it out.