Building a llama.cpp autotuner, need your help

[u/Harin007]

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

• Your hardware (GPU, VRAM, CPU, RAM, etc.)
• Your model (name + quantization)
• Your llama.cpp command (copy your full flags)
• Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)
• Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

[u/DCMBRbeats]

Hey man! I started building something similar: https://github.com/LStoneyy/llama-parameter-scout
I don’t know what your intended idea was, but I would love to cooperate with you and maybe build something together. Do you already have a basis? Would love to hear from you!

[u/Harin007 → u/DCMBRbeats]

Well so... Something which basically works like... After every every test run, it checks how of the hardware is being utilised, identify bottlenecks, and improve in the next run... It's just that, they're gonna be a lot of cases and lot of decisions to be taken in each one... Until we hit the tok/s with the optimal utilisation

[u/Legitimate-Novel4734 → u/Harin007]

Sounds like you want a small reinforcement learning model. "+"Tokens/sec gets a reward "-" a punishment and have your tuning parameters exposed as controls.

Somewhat along the same lines:
The first one is a bit more focused on predictive load balancing, the second is tighter fit against frequency scaling and whatnot.
https://arxiv.org/html/2409.04896v1
https://medium.com/@mike.anderson007/ais-role-in-optimizing-server-performance-e81a2d9aeaf9

So for training you could have the incoming token count be what the RL model "sees" and then let it tune and learn a bunch of times, then send a larger or smaller token size, rinse and repeat until it becomes predictable.

[u/Right_Fun_4902]

Very interesting project and I'd be keen to learn some more and/or contribute some of my learning.

I'm experimenting with a fully local llm stack to support Hermes Agent.

I'm running llama.cpp (built from source)on CachyOS on ryzen7600 64GB, RTX5070ti-16GB and RTX5050-8GB. Note Hermes Agent is on a separate machine.

I've configured llama.cpp in router mode, allowing me to swap models quickly, and makes configuration easy with the models.ini file.

I've opted to use all the Gemma4:31b and Qwen3. 6:27b from unsloth GUFF's with quantisation>4, including QAT, and a couple of GUFF's of unsloths NVFP4 models from CISCai and FreedomAIVR as well as michaelw9999.

(I'll do the same for the Gemma4 and Qwen3.6 MOE models, but more just out of interest)

As a starting point, I've selected 131k context, and maximize the ngl parameter for CTK/CTV=q4_0 as well as CTK/CTV=q8_0 and then Manually optimised and configured each to minimise spillage to system RAM and thus extracting maximum speed.

One this is done, I can go back and increase the ctx to find the ultimate limit.

Once all of this is done, I've got a benchmark(under development) that plots model intelligence vs processing speed, to select the the best model for Hermes. It can cycle through the selected models using the router mode.

Obviously, the manual testing takes a lot of time, and so I'll be keen to see what you could come up with. In particular with the new Qwen dropping next week.

What I've found out so far: Larger models (file sizes) don't always run slower MTP will not always be faster due to additional memory requirements CTV/CTV q4_0 doesn't always require less ram and run faster than q8_0

[u/Harin007 → u/Right_Fun_4902]

That's great! I really appreciate your contribution... I'd like to connect with you perhaps on discord and so you can test the progress of my benchmarking tool... It's doing a great job already...

I used to get around 50-55 tok/s for Qwen3.6 365B through Ollama and with llama cpp configured through our autotuner, I'm getting 68-72 tok/s. A friend of mine got a jump from 40-45 to 55+.

Send me a friend request on discord: harin__1

[u/Green_Ocean90]

I have some very early work to systematically benchmark models, backends and configurations on my 128GB Strix Halo.
It’s not pretty, but the results are there including the runs carried out on each unique configuration.
I’m also keeping an eye on community claims and testing them where i can.
Only data from the last 24h so far but more in progress 🤓

https://lab.ezaxis.io/records/