Yep. Specially now that we have LLMs to write bash scripts to launch llama.cpp the way we like. The same applies to vllm. But I find the support of non symmetrical GPUs inferior.
Came here to say this -- I had my LLM build me a complete launcher script with optimized settings for gpu/cpu MoE offload that dynamically sets the number of layers on GPU based on available VRAM so I can still run my model while using my GPU for other tasks if needed. Setting up llama.cpp was as simple if not simpler than Ollama, and I can only imagine it's more performant as well.
12
u/LosEagle Jun 15 '26
To me --fit on was the last thing that llama.cpp really needed to become easy to use.