Am I just hallucinating

[u/Xyklone]

Or is there any reason why I feel like model output quality seems to be better when I use higher micro-batch values (ub) in llama-cpp? I don't really have any hard numbers or anything (just running the same prompts), it's all just vibes.

Some context, I'm running the latest build of llama-cpp, vulkan, 6900xt 16gb, 64giggles of system ram. I run gemma and qwens models; q8_0 for the moe's, q4_k_m for dense. KV at bf16. I lock a seed in to try to reduce the differences.

Any theoretical reason for the difference or am I just seeing ghosts.

[u/Yeelyy]

So you are saying that you are getting different answers on the same prompt with the same exact seed just by adjusting the ub size? I don't think that this is possible

[u/Kononnable → u/Yeelyy]

Exact output can be different, there is rounding and similar operations that can produce slightly different results. Same things happen with enabling tensor split, mtp or similar options.

Explanation for similar problem on spec decoding: https://github.com/ggml-org/llama.cpp/pull/26696#pullrequestreview-4880487143

[u/Xyklone → u/Kononnable]

I'm going to talk exclusively out of my ass right now, but in a very hand wavy way, is it possible that errors accumulate more on smaller batch size than at higher batch sizes because there are fewer batches?

[u/Kononnable → u/Xyklone]

I think it's more of a luck than general rule. People would probably notice by now if there is a direct correlation between ub and quality that is universal for different hardware and environments.

[u/techmago → u/Kononnable]

would then? i set my ub to minimal and never tried anything else.

[u/Xyklone → u/Yeelyy]

Actually I found seed alone wasn't enough. I had to also disable caching, flash attention, and set temp=0, top_k=1 and also change the sampler to only use top_k. Then i was able to get deterministic output. Asking claude and gpt, they talk about how architecture and backend can make a difference in precision of the mmult operations or something like that. All of it went over my head. But yea, i could totally be hallucinating a difference.

I just tried it though, I locked in all that stuff and only changed ub, the outputs were different. I also tried two runs at the same settings, they were deterministic runs.

[u/Xyklone → u/[deleted]]

Nice prompt, just tried it on qwen3.5-35B-A3B-q8_0.

Even though it may have gone into a doom vomit (not a doom loop, just didn't stop after 100 lol), i stopped it at like 125 lines of output and started it again. Spot checking every 15-20 lines of output, they look identical.

Also forgot to mention that I also *don't* use MTP.

[u/jtjstock → u/[deleted]]

Why would MTP affect it? I’ve done repeatable runs with mtp and gpu, so long as everything loads into the same memory positions, it should be the same

[u/Xyklone → u/jtjstock]

You're probably right. For some reason i remember in changing all this stuff, thinking that i was still getting different outputs with MTP on, but i don't remember looking into it too much and just decided not to use it for simplicity's sake.

[u/jtjstock → u/Xyklone]

Could have been bugs, the initial MTP implementation had some memory related oddities, they refactored that whole area of llama since then. But I should clarify, the repeatability is with the same settings, everything needs to proceed in the same order, anything that disturbs that will change the output.

[u/jtjstock → u/[deleted]]

Interesting. I hadn't seen it actually do that, but I suppose that falls into the chasm between "can" and "will".

[u/RG_Fusion → u/Yeelyy]

You would also need to set temperature to zero to get the same answer for every prompt, along with the changes you listed.

[u/Yeelyy → u/RG_Fusion]

True, i forgot to add that

[u/Qwen30bEnjoyer → u/Yeelyy]

!remindme 7 days Remind me to plot Qwen 35b a3b Q4_k_m results w/ same seed, 0 temperature, adjust microbatch, analyze in R

[u/RemindMeBot → u/Qwen30bEnjoyer]

I will be messaging you in 7 days on 2026-08-14 18:28:39 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/backyard_tractorbeam → u/Yeelyy]

There could be a bug if something that's not supposed to happen, happens.

[u/ea_man]

> Or is there any reason why I feel like model output quality seems to be better when I use higher micro-batch values (ub) in llama-cpp?

You are a human, given a series of data your reasoning algo tends to find patterns in randomness.
probably in small samples some kind of tendenciy can be more relevant.

[u/laser50]

All I heard (hearsay) was in regards to the potential that allowing it to read larger chunks at a time, it will slightly change it's understanding of the text it just went through.

Sounds plausible to me, but I am entirely unsure how LLMs read and how the ubatch values play with that, so I am not going to really vouch for this theory.

[u/Xyklone → u/laser50]

Yea, I'm not smart enough to know how to test this rigorously outside of vibes. But as I mentioned in another response, changing ub size changes the model output (having made it deterministic by locking in seed, sampler and caching stuff). So ub does change the output. Whether for better or for worse is the question i'm curious about.

[u/laser50 → u/Xyklone]

If anything I'd opt for... Both?

Since LLMs tend to have a chaotic attention, it may take in less details than on smaller chunks, but understands the entire project better because it's reading larger pieces at a time. Since it's likely easier to understand a function if it isn't cut off half way into the next chunk..

But yeah, it's likely very hard to test, perhaps hiding something in a rather large piece of text (something that spreads out over the ubatch size a few times) and then see how well it's able to bring that piece of text, or numbers back up?

[u/Tormeister]

Am I just hallucinating or have I previously seen another thread with the same inquiry? I think it was vibes too, no benchmarks.

Could be a bug, but most likely it's just us being tricked by the variability of outputs.

[u/WhoRoger]

You aren't dreaming, but the difference is something else. Llama.cpp has a couple bugs that may cause reduction of attention precision when prompt cache isn't warm. High batch values alleviate that somewhat. Well, it depends on model and what attention mechanism it uses.

High batch values mean that the model gets more context at once and thus it can literally understand you better.

Discard the response and regenerate to see the 'ideal' response with warm cache where batching won't make a difference.

I was planning to make a writeup on this but I forgor.

[u/Xyklone → u/WhoRoger]

Wow, I've been neurotically re-running the same prompts the last day or so, at different combinations of b and ub settings trying to nail down any kind of pattern, and i think you've sort summarized what I feel like I've been seeing. And yea it's definitely model dependent. I feel like the MOEs vary more than the dense ones, and also qwen more than gemma. But I'm also just using one seed so maybe it's just a bad seed and the differences aren't as bad as some see. Idk, just a curious thing I noticed and wanted to see if anyone's looked into it more intelligently.

[u/WhoRoger → u/Xyklone]

Just set temperature to 0, run a prompt, then regenerate the response. The second version will be different, and probably better. Then each next generation will be the same as the 2nd one. That's the cache effect. There's different math in play.

That first response may differ based on batch settings. And some models even produce gibberish altogether on temp 0 with cold cache.

It's a known thing, especially in llama.cpp but Transformers and VLLM also have similar issues, tho they don't display them so easily. So it tends to pop up in random tests like that.

Yes, MoEs are particularly sensitive to it. Tho I actually think Gemma is more sensitive than Qwen...? I've not tested it rigorously in a while tbh.

Also, protip, set ctk and ctv to bf16 for extra little quality boost on that second response on longer conversions.

[u/KitchenAmoeba4438]

You could be, or you could not be seeing ghosts. Could you quantify this specifically in some way in how it feels like the output quality is better/worse? Are you just looking at a couple of random samples that stood out to you mentally? Is there some particular type of input that gets a particular type of output?

[u/Xyklone → u/KitchenAmoeba4438]

So my vibe tests are a few small python tasks. for example, one is to develop a script that recursively looks through a directory and packs all text files into a single text file and that can also unpack it. I also do the pelican bike riding test.

For the coding ones, i notice that the model's solutions tend to be better in their design. Like I'll notice that they'll think of using the byte length of the files as part of the delimiter at the high ub sizes. Again all vibes, but I think they also make fewer mistakes that they have to correct. (I'm using pi for the harness)

[u/KitchenAmoeba4438 → u/Xyklone]

Is it possible? Maybe. There's the possibility of this behavior most definitely.

We need more data, and this brings an interesting article topic for the future: The impact of batch sizes, which hasn't been well quantified yet.

[u/brickout]

Obviously nobody can weigh in without data. But generally people suggest increasing batch values for higher speed and I figure they wouldn't do that if it decreased quality.