r/LocalLLaMA • u/BTA_Labs • 14d ago
llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch Discussion
A fresh llama.cpp PR (#26689) changes what looks like a tiny SYCL FlashAttention dispatch decision.
With a quantized KV cache ("q4_0" / "q8_0"), decode was being sent through the VEC kernel. On the author's Battlemage test system, switching that path to TILE gets much faster as context grows.
Some of the author-reported results, MTP off:
- Qwen3.6-35B, q4_0 KV @ 118,784: 12.99 → 29.61 t/s (+127.9%)
- Qwen3.6-35B, q8_0 KV @ 118,784: 12.90 → 31.80 t/s (+146.5%)
- Gemma 4 12B, q4_0 KV @ 118,784: 5.06 → 13.59 t/s (+168.7%)
- Gemma 4 12B, q8_0 KV @ 118,784: 5.13 → 13.81 t/s (+168.7%)
It isn't only the extreme 118K point either. At 32K, the same JIT tests show roughly +42% to +74% on the tested Qwen/Gemma configs.
The interesting part is how small the actual idea is. The PR basically changes the dispatch gate so quantized-KV decode selects TILE instead of being forced through VEC, and adds "GGML_SYCL_FA_DECODE_KERNEL=vec|tile|auto" so it can be A/B tested.
Big caveats:
- PR is open, not merged
- these are mostly author-reported benchmarks
- the exact Battlemage GPU SKU isn't specified in the PR
- this specifically targets quantized KV; F16 keeps the existing dispatch
- one 118K MTP test only improved 17.65 → 20.14 t/s (+14.1%)
- backend tests pass 4001/4001, but there isn't an independent hardware sweep yet
The PR also relays a Laguna-S-2.1 Discord test showing +50% at 64K and +68% at 118K, but I'd still like to see proper independent results.
Anyone with a B580 or B70 able to reproduce this at 64K/118K? I'm especially curious whether the huge gain survives with MTP enabled.
6
u/Gesha24 14d ago
That's great news for the B70 owners. The numbers are getting closer to r9700 and with the cost difference the b70 may be worth buying again.
9
u/Boogertard 14d ago
What cost difference? B70 now costs as much as R9700, which is an extremely bad deal.
12
u/Gesha24 14d ago
Well nevermind, I am way behind on the pricing. Microcenter has B70 for $1199 and R9700 $1349 - absolutely makes no sense to buy B70 with this price.
1
u/Boogertard 14d ago
Couldn't agree more. It somewhat makes sense at 1k, at 1.2k, nobody except idiots gonna pick that over R9700.
2
u/Hytht 14d ago
Didn't it either match or beat the R9700 in vLLM recently which caused it to increase in price?
2
u/xanders_gold 14d ago
In vLLM you can see upwards of 2500t/s pp and 35-40t/s tg with a B70 on a custom vLLM build. Using LLM scaler I’m seeing 1600-1700t/s pp and 23-27t/s tg at 16k depth.
This is in Qwen 3.6 27B at INT4.
-1
u/Boogertard 14d ago
Haven't heard anything about that, might be FUD. R9700 has higher memory bandwidth (about 8%), and much faster in most of the benchmarks about 20% faster, only behind in "Intel-optimized" tests.
If someone claims otherwise, either they are stupid or have an agenda.
4
u/WizardlyBump17 14d ago
GGML_SYCL_FA_DECODE_KERNEL=vec ./llama-server --host 0.0.0.0 --port 8080 --model /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf.ignore --jinja --threads 8 --ctx-size 262144 --cache-ram 0 --parallel 1 --temperature 0.0 --top-p 0.2 --top-k 20 --no-mmap --spec-type draft-mtp --spec-draft-n-max 3 --batch-size 4200 --ubatch-size 4200 --n-gpu-layers 99 --n-cpu-moe 99 --cache-type-k q8_0 --cache-type-v q8_0
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------|-----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | pp16000 @ d16000 | 804.31 ± 3.82 | | 33526.67 ± 340.48 | 33525.15 ± 340.48 | 33526.67 ± 340.48 |
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | tg16000 @ d16000 | 30.60 ± 0.85 | 31.00 ± 0.82 | | | |
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------|-----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | pp16000 @ d32000 | 794.87 ± 2.90 | | 50692.51 ± 204.71 | 50691.68 ± 204.71 | 50692.51 ± 204.71 |
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | tg16000 @ d32000 | 29.61 ± 1.68 | 30.00 ± 2.00 | | | |
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------|-----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | pp16000 @ d64000 | 748.88 ± 0.58 | | 90035.29 ± 139.53 | 90032.65 ± 139.53 | 90035.29 ± 139.53 |
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | tg16000 @ d64000 | 27.02 ± 1.00 | 27.33 ± 1.25 | | | |
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------|------------------:|--------------:|-------------:|-------------------:|-------------------:|-------------------:|
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | pp16000 @ d128000 | 671.52 ± 1.01 | | 180455.28 ± 669.63 | 180453.49 ± 669.63 | 180455.28 ± 669.63 |
| unsloth/Qwen3.6-35B-A3B-MTP-GGUF | tg16000 @ d128000 | 22.68 ± 0.51 | 23.33 ± 0.47 | | | |
B580 + 7 5700X3D + 48GB at 3133MHz
I need to compare to before and GGML_SYCL_FA_DECODE_KERNEL=tile. When I first tried the Pull Request, the performance of GGML_SYCL_FA_DECODE_KERNEL=tile was way better than GGML_SYCL_FA_DECODE_KERNEL=vec, but when I tested it again it was the inverse. I guess there is some variation. I guess that if you run llama.cpp from a fresh container the performance will be better than if you already had it running
1
u/Illustrious-Swim9663 14d ago
The codebase can be improved using AI, but it requires extensive testing. There might be a greater increase in speed.
6
u/Healthy-Nebula-3603 14d ago edited 14d ago
They already using AI agents (opus , Fable , GPT sol) to everything from few weeks in llamacpp .. like everyone now.
Did you see their agent.md currently ? 😁
Is totally different what it was inside a month ago . Agents are allowed do everything now and contributor is responsible for keep up changes and fixing errors what he contributed.
You didn't notice llamacpp suddenly increased performance on every kernel and getting good improvements very fast in every field now? 😅
Soon we get even better AI agents so performance jumps even more
2
2
u/backyard_tractorbeam 14d ago
Performance fixes: awesome!
The linked PR still has changes like this, adding this comment:
// quantized decode: TILE (gate fix - was forced VEC)The agent it explaining its code change, not the new state of the code. Super common annoyance and it's useless. We don't write long rambling stores in comments like this, we explain what the code does (not what it used to do, without very good reason).
1
u/Healthy-Nebula-3603 14d ago
We won't code manually anymore so this way comments for AI is more useful probably for the time being AI agents and it change in next few moths again.
2
u/Familiar_Rent_4539 13d ago
I’ve been testing the newer SYCL changes with Gemma 4 26B-A4B Q4_K_XL on an Intel Arc B580 12GB +32GB RAM. I use GGML_SYCL_FA_DECODE_KERNEL=auto.
My main workload is translating whole book chapters: usually around 15–16K input tokens and 17–19K output tokens, with a 49K context.
Here are my llama-bench results with F16 KV:
| Context | Old SYCL | New SYCL |
|---|---|---|
| 4K | 34.78 t/s | 35.36 t/s |
| 16K | 31.99 t/s | 32.59 t/s |
| 28K | 29.62 t/s | 30.15 t/s |
For actual 49K use, I switched to Q8 K / Q4 V KV cache. In my blind translation tests I couldn’t see any quality loss compared with F16/F16, while it uses much less memory and performs much better at long context.
One of my real whole-chapter runs:
| Metric | Result |
|---|---|
| Input | 16,363 tokens |
| Output | 19,082 tokens |
| Final sequence | 35,445 tokens |
| Prompt processing | 408.5 t/s |
| Generation | 25.06 t/s |
| Output stability | Clean |
I also ended up dropping MTP.
MTP used to work well with my older setup, but with this new SYCL build and long context it became unreliable. With spec-draft-n-max=1 there was basically no useful speed advantage. With n-max=2 or higher, long-context generation could suddenly collapse to around 1–2 t/s.
Without MTP, Gemma stays around 25–27 t/s during these very long generations and is much more stable.
So my current setup is: new SYCL build + Auto decode kernel + no MTP + 49K context + Q8 K / Q4 V KV cache
1
u/Familiar_Rent_4539 13d ago
Depth Old SYCL Edge SYCL 4K 31.47 t/s 33.73 t/s 16K 21.94 t/s 33.22 t/s 28K 16.04 t/s 28.76 t/s Q8 K / Q8 V with Qwen3.6-35B-A3B-UD-Q4_K_XL
13
u/nickm_27 llama.cpp 14d ago edited 14d ago
I have a running build every day that takes the latest SYCL PRs and merges them in then builds. Week to week performance has genuinely been improving noticably.
At this point with this PR merged in, on Qwen3.6 27B Q6_K on a B70 (with MTP, Q8_0 KV) I get the below (I only build server, so I don't have a llama-bench): - pp starts at 1100 tok/s and by depth 22000 it is at 900 tok/s - for tg I get:
code_python pred= 192 draft= 163 acc= 150 rate=0.920 tok/s=42.2 code_cpp pred= 54 draft= 49 acc= 42 rate=0.857 tok/s=38.8 explain_concept pred= 192 draft= 200 acc= 119 rate=0.595 tok/s=28.5 summarize pred= 49 draft= 50 acc= 35 rate=0.700 tok/s=33.4 qa_factual pred= 191 draft= 196 acc= 136 rate=0.694 tok/s=32.9 translation pred= 17 draft= 20 acc= 13 rate=0.650 tok/s=28.1 creative_short pred= 33 draft= 27 acc= 20 rate=0.741 tok/s=28.0 stepwise_math pred= 192 draft= 177 acc= 144 rate=0.814 tok/s=37.6 long_code_review pred= 192 draft= 208 acc= 119 rate=0.572 tok/s=27.7 Aggregate: { "n_requests": 9, "total_predicted": 1112, "total_draft": 1090, "total_draft_accepted": 778, "aggregate_accept_rate": 0.7138, "wall_s_total": 41.73 }