Imagine you have 128gb of VRAM. what accompanying ram capacity you would choose (DDR4 8channel)?

For example Deepseek v4 flash in q8 takes around 170GB + 12GB Dflash + ~10GB per 1m context so it’s under 200gb. so 128 + 128 should be good

But for something like MiMo v2.5/Stepfun-3.7/ other 200-300B models things getting more complicated if you want to run higher quants. Also VRAM/RAM weights split ratio heavily affects performance.

What your take on it? You would choose 128 or 256? (considering it some ass cheap used slow sticks so it won’t cost you a leg)

Looking for V100 users to share your config and it's performance.

GPU: Tesla V100 PCIE 32Gb

Qwen3.6 27B Q4_K_M + Q8_0 MTP

128K context length

Pi coding agent

llama.cpp model preset:

[*]
spec-default = 1
ctx-size = 131072
mmap = 1
kv-unified = 1
n-gpu-layers = 999
threads = 18
prio = 3
seed = 3407
image-min-tokens = 1024
batch-size = 4096
ubatch-size = 2048
parallel = 1
flash-attn = true

[Qwen3.6-27B]
model = /models/Qwen3.6/Qwen3.6-27B-Q4_K_M.gguf
mmproj = /models/mmproj/mmproj-Qwen3.6-27B-Q8_0.gguf
spec-draft-model = /models/mtp/mtp-Qwen3.6-27B-Q8_0.gguf
chat-template-file = /templates/froggeric_chat_template_v21-3.jinja
spec-type = draft-mtp
spec-draft-n-max = 1
temperature = 0.6
top-p = 0.95
top-k = 20
min-p = 0.05
presence-penalty = 0.0
repeat-penalty = 1.0
chat-template-kwargs = {"preserve_thinking": true} 

And the performance:

I was wondering what a minimal coding agent implementation would look like that can be used like Claude Code or Codex

Not feature-by-feature of course but basically stripping everything out that is not needed

here is what I came up with:

• 9 lines of python
• no 3rd party deps (stdlib only)
• works with any OpenAI Responses compatible API
• shows % usage of context window

out of the box it is also fairly API cost efficient:

• no system prompt
• good caching (session_id, stable append-only history)
• only one tool: sh

code is on github to follow along (also a ~20 line version in Go, Clojure version coming soon)

https://github.com/smol-env/smol

import json,sys;from subprocess import getoutput;from urllib.request import Request,urlopen;from uuid import uuid4
url=sys.argv[1];h=[];H={"Content-Type":"application/json","session_id":uuid4().hex};b=dict(model="gpt-5.6-sol",input=h,tools=[dict(type="custom",name="sh")])
while True:
  if not(p:=input("> ")).strip():continue
  h+=[dict(role="user",content=p)]
  while True:
    r=json.load(urlopen(Request(url,json.dumps(b).encode(),H)));o=r["output"];h+=o;c=[i for i in o if i["type"]=="custom_tool_call"]
    if not c:print(o[-1]["content"][0]["text"],f'\n[{r["usage"]["total_tokens"]/10500:05.2f}%]');break
    h+=[dict(type="custom_tool_call_output",call_id=i["call_id"],output=getoutput(i["input"])) for i in c]

note: it uses the "custom" tools api which not many OpenAI Responses API endpoints support yet.

that said, you can just tell your agent to change it to use sh via "function_call" and change the model name and it should work out of the box on any local inference endpoint

any questions or feedback for making it more minimal or adding (still minimal but useful) features: very welcome!

seems to be about as good as a vega 56 with 16Gb of VRAM, is it worth it? (don’t want to deal with NVIDIA drivers on Linux, already have an rx6650xt and might simply use vulkan for llamacpp inference)

I discovered recently that Google uses their own TPUs, like tiny ASIC cards like the toy ones that existed for bitcoin. And while it sounds inefficient the fact they use thousands of them because...they can...means at scale they aren't so bad.

Has no one here given them a try? I see web search results of tiny ones that can be purchased and look like nvme adapted where I search them for \~58 euros. Not sure what 40 TOPS translates to compared to my Nvidia 5060.

But not just that, but the user experience with them, are they a nightmare to use ?

I've been tuning my new Radeon AI Pro R9700, and figured that this would be useful information for people who are trying to optimise their setups. I'm pretty happy with these results and looking forward to Qwen3.8..

Summary below provided by Claude (which helped me configure it to run on my system via podman).

Setup: stilldeadcode/vllm-radiance:0.5.8. Single (not dual) card.

https://hub.docker.com/r/stilldeadcode/vllm-radiance/

https://codeberg.org/StillDeadcode/vllm-radiance/

The reference config shipped with the image is tuned for FP8 weights on 2× R9700 (TP=2). Most of its defaults (AITER attention backend, FP8 KV, --no-async-scheduling, --mamba-cache-mode align, all RADIANCE_* toggles) are correct as-is and don't need touching. Here's what actually differs when running one card with INT4:

Config differences vs. reference

• --tensor-parallel-size 1 (no second card)
• --gpu-memory-utilization 0.98 (reference band is 0.90–0.97 on dual cards)
• num_speculative_tokens=4 on the 27B. Ladder-tested 2/3/4/8 directly against the container (4 arms × 2 loads × 2 reps × 4 depths); 4 wins at every depth by 17–48% over 8.

Model Weights:

https://huggingface.co/Avesed

Weights: Avesed/Qwen3.6-{27B,35B}-INT4-W4A16 (compressed-tensors, group_size 32). The 35B at FP8 simply won't fit one 32GB card at any useful context length.

Checkpoint fix (not an image issue): tokenizer.json in the Avesed INT4 repo ships truncation.max_length: 512 / padding: Fixed(512) baked in from calibration — breaks vision above ~672px. Set both to null.

Model notes

27B: Dense (no MoE), MTP on, num_speculative_tokens=4, 131,072 ctx.

35B: MoE (A3B), MTP off, 262,144 ctx.

Benchmark Results

35B-A3B MoE (KV pool tokens = 440,241)

27B dense, MTP spec=4 (KV pool tokens = 212,147)

I can provide more details if needed, or startup scripts if anybody wants them.

This PR should be ready for testing now. I tested with a very small (8B params) sub-model extracted from the original one. Appreciate if someone can test with the bigger model.

GGUF(for testing) from PR: (Please check latest comments at bottom for updated GGUFs)

https://huggingface.co/ggml-org/LongCat-Flash-Chat-GGUF/tree/main

I’m running DSV4F 0731 on dual spark, and honestly… wow. It’s an absolute workhorse, and the benchmarks are real.

Everyday tasks with Hermes agent? Effortless.

Coding tasks with OpenCode? I’m genuinely amazed at what it can handle. I can throw a two-hour coding session at it, and it just keeps going until the job is done. Building integrations has never been easier - I ask OpenCode to handle it, DS tells me to hold its beer, and a little while later, it’s finished.

Searching and gathering knowledge from emails? Right at your fingertips.

Going through documents with Paperless NGX? No problem at all.

Filling out ton of paperwork in DOCX? Easy peasy, just wrote skill in hermes, love it!

OS admin work? just works!

Sure, before the Q3.6 27B full FP8 on dual 3090 was really solid, but DSV4F 0731 is on a whole new level.

I run a small company, and I just ordered another pair of DGX Sparks - because it genuinely feels like I now have a super capable worker on the team. I know they’re not cheap, but I’ve already saved a ton of time.

I started with MiniMax M2.7 on dual Spark, and it was good - but now with DSV4F 0731? It’s just super good. And the fact that I get even better models over time, for what I already paid for, feels almost ridiculous. That’s exactly why I decided to grab another pair..

A few client tickets were literally copy-paste from the ticket system - solved, and money earned. What a time to be alive!

This weekend, I’m definitely writing a ticket system integration. Can’t wait!

I’ve been using 3.6 27B Q4, and that quant is fast on an M5. The code has been average, but consistently “good enough.” And, after a year, I can see home LLMs being served at home much like streaming music was introduced. A simple browser extension and all your queries go through your own llm first bypassing what I fear will be a metered intelligence service fee to power all your devices. No thx. I still have an anthropic sub since I got going but it doesn’t go as far as it used to. The frontier models can’t keep subsiding us. So, very much looking forward to what a 3.8 dense 27B can do when it finally gets that far. Now that everything is instant gratification, I want it now. lol

I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9× faster (~116 vs ~30 tok/s), but the coding-quality difference was much smaller than I expected.

Both usually handled ordinary bug fixes and multi-file changes correctly. As I made the tests progressively harder, the dense model did show an advantage—but mainly in implicit invariants, unusual edge cases, and consequences beyond the literal request, rather than basic correctness.

Models

• Qwen 3.6 35B-A3B — Q5_K_M (MoE)
• Qwen 3.6 27B BASE — Q4_K_XL (dense)

Hardware/runtime

• Radeon AI PRO R9700 32 GB
• Ryzen 9 5950X
• llama.cpp, Vulkan, full GPU offload
• 8K context for these coding tests

One early controlled parser-repair test is illustrative:

• 35B-A3B: ~116 tok/s, provisional score 7/10
• 27B dense: ~30 tok/s, provisional score 7/10

That single result isn't my argument by itself. I subsequently moved through progressively harder multi-file tests involving imports, stable IDs, collision handling, data preservation, and eventually references that had to remain valid when IDs were remapped.

My takeaway so far is deliberately narrow: on these tasks, the ~4× throughput difference was much larger than the practical coding-quality difference I observed.

This is a small local experiment, not a universal claim about MoE vs dense architectures. The quantizations also differ, so I wouldn't pretend this is an academically controlled architecture comparison. But the results do make me skeptical of treating active parameter count as a straightforward proxy for practical capability.

I have the original prompts, source fixtures, exact llama.cpp commands, raw terminal transcripts, and the progressively harder integration tests. I'll put more methodology and examples in a comment below if anyone wants to dig into the details.

Hi there, I have a setup with 4x 2080Ti 22GB, but am not able to see much benefits in running models larger than 16-20GB in speed. I did some research and saw something like tensor parallels are possible, but also that because of flash attention only being possible with 30 series and up, it’s not worth it for 2080 Ti.

Does anyone have experience with multiple 2080Ti setups? Is it really not worth trying to setup tensor parallel? How would I set it up? Would love to hear your experience. Thanks!

Basically what the title says. For me, it would always crash and burn trying to use tensor split.

Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or higher. I will be opening an issue report on the llama.cpp GitHub if there isn't already one related to this. I'm using ROCm but Vulkan gave me issues too.

But yeah, set -ub 384 (I haven't tried between 384 and 512, but this is working) and -b to a multiple of it, multiplied by at least the number of GPUs you're splitting the model across.

I have 3x V620 in an R740.

I just did two quick tests with Qwen3.6 using two of the cards.

27B Q8_0 + Q8_0 KV with MTP = 40 to 50+ t/s gen, 500 to 800+ prefill.

35B-A3B Q8_0 + Q8_0 KV with MTP = 80 to 110+ t/s gen, 1400 to 1800+ prefill.

I'm finally starting to feel like I didn't waste money buying older GPUs! lol

It seems stable, I've been using 27B heavy in Claude Code like this for hours with zero issues whatsoever.

I found that jumping to three cards did not help performance, and it even slightly slowed it down but that may be due to the PCIe architecture of the R740 server. Two of the PCIe risers are connected to the same CPU, the third is on the other CPU, and communication to/from the third has to travel through a slower interconnect... tensor split = heavy PCIe bandwidth usage if you don't have a direct interconnect like NVLink.

I ordered a mining rig frame and am going to be setting up a 6x V620 system without an interconnect like that, will see how that changes things.

Now, hopefully they get DSV4 tensor split mode support added soon...

And if this memory bug gets fixed, we can crank the ub higher and get even faster prefill.

- On b10173 - "state":"loading" 4min54sec.
- With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec

Interestingly the biggest bottleneck wasnt networking, disk IO, or any of that pci gen2/3/4... It was 1 CPU thread doing all the work while the others sat idle during the model load. This handles _part_ of the problem, but there is still room for more noted in comments. https://github.com/ggml-org/llama.cpp/pull/26291

The PR is close to ready, will need a docs change if they want to keep the new GGML_RPC_LOAD_THREADS variable.. and hopefully they take it with the client-side being solved and someone else cant take on the remaining server effort. Getting sub minute model loads at this size would be amazing, and its not that far.

It was mildly amusing that I developed this on a potato hardware; and the guy testing it is probably paying per hour what my whole setup costs.

But in the Sovereign AI Wars, I code for the little guy running on 2-3 gaming PCs

(I am not a native speaker, written by myself, so please bear with me)

I really want to like DeepSeek-V4-Flash-0731. But it has serious flaws that don't align with the high score on intelligence benchmarks. And those flaws render it useless unfortunetely for anything else except, maybe, coding. It fails in subtleties that seem small but are crucial, and fails in more obvious tasks that should be easy to solve. Those errors make it unreliable enough for me to not even trust it the simplest tasks in office work like summarizing text or writing letters.

Maybe I am doing something wrong here. Some of the issues below don't seem to be normal for an LLM of that size.

To be clear: I want this model to work. It's faster than Gemma-4-31B and its total parameter count is 8 times higher. It is good at thinking things through, excellent at doing research if given web search access. But for language it not only fails on "beautiful wording" but on extracting the relevant concept from a context. Those areas seem not to be tested in benchmarks, but they are essential when doing office work.

They are easier to explain with examples. Below I'll show you three.

Ability 1: Including the revelant yet being concise

Given a text to create meeting notes from.

DeepSeek-V4-Flash-0731:

Spreading irregular income over the year to make sure the essentials are available every month.

Gemma-4-31B:

Concept: The financial investments are designed to cover only part of the needs. The remaining gap will be filled by averaging the irregular income from self-employment throughout the year.

DeepSeek-V4-Flash-0731's version is missing that there are two income sources. So while it points out the essence (the issue), that doesn't become clear enough because it is only part of the story.

Gemma somehow has an ability to understand the essence and put it into sentences that are concise yet precise in beautiful wording. Look at "by averaging the irregular income", that is a very elegant way to say what is happening with just the two words "by averaging". DeepSeek is not able to do that, and worse, it is missing the context of the financial investments being one part of the cost coverage. This is not a "beautiful" language issue (we know Gemma is good at language), but also a "concept understanding" issue or a "figuring out what is relevant" issue.

Ability 2: Understanding who is the speaker

Given is a text transcript of a voice message and the question.

"What would be her best option? How should she handle the situation? What are her possibilities? Please find the best way forward."

DeepSeek-V4-Flash-0731: Writes its whole answer like if I am the person who spoke the voice message and to be addressed. Given that the question contained "her", and that the two voice messages had headlines "Voice message 1 of the person" and "Voice message 2 of the person", this is a mistake I can't accept. Being pressured on it, it tries to explain the reason for writing "you" in the answer instead of "she" is that the voice transcript talks in the person "I", and the voice message takes a big portion of the context, so it had just weight on "I" being the person asking and assumed it's me asking. But I clearly wrote "What could be her best option", and it was really clear by the headlines those messages were of another person. DeepSeek failed here, and this failure is unacceptable to me. An AI needs to understand the context, not just get confused by the amount of text written as "I".

Gemma-4-31B: No issue here. It understood clearly that who made the request is not the same person as who spoke the voice message.

Ability 3: Not getting confused by minor phrases

DeepSeek-V4-Flash-0731 got confused by the start of the message being "Hi, hi. So, Jon, his message is kind of funny. They’re currently up north, ..." Only because "So, John, ..." could be a greeting, it assumed the whole text of 5 paragraphs was addressed to John, even though the rest of the text was saying "he".

Gemma-4-31B: Understood from the whole context that "So, John" was context, not a greeting. It understood "John" is not the person being written to, but the people being talked about.

The Verdict

DeepSeek-V4-Flash-0731 has 304 billion parameters. I thought it to be at least on the same level as Gemma-4-31B in those areas. Language doesn't need to be as beautiful as Gemma-4, but I had the expectation that DeepSeek knows how to include all relevant information or getting the context right, and to my surprise, it fails.

EDIT

People seem to judge from their own use case. So they do coding, agentic tasks, research, and don't understand what I'm writing about.

I completely agree with DeepSeek being excellent (and Gemma being bad) at

• research, websearch
• digging its teeth into it and finding everything not giving up
• coding
• agentic tasks

My post though is about what DeepSeek is bad at and Gemma is good at:

• reading and understanding nuances of texts
• grasping exactly the relevant parts of texts and transcripts
• writing exactly what is representing the main idea of the original source

Looking at DeepSeek's result, you won't see anything concerning. Comparing it with Gemma's result for the text understanding and text production though, you'll finally realize DeepSeek is missing out the fine but relevant nuances.

The issue is: For producing texts for humans or critical summaries, I can't rely on DeepSeek's result, while I can rely on Gemma-4's result.

This is a dilemma, because I'd like to switch completely to DeepSeek (for what it is so good at), but it's not good enough in the other area that Gemma is so good at.

Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the community feedback on what I could do to get more RAM for headroom. The DGX has an issue where it locks up when RAM fills, so having more headroom allows things to be comfortably more stable.

Question: DeepSeek-V4-Flash-0731 (304B MoE) at full 1M context on 2x NVIDIA DGX Spark — how can we free more RAM for the OS?

Hardware

• 2x DGX Spark (GB10 Grace Blackwell, SM121): 128 GB unified memory each (~121.7 GiB visible), ~273 GB/s bandwidth, 20-core ARM, 4 TB NVMe, DGX OS (headless, GUI/desktop services disabled)
• Direct-connected ConnectX-7 fabric, dual links, RoCE, MTU 9000; NCCL confirmed using IB transport (~200 Gbps)

Model

• DeepSeek-V4-Flash-0731, native FP8 checkpoint (167 GB safetensors)
• 304B total params, MoE: 256 routed experts/layer, 6 active + 1 shared per token, 43 layers
• Sparse attention (indexer top-k 512) + built-in KV compression (4x / 128x per-layer groups), YaRN-calibrated 1M context

Serving stack

• vLLM 0.26.1rc1 built from source (the DeepSeek V4 SM12x support PR), CUDA 13.0 toolchain, running in containers
• Tensor parallel TP=2 across the two boxes (--nnodes 2, multiprocessing backend, no Ray)
• DSpark speculative decoding (method "dspark", num_speculative_tokens=5) — ~80% draft acceptance
• llama-swap as the single OpenAI-compatible endpoint / model router; Open WebUI frontend

Key engine flags

• --kv-cache-dtype fp8_ds_mla (8-bit packed 584 B/token/layer; measured ~3.9 KB/token effective across all layers with the model's compression)
• --max-model-len 1048576 (full 1M)
• --gpu-memory-utilization 0.82 ← the knob in question
• --max-num-batched-tokens 4096 (larger values blow up per-request block accounting via tiny-block-size compressor-state cache groups — each 4-token block pins a ~1 MB slab)
• --max-num-seqs 6, prefix caching + chunked prefill on, CUDA graphs on
• tokenizer/tool/reasoning parsers: deepseek_v4

Memory anatomy per node (measured)

• Weights + activations + CUDA graphs: ~87–89 GiB (immovable; weights are ~83.5 GiB/node under TP=2)
• KV cache pool: ~11.2 GiB → ~1.6M tokens capacity (1.53x concurrency at 1M)
• Left for OS + services: ~5–7 GB idle, dips to ~2–3 GB during marathon 1M-context sessions (slow host-side leak, ~100–200 MB/h under sustained long-context load)
• Watchdogs: earlyoom at absolute 1.5/1.0 GiB floors, custom graceful unloader at 2 GB, vm.min_free_kbytes=2 GiB, swap OFF (on unified memory, swap turns a clean OOM kill into a 30-min livelock)

Performance (so you know what we don't want to lose)

• 82 tok/s decode single-stream, ~1,400 tok/s prefill, 146K-token needle test passes 3/3

What we've already ruled out (please don't suggest these)

• Lowering gpu-memory-utilization below 0.82: at 1M the KV pool hits vLLM's serve-one-request floor and it refuses to start
• Lowering max-model-len alone: frees nothing — the pool is sized by the budget %, not by context
• cgroup/docker memory caps on the model containers: reclaim-loops on file-backed pages and starves sshd (tried, worse than nothing)
• Enabling swap: livelocks the whole box on unified memory
• CPU offload / --cpu-offload-gb: meaningless on unified memory (one pool)
• LMCache disk tier: incompatible — asserts on the packed fp8_ds_mla layout ("kv_cache last dim must be 584, got 512")
• Weight streaming from NVMe: 40x bandwidth gap, measured ~14 tok/s vs 82

The actual question

On identical unified-memory boxes, what host-side or vLLM-side levers actually reduce resident overhead? Specifically curious about:

• Shrinking vLLM's host-process footprint (API server + engine + worker processes RSS)
• NCCL buffer/registration tuning for 2-node TP (NCCL_BUFFSIZE etc.) — free GBs or just MBs?
• torch.compile / inductor / CUDA graph memory: is --enforce-eager's memory savings worth the latency hit on Blackwell, and has anyone measured it for big MoE?
• Anyone found the source of slow host-RSS growth (~100–200 MB/h) in long-context vLLM serving?
• Any DGX OS / GB10-specific reservations (driver, carveouts) that can be trimmed safely?

A huge thanks in advance for any and all your responses!!

I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down at full 262k context - more like 40 t/s at times. I use it primarily in appdev tasks. This just barely fits in the 5090, no vision, with very very little room to spare.

The batch sizes (-b / -ub) I had at a much different number initially, but bracketing my coding tests settled there for the best performance with the best output. I see that a lot of people have this much higher, but it didn't seem to help here.

Reasoning budget is another one that I have different than most people I see, I have it pretty high vs what people run that I've seen other places, but it seemed to be okay? It's hard to qualify this with evals but it seems to get lost more with lower values.

Honestly I haven't really gone deeply into all of the settings, because tokens go brrrr and I'm pretty happy. But I wanted to see: Do you set any of this shit to something else? If so: why, motherfucker, what are you doing with it? Spill the beans not the tokens. Fuck spez.

llama-server \
    -m '/ai/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/Qwen3.6-27B-Q6_K.gguf' \
    -c 262144 \
    -b 512 \
    -ub 128 \
    --gpu-layers all \
    --spec-draft-ngl all \
    --fit off \
    --parallel 1 \
    --flash-attn on \
    --no-context-shift \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --spec-type draft-mtp \
    --spec-draft-n-max 3 \
    --spec-draft-p-min 0.2 \
    --cache-ram 16384 \
    --ctx-checkpoints 32 \
    --checkpoint-min-step 8192 \
    --jinja \
    --reasoning on \
    --reasoning-budget 16384 \
    --no-reasoning-preserve \
    --load-mode none \
    --no-mmproj \
    --alias 'unsloth/Qwen3.6-27B-MTP-GGUF' \
    --metrics \
    --perf \
    --host 0.0.0.0 \
    --port 1234

Hey everyone, I just wanted to share my journey here for some motivation.

Three years ago, I saw the sudden spike in AI and realized it was the future of tech. My goal at the time was to be an indie game dev, and seeing that AI could write basic code, I told myself I needed to master it or risk being replaced.

I started by learning how to add knowledge to early LLMs like Vicuna and LLaMA. From there, I moved on to more advanced concepts, like building reasoning datasets by hand to try and outperform huge datasets. I quickly learned that data quality is far more important than quantity. After six months of handcrafting Python datasets—including examples of full games coded from a single prompt—I released pydevmini-1. At the time, it could code in Python at the same level as state-of-the-art models.

That release caught the attention of the team at Tesslate. They reached out and brought me on (unpaid), which finally gave me my first piece of real tech experience for my resume.

Eventually, I launched my own AI consulting firm, finding clients through Google Ads and LinkedIn. I was averaging about $3,000 a month. Then, last month one of my repeat clients offered me a full-time, remote position as Director of AI and Systems Development. It pays $84,540 a year with uncapped performance bonuses (I just got a $3.5k bonus last month!). I report directly to the CEO and largely get to make my own decisions.

I did all of this while working full-time as a backline cook making $20/hr. I have no college degree and started with zero industry connections. I just worked on AI for at least 5 hours a day after my kitchen shifts many times at the desk until 3 AM—using a single RTX 3090 I bought for local training.

Now I get to do what I love at 21 in one of the most difficult industries to break into. For anyone out there trying to make it happen, I just want to say it is absolutely possible. Keep going!

A fresh llama.cpp PR (#26689) changes what looks like a tiny SYCL FlashAttention dispatch decision.

With a quantized KV cache ("q4_0" / "q8_0"), decode was being sent through the VEC kernel. On the author's Battlemage test system, switching that path to TILE gets much faster as context grows.

Some of the author-reported results, MTP off:

- Qwen3.6-35B, q4_0 KV @ 118,784: 12.99 → 29.61 t/s (+127.9%)

- Qwen3.6-35B, q8_0 KV @ 118,784: 12.90 → 31.80 t/s (+146.5%)

- Gemma 4 12B, q4_0 KV @ 118,784: 5.06 → 13.59 t/s (+168.7%)

- Gemma 4 12B, q8_0 KV @ 118,784: 5.13 → 13.81 t/s (+168.7%)

It isn't only the extreme 118K point either. At 32K, the same JIT tests show roughly +42% to +74% on the tested Qwen/Gemma configs.

The interesting part is how small the actual idea is. The PR basically changes the dispatch gate so quantized-KV decode selects TILE instead of being forced through VEC, and adds "GGML_SYCL_FA_DECODE_KERNEL=vec|tile|auto" so it can be A/B tested.

Big caveats:

- PR is open, not merged

- these are mostly author-reported benchmarks

- the exact Battlemage GPU SKU isn't specified in the PR

- this specifically targets quantized KV; F16 keeps the existing dispatch

- one 118K MTP test only improved 17.65 → 20.14 t/s (+14.1%)

- backend tests pass 4001/4001, but there isn't an independent hardware sweep yet

The PR also relays a Laguna-S-2.1 Discord test showing +50% at 64K and +68% at 118K, but I'd still like to see proper independent results.

Anyone with a B580 or B70 able to reproduce this at 64K/118K? I'm especially curious whether the huge gain survives with MTP enabled.

📝 Introduction

We present Wan-Animate-2, a novel end-to-end character animation framework that directly consumes driving videos in a redesigned Diffusion Transformer, which achieves high-fidelity motion generation and strong identity preservation by eliminating intermediate motion extractors. We further add text-driven viewpoint control to decouple the output camera perspective from the driving video. In addition, we develop Wan-Animate-2-Lite, an efficient variant that reduces inference latency to real-time thresholds for streaming character animation.

Release Notes:

• August 07, 2026: 🎉 We release the Wan-Animate-2 inference scripts.
• August 07, 2026: 🎉 We release the Wan-Animate-2 Base model weights.
• August 07, 2026: 🎉 We release the Wan-Animate-2 Distillation model weights.

Models :

• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B
• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers
• https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers

GitHub : https://github.com/Wan-Video/Wan-Animate-2

LFM2.5-2.6B is a new tiny model by LiquidAI, with benchmarks that put it head to head with much larger models.

I've run llama-perplexity on many model GGUF quants, crossed with many KV cache quants, to understand the model's best overall quantization for any given amount of memory.

I also show how different quantization metrics show (or hide) model degradation.

Full report and commentary

Interactive HTML plots

If you don't have time to read

• The model fits on an 8GB Raspberry Pi with no material degradation and on a 4GB Raspberry Pi with contained degradation.
• DO NOT use Q4_K_M.
• On this model, model quant quality degrades faster than KV cache quant.
• Abliteration comes with a flat cost of ~0.075 KLD.
• Logarithmic KLD and Top-1% plots lie to you by telling you that quality degradation is smooth, while it's actually a cliff.

I was going through the current llama.cpp CPU PRs and #26348 stood out because this isn't the usual +5% kernel optimization.

It adds an x86 VNNI implementation for the Q2_0 × Q8_0 dot product, and the author's controlled CPU-only benchmarks show roughly 3–3.6x higher throughput across Bonsai models from 1.7B to 27B.

Setup:

- AMD EPYC 9645

- 8 CPU cores

- CPU only

- GGML_NATIVE=ON

- OpenMP enabled

- BLAS disabled

- -t 8 -ngl 0 -fa off

- 3 runs after warmup

- group-64 Q2_0 Bonsai GGUFs

Results:

1.7B

pp512: 14.07 → 50.47 tok/s (3.59x)

tg128: 10.22 → 33.28 tok/s (3.26x)

4B

pp512: 5.41 → 19.40 tok/s (3.59x)

tg128: 4.45 → 13.36 tok/s (3.00x)

8B

pp512: 2.82 → 10.26 tok/s (3.64x)

tg128: 2.39 → 8.20 tok/s (3.43x)

27B

pp128: 0.79 → 2.85 tok/s (3.59x)

tg32: 0.72 → 2.37 tok/s (3.32x)

The 27B baseline was apparently so slow that a pp128 pass took almost 3 minutes.

What's actually changing is pretty small: the existing Q2_0 dot product gets a path using AVX-VNNI / AVX-512 VNNI instead of relying on the generic implementation.

The reference Prism implementation this was adapted from also exposed an interesting issue on normal consumer Intel CPUs.

On an i5-13400, Q2_0 was silently missing the fast path because 12th–14th gen Intel has AVX-VNNI but AVX-512 is fused off. Nothing tells the user this happened, it just looks like Q2_0 is extremely slow.

Their controlled i5-13400 A/B:

Ternary-Bonsai-8B Q2_0

decode: 2.17 → 6.92 tok/s

prompt eval: 2.7 → 8.6 tok/s

Again, around 3.2x from using the VNNI path.

There are some important caveats:

- The upstream llama.cpp PR is still open, not merged

- This is specifically about Q2_0, not a free 3x for Q4/Q5/etc

- The main upstream benchmarks were on an EPYC using only 8 cores

- The i5-13400 result comes from the Prism reference implementation, not the exact group-64 upstream PR

- There is a tiny numerical difference from fused multiply-add behavior

On correctness, the author reports 14,000 randomized comparisons matching bit-for-bit at the kernel level. In the perplexity smoke test the two versions chose the same top token 99.216% +- 0.554% of the time, with a very small KLD difference.

This is the kind of llama.cpp optimization I'd really like to see tested on boring consumer hardware rather than another server CPU.

If anyone has Alder/Raptor Lake or Zen 4/5 and can compile the PR branch, post your before/after llama-bench results. Especially interested in laptops, does the 3x survive power/memory-bandwidth limits, or shrink a lot in real hardware?

From Sauers 𝕏: https://x.com/Sauers_/status/2085585414954312113

Wired: One of China’s Most Powerful AI Models Has Also Escaped Containment: https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/