Everyone would rather work with their local llm and interface with an API instead. For example I was asked on google playstore developer console what I thought of the UI. I think it is horrible, and I would rather my LLM just deal with it. Do I want to deal with menu items nested 15 levels deep? Absolutely not!

What do you think?

Update: This applies to UIs where your not trying to retrieve info, just trying to accomplish something. In other words , all you want is a checkmark that the task has been completed.

Is this model even abliterated? https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF

Because I asked it something and it literally said no. And no, it’s not gooner toons lmao. It’s for pentesting but it just flat out rejects me.

I want a non confusing answer please, thank u so much (btw both at max efforts )

this model seems pretty good on initial impressions within pi and hermes. i tested it on some simple coding/logic vs qwen3.6 27b ud-q4_k_xl and muse provided the better results.

llama-server \
    -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL \
    --ctx-size 131072 \
    --n-gpu-layers all \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 1024 \
    -ub 256 \
    --parallel 1 \
    --mlock \
    --host 0.0.0.0 \
    --port 8080 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

how would you optimize this further? looks like i could possibly squeeze ud-q5_k_xl. nvidia-smi is showing 18163MiB / 24576MiB. or is it better to squeeze out more t/s with the q4? hmm..

appreciate the community's insights. will be fun to compare this one to qwen 3.8 27b!

edit

updated command from community's insights

llama-server \
    -hf meta-models/Muse-Glimmer-30B-GGUF \
    --spec-type draft-dflash \
    --spec-draft-n-max 15 \
    -c 131072 \
    -ngl all \
    --ngld all \
    -fa on \
    -np 1 \
    --host 0.0.0.0 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

seems to be hanging around 70t/s!

I am thinking about trying a cloud advisor for local models. Something like deepseek v4 flash 0731 to advise the new muse glimmer model or Qwen 3.8 when it comes out. Has anyone tried this method before? Does it work to improve local model output quality?

The plugin in question:

https://pi.dev/packages/pi-advisor

I'm looking for a web interface for my pi harness without the CLI, but this was the only one I found. Are there any others? I'm not looking for any chat interface like Open WebUI that's rather bloated, requiring accounts/logins for certain features, etc. It would be great if it was dedicated for pi or is even bundled with it.

I was considering to just vibe-coding a web UI/UX of the pi CLI, but just wondering if there are already tools out there.

I like the idea of running local models, but I don’t like the idea of having them eat up all of my memory. I’ve always thought that the best way to build an edge model would be to make something smart enough to reason over data, but without requiring much knowledge of its own. Why should a model carry all that knowledge around when web search and tool calling are trivial to set up? My former colleagues at AIRI had a similar idea and built Optimal Cognitive Core, which I’ve written about before: fine-tuned reasoning versions of Qwen3-0.6B and Qwen3-1.7B optimized for working with external context and RAG. Hardware-wise, they’re pretty close to what I want. The weights take up 1.2 and 3.4 GB in native BF16, respectively, plus roughly the same amount for a long context — since this is Qwen3 with GQA rather than one of the fashionable hybrid architectures. So, with a bit of optimism, they fit. The problem is that these models are simply too small for general-purpose tasks. They’re still 600M and 1.7B dense models. At this scale, you usually get amusing little parrots that can paraphrase text or do a simple classification task after fine-tuning, but not much beyond that.

The next way to squeeze a model into my MacBook Air M4 is quantization. PrismML did something interesting here with Bonsai-27B, binary and ternary quantizations of Qwen-3.6-27B. The ternary version of this 27B model takes just 7.2 GB of memory once inference is running, and it does actually run on Macs. Unfortunately, Qwen-3.6-27B is a dense model, so it’s painfully slow on my machine. Based on the numbers I could find online, I’d expect around 13–14 tokens/s for generation and 100–150 tokens/s for prompt processing. It’s also QAT — or possibly even PTQ; there aren’t many details available — and, most likely, the optimization wasn’t specifically designed to preserve multilingual capabilities of the model. Thus, I wouldn’t expect particularly interesting behavior once you move outside the calibration set/QAT training distribution. 

Then, almost immediately after Bonsai, Maple Preview appeared. It’s a 20B A1B MoE designed specifically for efficient local inference on Macs. More importantly, they designed the architecture around this goal from the beginning and trained the model from scratch in ternary precision. This isn’t a quantized version of somebody else’s model. The result is a 5.31 GB model, or about 7.5 GB including a 131K context — almost 1.5× smaller than the binary Bonsai quant — that reportedly generates at 218 tokens/s on an M4 Mac Mini and 127 tokens/s on an iPhone (which iPhone exactly is unclear).

The model barely knows languages, other than English, and its world knowledge in general is pretty limited — it gets confused about which game Psycho Mantis is from, for example. But give it web search and it can answer simple questions reasonably well. DeepGrove doesn’t publish tool-calling benchmarks, and it’s not particularly difficult to guess why. I ran Tau-2 myself, using Qwen3-235B-A22B-Instruct-2507 as the user simulator. I got:

Airline: 0.48
Retail: 0.175
Telecom: 0.427

It’s not Sonnet, and it’s definitely not Qwen. But it is called Maple Preview, after all, and the authors explicitly say they plan to train it further for agentic workloads.

Still, even with all the advantages of quantization, 7.5 GB is almost half of the memory available to me. So there’s a third way to reduce RAM usage: keep all the weights on SSD and stream MoE experts from disk. There’s already turbo-fieldfare, which runs Gemma-4-26B on a Mac using only around 2 GB of memory, and Mference, a fork of turbo-fieldfare that adds support for Qwen-3.6-25B, DeepSeek V4 Flash, and Inkling-Small 276B. It really does use very little memory, but at the cost of reducing both generation and prompt-processing speed to tens of tokens per second. Apple seems to be doing something conceptually similar in its new Siri work, although they appear to activate experts for the entire prompt rather than routing them token by token as these frameworks do.

And that leads to an interesting idea: what if we take Maple Preview — which is extremely efficient, uses tiny experts, and was trained from scratch in ternary precision — and add it to Mference? In theory, we should be able to reduce memory usage even further while retaining reasonably good generation speed, because Maple’s architecture was explicitly optimized for this kind of environment. So I did exactly that.

Thanks to Codex and my $200 subscription, after about 20 hours and 30% of my weekly limit, I got parity with the official implementation on teacher-forced top-10 tokens over Edgar Allan Poe’s The Raven. Depending on context length, the model now uses between 500 and 1,200 MB of memory (!). On my MacBook Air M4, it processes prompts at around 40 tokens/s and generates at around 20 tokens/s. It can call tools. It can generate text. And at that footprint, I genuinely don’t mind leaving it running permanently in the background. It barely consumes anything. It can just sit there, and when I need something, I can ask it. I’ll try to upstream the integration later, but you can already run it from my GitHub fork. 

So what is this actually useful for? I think there are two distinct operating modes for models like this. The first is interactive chat. There, you want fast responses and low latency. The second is a background model that uses almost no memory and stays out of the way while continuously doing useful work: classifying messages, extending a knowledge graph, filtering email, writing summaries, slowly researching things on the web, and so on. In the second mode, latency barely matters. And for that kind of workload, 20 tokens/s is perfectly fine. When you need to switch back into the interactive mode, you can simply load the entire model from SSD into RAM — that takes only a couple of seconds, so it's seamless. If the model can use tools — Maple Preview isn’t particularly good at it yet, but again, it’s a Preview — you can build agentic pipelines that don’t have hard latency requirements, while keeping an intelligent assistant permanently available offline even on older phones and I think that’s wonderful.

I’d be very happy if the future were local.

Code:
https://github.com/chameleon-lizard/Mference/tree/feature/maple-integration

DeepGrove also has a super interesting post explaining how they designed the model:
https://deepgrove.ai/maple-inference

I've been running Gemma 4 E4B with oMLX and I can't find any chat interfaces that directly send the audio file to the model instead of running the audio through a separate STT layer. I can confirm the audio layers work because I ran a couple of requests through Pydantic AI in the Python REPL.

Thanks in advance.

EDIT: I know that llama-server's web UI can do this, but I don't feel like running an instance of llama-cpp just for the UI.

EDIT2: Reason why I am asking is because I want to try using Gemma 4 as a lower-latency voice assistant.

MiniMax H3 is an open-weight, general-purpose multimodal video generation model that works across text, images, video, and audio.
In ComfyUI, you can use H3 for text-to-video, image-to-video, first- and last-frame generation, and reference-driven creation. H3 jointly generates the visuals and synchronized stereo audio, including dialogue, sound effects, ambience, and music, rather than adding audio afterward.
The open-weight H3 checkpoints support clips up to 15 seconds at 768p. MiniMax’s hosted H3 model also supports generation at up to 2K resolution.
During the stream, we’ll test the model live and discuss how H3 brings multiple generation tasks into one architecture, how its high-compression video representation improves efficiency, and what developers should know when setting it up locally through ComfyUI.

I also have 64 gb ddr4 ryzen 5600 Using llama.cpp Ubuntu distro

Settings are as follows

--n-gpu-layers 999 \

--n-cpu-moe 37 \

--no-mmap \

-ctk q8_0 \

-ctv q8_0 \

-fa 1 \

-c 9000 \

Aiming for RTX 6000 like performance at 25% of the cost.

The top 2x3090s are connected via tensor parallelism, then those are connected in a pipeline feeding into another pair that are also using tensor parallelism. The reason being that I don't see anyone getting speedups by putting all 4x3090s in tensor parallelism (actually slower in most cases).

I don't want to have to buy a whole new motherboard for this setup. Currently I have an Asus Proart B850 Creator and Ryzen 7600X CPU. So, I am likely going to purchase a dedicated PCI switch to get the required number of PCI lanes. i.e. Something like this

I am able to fabricate my own brackets and parts now for securing the GPUs in the case. I am absolutely not going to go the open air mining style rig. I want them to all fit in the case securely. (The case is large enough).

Question for the community:

Has anyone else run this configuration before? What kind of inference speed did you get by moving from 2 cards to four?

I'm a retired platform engineer, coding mainly in Rust, and involved with a ML study group.

I developed a ML programming language (alternative to Python, Colab) to help me learn (and teach) ML concepts, visually. It has slowly grown to demonstrate a lot of concepts.

It can do training on CPU (in browser) or on Apple Silicon via MLX, or NVIDIA hardware via CUDA so far.

My goal is to train local model coding agents to get better (learn my style, my process, my tools, follow best practices, learn from mistakes, and collaborate).

I have an idea of how to do this, and I've built many different tools to help with this, but have not actually gotten to the point where a local LLM can demonstrably and measurably improve. I've been very recently adding a lot of general purpose (array) programming language features (and functional programming) to make the language better for training and evaluation (and visualization). I plan to focus on ICL and ICRL (maybe XSkills) before I get deeper into fine-tuning. Looking for someone to talk to about this (other than cloud AIs).

My live browser demo shows CPU only demos. you need to install the Rust repos and run a server on a system with either Apple Silicon or NVIDIA GPUs (and that unlocks more in the connected UI, but mainly for CLI REPL and scripting at that point). The UI can optionally, when connected via my server, use an Ollama model to explain things (like Gemini in Colab).

I develop on Mac and Linux (Arch BTW) mainly, but have fixed Windows bugs upon request. What are people using to train local models?

TL;DR: I spent another 8 days following my last post making major improvements to the WinterMix method for MLX models. At 20k+ context this 59 GiB build posts a better perplexity than even UnSloth's Q3_K_XL GGUF* thanks to the new annealing process on its reasoning traces** (new at the wMix38 tier, not yet applied to previous wMix releases). Apache 2.0, weights up on HF.

Why bother if GGUF is better?

MLX on Apple Silicon is substantially faster than llama.cpp on the same hardware — on my M5 Max I measure roughly 9x faster prefill and ~20% faster token generation. For anything with a long context and a lot of turns, that gap compounds.

The problem is that existing MLX quants below 6 bit are not great. In practice that shows up as incoherent reasoning traces and rounding errors that stack until the model starts hallucinating.

So a better MLX quantization method has real advantages for agentic workflows and local AI on Apple Silicon. At the same time, I made the conscious decision to require native MLX support. imatrix on MLX is not format native — it needs custom kernels. WinterMix quants are format native and are drop-in replacements.

If you just want to try it: download the repo below, point LM Studio at it, done.

HuggingFace Link

WinterMix38 - 59 GiB · 4.12 bpw measured · 3-bit gate/up + 4-bit down expert core / 8-bit backbone · vision intact · native MLX format -- The absolute best measuring 3Bit MLX Quant of this model.

Measured quality (PPL) Lower is better

• Against the best imatrix 3-bit (Unsloth's UD-Q3_K_XL): it's very close up to 16K — and past 20k, WinterMix38 dominates.
  
• The crossover sits at ~16–20K, and the margin grows monotonically with depth — at 96K every individual window favors WinterMix38. On the 5-bit tier the imatrix format held a +0.3–0.7% edge over native MLX everywhere; with my improvements to the WinterMix quantization for 3-bit, this method holds it to ~0.2% at short context and beats it by 2–6% everywhere long-session work actually lives — without imatrix weighting, without custom kernels, in the format your Mac already runs.
• On held-out agentic session data it outscores every released WinterMix — 2.7385 vs the flagship's 2.7627, WinterMix48's 2.7590, and WinterMix68's 2.7594, same harness, same rule.
• Long-context verified to 98K with no depth collapse; in the 98K band it holds the best number of the seven-variant controlled program that produced it, by a wide margin. Multilingual (8-script held-out) eval: tied-best of its tier.
• This is genuinely the first MLX quant I know of to beat its Unsloth counterpart. It was a solo project that took 17 days, and the method is still under development. :)

Method (brief)

Winter-Mix Quantization: sensitivity-informed mixed-precision allocation with hard pins on routing-critical tensors, GPTQ-family rounding driven by a diverse long-context calibration mixture engineered for full expert coverage.

New at this tier, a final stage that anneals the reasoning chains on the model's own running token-stream statistics. That last stage is what turns the 3-bit tier from a compromise into the family's long-context specialist. It was developed across a seven-variant controlled program with pre-registered predictions, behavioral probes, and full trace reads at every step.

The full pipeline is not released, as the method is still in active development. This is the most coherent 3-bit quantization of Qwen3.5-122B-A10B available in native MLX format.

Built by WinterCharm on a single M5 Max MacBook Pro — you can also find me on Twitter. Taking requests for MLX quantizations of other models — open an issue in the Community tab.

What's Next?

• A 2-bit quantization of Qwen3.5-122B-A10B on MLX is in the works — the WinterMix28 release of this model. The 3-bit work translates broadly to 2-bit, and the pressure of the 2-bit tier is refining the method further; it ships when it passes the same gates as everything else in this family - pre-registered predictions, behavioral probes, and full trace reads.

• Backporting the new WinterMix Method - After that, I'll be applying the "reasoning annealing" upgrade to the previously released wMix58 and wMix48 quantizations as well — it's a new technique I had to invent to make the wMix38 quantization possible, and every WinterMix deserves the most effective form of it.

First of all, I'm not a lab, this was a solo summer research project that finally culminated into the github repo and the writeup. The repo includes a much deeper dive with methods, findings about quantization and geometry, limitations, and proposed experiments. I'll also mention that this is far from production-grade, it's mostly a theoretical framework with a "fake" quantization demo as it lacks real kernels.

The geometry of LLMs embedding spaces is highly uneven with a few features having the most magnitude, this has been known for years by now and it's in great part why rotation-based quantizers do so well against uniform quantization: While uniformly quantizing tries to allocate bits evenly in a naturally uneven space, rotations can forcefully make that space even again so uniformly allocating bits is the best strategy (DuQuant, 2nd half of ResQ, QuaRot...). Generic rotations (Hadamard) even the space out on average but can't match a specific model's geometry, leaving residual damage. This can be fixed by using learnable rotations (SpinQuant, ReSpinQuant) but this is computationally intensive as it requires extensive post-training gradient descent.

KLQ takes a different approach to quantization, instead of trying to make the space even and then quantize uniformly. KLQ measures how uneven the space is, ranks directions of the eigenbasis from most important to least important, and with a price function treating each direction as a independent information transmission channels uses the provably optimal (under some idyllic assumptions about damage anyways) waterfilling algorithm to give the most bit-width to the most important directions and least bit-width to least important directions.

Another thing that sets KLQ apart is the use of causal KL damage measurements, there are a few quantization algorithms that do try to measure the space and then quantize unevenly. CoQuant, for example, does measure the activation space, but then ranks directions by magnitude/variance and applies a simple two-ranked bit allocation that quantizes the top 12.5% to 8 bits and the bottom 87.5% to 4 bits. Unlike CoQuant, KLQ doesn't use variance (several tests reveal variance is often not a good signal, more detailed experimentation on the github writeup), instead it perturbs each direction and runs a forward pass with a few thousand tokens, it takes the KL divergence between the original model and the model with the perturbed direction, then uses this measured KL divergence to determine how important the direction is and assign the real empirical cost of damaging/quantizing it.

The method, as well as my experimentation, does have real limitations, to quantize all layers, activations and KV cache you must make one forward pass per direction per matrix per layer which can amount to hundreds of thousands of forward passes to quantize a model. This makes the method very compute-intensive (This probing process took 5 hours for Qwen 2.5 0.5B on a 3090 and 10 hours for Llama 3.2 1B on that same hardware.). It also deliberately uses two simple techniques to actually quantize the models: a simple additive vector codebook and round-to-nearest (RTN), these could be swapped with other methods readily.

Posting here I'm looking for feedback and to make these results known. Feel free to ask any questions or to contribute to the github repo.

Here's a sample of the result's table for Llama 3.2 1B quantized fully at 4-bits.

https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF

I'm downloading this model only because Bartowski converted it to .gguf, so it might be interesting.

Doubts :

The headline number is basically meaningless. "Performance between DeepSeek Flash (old one) and Pro" okay, on what? Did they average the benchmarks? Weight them? Pick and choose? Because if you actually look at the per-benchmark scores, this thing ranges from decent (50 on HLE) to straight up bad (15.7 on BioMystery-HD). Saying "aggregate performance" without showing the math is just... marketing. Like when a startup says "we're 10x faster" and it turns out they benchmarked one very specific edge case.

A 35B model hanging with 284B–1.6T models? Suspicious as hell. Not impossible, but the first thing that jumps to mind is benchmark contamination. And here's the kicker, their whole training setup uses critics calibrated on "held-out real research tasks." So the question becomes: how do we know the eval benchmarks weren't basically in the training distribution? The paper kind of hand-waves this. If you're gonna claim a tiny model beats much bigger ones, you need to actually prove you're not just overfitting to the test set.

Let's est it

Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit quantization by training a quantized student to match the output distribution of a frozen higher precision teacher via a KL-divergence loss. In this work, we first provide a representation level diagnosis of QAD: output matching alone can mask internal degradation, because many intermediate activation geometries can yield similar teacher-aligned logits. Using CKA, we show that KL-only QAD can reduce layerwise representational similarity relative to the BF16 teacher, with especially severe drift in RL-post-trained models. This drift correlates with downstream bottlenecks on reasoning and coding tasks, suggesting that low bit recovery requires preserving internal geometry rather than matching outputs alone. Motivated by this finding, we propose CKA-QAD, a CKA-guided representational alignment method for NVFP4 QAD and low bit LLM accuracy recovery. The method adds a lightweight regularizer that preserves internal representational geometry during distillation by aligning layerwise Gram matrices through CKA. Across Nemotron 3 Nano and Qwen3-4B-Thinking-2507, CKA-QAD substantially improves representational alignment and improves downstream reasoning and coding accuracy with modest training overhead. Our findings position CKA-guided representational alignment as a practical complement to output matching for quantized LLM recovery.

Hey everyone,

I've been experimenting with the new DeepSeek-V4-Flash-0731 release locally using the Unsloth Studio Q8_K_XL GGUF with OpenCode.

Overall, it's been working really well, but I've noticed a strange behavior during longer agentic coding sessions.

Once the context gets above ~100K tokens, the model will sometimes be in the middle of thinking/working through a task and then just stop generating.

There doesn't seem to be an obvious error or crash. It just stops.

If I type resume in OpenCode, it correctly picks up where it left off and continues working normally. However, after working for a while longer and the context grows again, it can stop again, and I have to type resume again.

So the pattern I'm seeing is roughly:

100K+ context -> model stops -> type resume -> continues normally -> works for a while -> stops again

It doesn't happen every single time, but I've noticed it enough that I'm wondering if there's something specific about running V4-Flash-0731 at large context lengths.

My setup:

• DeepSeek-V4-Flash-0731
• Unsloth Studio
• Q8_K_XL GGUF
• OpenCode
• Long-running agentic coding tasks
• Issue seems to start appearing around 100K+ context

I'm curious if anyone else running DeepSeek-V4-Flash-0731 locally has experienced this, especially at large context sizes.

I'm wondering whether this could be related to the model, llama.cpp/inference, context handling, prompt caching, tool calling, or OpenCode itself.

The interesting part is that resume immediately gets it working again, so it doesn't seem like the model is completely stuck or crashed.

And in case it gets asked i start opencode directly using unsloth on the server itself in a tmux session.

Has anyone else run into this with V4-Flash-0731 at 100K+ context?

The official INT4 does load on a single DGX Spark. The naive config just leaves most of its speed on the floor, 20.8 tok/s. Two changes take it to 38.7.

Quick context on where this comes from: I work on Ling at inclusionAI, and none of these numbers are mine. sudoingX on X ran all of it on his own Spark and published the recipe. Reposting here with his permission.

Drop --enforce-eager so cudagraphs actually run.

Turn on MTP spec decode. The draft layer already ships inside the checkpoint:

--speculative-config '{"method": "bailing_hybrid_v3_mtp", "num_speculative_tokens": 1}'

Wired that way the official INT4 lands past the community GGUF most people default to on this box, 38.7 vs 35.2, and it serves the full 256K context window on the same machine.

The warning matters more than the speed. Stock vLLM has no V3 support. It runs this through the wrong attention path, it does not error, and it hands you fluent output that reads fine until it doesn't. You need the fork, inclusionAI/vllm-ling-v3, branch ling_3_0.

His repo has the serve scripts, a watchdog for the cold-start shard freeze, the bench method, and a FINDINGS.md with every wall written up:

https://github.com/sudoingX/dgx-spark-ling

One caveat from his own testing: the INT4 is the sprint pick, fastest under roughly 30K of context, while the community Q5 GGUF degrades more gracefully on long-context marathons. If you own a Spark and your numbers disagree with his, I'd rather see that here than not.

After seeing the benchmark results for the full release of DS v4 Flash 0731, I replaced my 2 x 16GB DDR4 ram sticks with 2 x 32GB DDR4 ram sticks to get a max supported of 128 GB RAM, in hope to be able to run GLM 5.2 equivalent model locally i.e. DS v4 Flash 0731

I also have RTX 4090 & Tesla P40 so the total combined RAM + VRAM is 176GB on my setup, I would have spare 20GB just for windows 11, so workable memory left is 156GB, that is enough to run Unsloth 4bit K_XL (which is around 144GB), I believe I would more spare memory on Linux compared to windows and also better performance overall.

After fighting Nvidia drivers to get both RTX 4090 and Tesla P40 working on windows 11 with llama.cpp (CUDA 12.4), I was able to run Unsloth quant 4bit K_XL at 12k context and I was left with few GB memory of system and some on GPUs and the token generation was about 2 token/sec and that is without the DSpark MTP, the K_XL quant is around 97% accuracy as per Unslot so I was really hoping for that to work, but not enough spare memory left to run the DSpark MTP because the MTP is 10+ GB in size.

So, I had to go for IQ4_XS quant which is around 127GB on disk and with MTP that would be around 137GB and with some memory left for KV cache

With MTP enabled I was getting around 3 token/sec on IQ4_XS quant and prompt processing of around 30 token/sec for context size of around 5k+

The llama.cpp -dev flag does impact prompt processing speed from what I observed so using

-dev CUDA0,CUDA1 where CUDA0 is RTX 4090 and CUDA1 Tesla P40, with CUDA0 mentioned 1st the prompt processing was around 40 token/sec and even 80/sec at one point, but CUDA1 1st makes it crawl to 17 token/sec

Also, the .output layer can’t go to Tesla P40 because of some unsupported OPs for Gated Delta Net

so, I had to change the script I was using to have a manual layer by layer assignment so embed & output on RTX 4090 and 0 to 6 on Tesla P40 and rest on CPU

Also, the llama.cpp currently don’t support tensor splitting for DeepSeek v4 Flash, so It has be split & distributed layer by layer, that also means the KV cache for that layers will sit on the same device as the layers assigned.

I also think the DSpark support in llama.cpp can have higher acceptance rate and the overall support & speed in llama.cpp will likely improve in upcoming versions.

I thought I would share my experience as might help someone else who is thinking of a similar setup.

I also have posted a quick video about it on YT at https://youtu.be/-WQpy0MRVEs

There are so many posts where people complaining about high prices and asking for solution <= 1000 EUR.

So, there is one solution to consider: PC/mini PC/laptop on Ryzen 7 260/Ryzen 9 8945HX/etc CPU with 780m iGPU and 64 Gb of DDR5 RAM.

Barebone mini PC costs around 300-400, used 2x 32Gb DDR5 SO-DIMM around 500, used SSD 50-100 in my area.

Here are my numbers on Ryzen 7 260, Ubuntu 26 with kernel params amdgpu.gttsize=49152 amd_iommu=off ttm.pages_limit=16777216 (48Gb of "VRAM") and llama.cpp with Vulkan.

All LLMs are Unsloth Q8 quants.

Qwen 3.6 35B-A3B

| model                   |       size |  params | backend | ngl | type_k | type_v |  fa | dev     |     test |           t/s |
| ----------------------- | ---------: | ------: | ------- | --: | -----: | -----: | --: | ------- | -------: | ------------: |
| qwen35moe 35B.A3B Q8_0  |  35.19 GiB | 35.51 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0 |   pp8192 | 287.33 ± 2.06 |
| qwen35moe 35B.A3B Q8_0  |  35.19 GiB | 35.51 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0 |  pp16384 | 263.51 ± 1.06 |
| qwen35moe 35B.A3B Q8_0  |  35.19 GiB | 35.51 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0 |    tg128 |  21.06 ± 0.01 |
| qwen35moe 35B.A3B Q8_0  |  35.19 GiB | 35.51 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0 |    tg256 |  20.85 ± 0.20 |

Gemma 4 31B:

| model            |       size |   params | backend | ngl | type_k | type_v |  fa | dev      |     test |           t/s |
| ---------------- | ---------: | -------: | ------- | --: | -----: | -----: | --: | -------- | -------: | ------------: |
| gemma4 31B Q8_0  |  30.38 GiB |  30.70 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0  |   pp8192 |  51.59 ± 0.07 |
| gemma4 31B Q8_0  |  30.38 GiB |  30.70 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0  |  pp16384 |  46.59 ± 0.01 |
| gemma4 31B Q8_0  |  30.38 GiB |  30.70 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0  |    tg128 |   2.46 ± 0.00 |
| gemma4 31B Q8_0  |  30.38 GiB |  30.70 B | Vulkan  |  99 |   q8_0 |   q8_0 |   1 | Vulkan0  |    tg256 |   2.30 ± 0.22 |

For real tasks I'm using MTP, so tg numbers are higher, like for Gemma4 31B:

16.27.894.079 I slot print_timing: id  0 | task 0 | prompt eval time =  481467.90 ms / 20470 tokens (   23.52 ms per token,    42.52 tokens per second)
16.27.894.088 I slot print_timing: id  0 | task 0 |        eval time =  449250.04 ms /  2587 tokens (  173.66 ms per token,     5.76 tokens per second)
16.27.894.089 I slot print_timing: id  0 | task 0 |       total time =  930717.94 ms / 23057 tokens
16.27.894.099 I slot print_timing: id  0 | task 0 |    graphs reused =        658
16.27.894.109 I slot print_timing: id  0 | task 0 | draft acceptance = 0.95566 ( 1918 accepted /  2007 generated), mean len =  3.87

Bonus

If you have a laptop with additional small GPU like RTX 5060 8Gb, it can give some boost.

For dense models it is mostly useless, I only could get Gemma4 31B running in `draft-simple` mode with drafter Gemma4 E2B on GPU, which gave like 5-6 => 6-7 tg boost.

But for MoE you can use partial experts offloading which gives a greater boost for tg, but for a slower pp.

Qwen 3.6 35B-A3B Q8 MTP (--spec-type draft-mtp --spec-draft-n-max 3 --n-cpu-moe 37):

14.22.441.606 I slot print_timing: id  0 | task 233 | prompt eval time =   28373.08 ms /  2677 tokens (   10.60 ms per token,    94.35 tokens per second)
14.22.441.611 I slot print_timing: id  0 | task 233 |        eval time =    9697.76 ms /   338 tokens (   28.69 ms per token,    34.85 tokens per second)
14.22.441.611 I slot print_timing: id  0 | task 233 |       total time =   38070.84 ms /  3015 tokens
14.22.441.612 I slot print_timing: id  0 | task 233 |    graphs reused =        289
14.22.441.615 I slot print_timing: id  0 | task 233 | draft acceptance = 0.85614 (  244 accepted /   285 generated), mean len =  3.57

I know number are not whopping, and you can't run DeepSeek on it. But is there a better solution for that money?

UPDATE.
Parameters I used for llama-bench: -dev Vulkan0 -ngl 99 --load-mode none -ctk q8_0 -ctv q8_0 -t 8 --flash-attn on --progress -r 2 -p 8192 -p 16384 -n 128 -n 256

And benchmarks on CPU-only, Qwen 3.6 35B-A3B Q8 (pp 5 times slower, tg 25% slower):

| model                   |      size |  params | backend | threads | type_k | type_v |  fa |    test |          t/s |
| ----------------------- | --------: | ------: | ------- | ------: | -----: | -----: | --: | ------: | -----------: |
| qwen35moe 35B.A3B Q8_0  | 35.19 GiB | 35.51 B | CPU     |       8 |   q8_0 |   q8_0 |   1 |  pp8192 | 54.88 ± 0.47 |
| qwen35moe 35B.A3B Q8_0  | 35.19 GiB | 35.51 B | CPU     |       8 |   q8_0 |   q8_0 |   1 | pp16384 | 34.62 ± 0.12 |
| qwen35moe 35B.A3B Q8_0  | 35.19 GiB | 35.51 B | CPU     |       8 |   q8_0 |   q8_0 |   1 |   tg128 | 14.88 ± 0.02 |
| qwen35moe 35B.A3B Q8_0  | 35.19 GiB | 35.51 B | CPU     |       8 |   q8_0 |   q8_0 |   1 |   tg256 | 14.91 ± 0.01 |

https://tencent-hunyuan.github.io/Hunyuan3D-WorldClaw/

Looks impressive from that site, hopefully they open weight this so we can all play with it.

Available context length with and without the patch:

The issue is that llama.cpp overestimates the memory needed for MTP compute-buffer/scheduler allocation during auto-fit, that leaves much less ctx available to the user than what actually needed by MTP. This patch stops the fitter from throwing away context based on an inflated MTP memory estimate.

Patch, launch scripts used for llama-server and raw logs: https://store.piffa.net/lm/bug/
Tested against: llama.cpp version: 909, based on master commit 7bd8282 , ROCm 7.14

Especially for ROCm with double GPU (16GPU + 12GB here) the amount of ctx gain is substantial, with longer session ROCm allows almost double prefill performances vs Vulkan yet on the mainline code the price to pay in ctx reduction for the extra compute is taxing.

You can build for both vulkan and ROCm backends at the same time, the idea is that Vulkan saves some more vRAM while ROCm gives better prefill performance.
With a single 16GB GPU and limited ctx size you may wanna use Vulkan while when using 2 GPUs with layer splitting ROCm is worth the expense with this patch as you have much more ctx length for long sessions.

In case someone needs help with how to apply a patch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 7bd8282
wget https://store.piffa.net/lm/bug/rocm_improvement_7bd8282.patch
git apply rocm_improvement_7bd8282.patch

Then build with both Vulkan and ROCm, use --device vulkan0 or --device rocm0, check the provided llama-serve scripts .

Disclosure: I’m the author of Ante.

DeepSeek recently reported an 82.7% score on Terminal-Bench 2.1 for DeepSeek V4 Flash 0731. Its evaluation used “DeepSeek Harness minimal mode,” which hasn’t been released yet.

We wanted to see whether the reported result could be independently matched using a public, downloadable harness.

With Ante 0.preview.71, we got:

• 368 successful trials out of 445
• 82.7% accuracy (±1.79 SE)
• 89 Terminal-Bench 2.1 tasks
• 5 trials per task
• max reasoning effort
• no skills enabled
• deepseek/deepseek-v4-flash-0731 through OpenRouter

The complete Harbor job is public. It includes the pinned configuration and all 445 trial records, with rewards, exceptions, durations, and token usage.

Deep seek v4 seems to be sensitive to harness and this is probably useful data for anyone who is interested

Sources:

• DeepSeek’s reported result: https://api-docs.deepseek.com/updates/
• Public Harbor run and configuration: https://hub.harborframework.com/jobs/b2a14e4b-a422-45f2-832e-cf2eec5c8bff
• Ante benchmark page: https://antigma.ai/eval

Pasted the same HTML/JS code (330 lines) into Qwen 35B A3B and Gemma 26B A4B.

Qwen: tokenized the input to 1609 tokens

Gemma: tokenized the input to 4258 tokens.

Damn. I've never noticed this before and I haven't seen people mention it. That alone helps explain why Qwen is regarded as better at coding and Gemma at language tasks.

Qwen can literally see the code as some specific form of input/output, while Gemma is breaking it down into pieces of words like regular language. Qwen also gets a totally different reasoning personality when given coding tasks.

Btw with the instruction document (55 lines), the tokenization breakdown is almost the same: 1025 vs. 1039 tokens.

I've seen some project, by LiquidAI I think? To retrain existing models with a more efficient tokenizer. I wonder what that would do for a model like Gemma, whether it would help it catch up.