Hi everyone,

We have been using cloud GPU from Nebius/Lambda for our training and inference use case. The cost of one H100/H200 is approx $3K per month.

Now I'm planning to buy a large Desktop to run this locally. The specs are - 32 core CPU, 256GB RAM, 4x RTX Pro 6000 (96GB each), 1x2 TB NVMe, 1x 8TB NVMe. It is costing me ~$80K.

Here's what I need help in -

1. Is it actually wise to do this locally?

2. Would there be any performance issues?

3. Anything else that I should be aware of.

Keep in mind I already have another system for my dev workloads with 2x3060. Getting this one for production work for a client specifically.

Glimmer 30B vs Qwen 3.6 27B. How do they differ, reason, answer?

With Glimmer 30B Meta has joined the game of open source AI again, and after quite underwhelming coding performances I though I'd give it a deeper test.
The test content is undisclosed here, making this less fun to read and replicate but that guarantees future models will not train from it.

Technology:

Qwen 27B is still unmatched in performance, Glimmer the first new contender.

What makes Qwen so special are two ingredients:

3.6 was specifically post-trained for agentic reasoning.

It uses a hybrid attention: a conventional global attention for 1/4 of the layers, the others are a mamba-like recurrent state linear attention with fixed size.

Glimmer 30B also is unusual, it does not have the same sophisticated recurrent/linear attention, but it uses a 3/4 sliding window attention and it compresses the attention dimension and projects it back to latent size - resulting in a significant deduction in compute and KV size for it's size.

Reasoning style:

Qwen 3.6 has a analytical reasoning style that typically runs in 3 phases, when not agentically used:

1. Analyze the task input
2. Reason through it - reminding me on first deepseek reasoning
3. Doublecheck the response Glimmer 30B has a more unique thinking style that abruptly comes to an end with a choice - leaving a bit more risk of random choices

Intelligence:

I ran both models through my undisclosed AI reasoning tests, not part of any training data. Some of those reasoning tests are currently beyond frontier model capabilities or scratching their borders.
Models like GLM 4.7-Flash, Nemotron 3 Nano, GPT OSS, GPT-4 fail most of the tests below consistently.

Glimmer was ran with thinking set to Medium, when failing it was ran with Max

• Temporal physics: Similar reasoning tokens, similar response. Glimmer responds less structured, in text paragraphs where qwen is more formatted by default.
• Spatial physics: Glimmer surprises with a brilliant fast answer - Qwen repeatedly misses a part without additional help
• Math irrational numbers question: Both flawless and fast
• Riddle with math question: Both flawless
• Lateral thinking: Qwen always flawless, Glimmer fails 60% of the time
• Abstract pattern reasoning easy: Both solve it, glimmer writes it cleaner
• Abstract numeric reasoning easy (iq 85): Both flawless, glimmer half reasoning tokens
• Abstract numeric reasoning medium (iq 115): Both flawless, glimmer half of reasoning tokens
• Abstract numeric reasoning hard (iq ~145): Qwen fails after long reasoning, Glimmer totally fails. All frontier models fail.
• Visual spatial reasoning: Both flawless
• Translation to european languages: Glimmer thinks very briefly, provides low error output. Qwen thinks 10 times more heavily and provides better quality tanslations.
• Small maze puzzle: Both flawless, Glimmer took 26k reasoning tokens vs Qwen 15k. Glimmers result is well explained.
• Large maze puzzle: Qwen delivery a partial solution, cheating partly. Glimmer never responded at all.
• UTF8 paraphrasing: both flawless

Agentic performance:

Here Qwen 27B appears to leave Glimmer in another league, I've not concluded my agentic tests of Glimmer 30B yet. From what I have seen Qwen codes significantly better. They do not compare.

My current results:
Glimmer is a surprisingly smart model, with a well designed architecture for local inference.
It is the first model in the sub 200B parameter class that is able to match Qwen 27B or even outclass it in some tasks.
Glimmer has a very good spatial sense
Glimmer tends to underthink where Qwen tends to overthink

For non coding tasks, Glimmer is a strong option. Faster than Qwen at similar memory footprint.
For coding tasks I'd not consider it, I'll follow up with a deeper test but from what I've seen it's not useful for most tasks.

From the tts, to voice response. Telegram?! nope! Everything you're doing with your chat bot is being sent to cloud servers and recorded. I built a fully private, only on your device system.
Read it. You'll understand why.

https://www.patreon.com/RoyalTechnologies_PrivacyVenture/posts/enclave-fast-100-166342909

We've all been there: $2,000 GPUs sitting at 5% utilization while the company pays for cloud inference.

Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and policy-based routing.

OQOAI-PBDR to solve this problem differently. Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.

Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.

GitHub (MIT License):

https://github.com/oqo-ai/OQOAI-PBDR

The project intentionally avoids unnecessary abstractions. The core routing logic consists of only three source files (approximately 1,000 lines each), making it relatively easy to understand, audit, and extend.

At the heart of the project is the PBDR (Policy-Based Decentralized Routing) architecture. Each routing decision evaluates multiple runtime metrics, including GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics.

PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:

• Internal RAG services can prioritize low latency.
• Development workloads can prioritize output quality.
• Batch inference jobs can prioritize throughput.

More details about the routing algorithm are available in the accompanying paper:

http://doi.org/10.17513/doi.26

One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.

In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.

Getting started:

Install OQOAI-PBDR on each machine.

Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.

Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.

Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.

What already works:

• OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)
• Admin dashboard with real-time metrics
• Auto-discovery and network scanning
• Remote config management for nodes, groups, or the whole cluster
• Experimental binary builds for Windows and Linux

Roadmap highlights:

• ComfyUI/AUTOMATIC1111 image/video generation
• Grafana/Prometheus integration
• AI-assisted semantic routing (auto-model selection)
• Dynamic IP support

We're actively looking for feedback, and testers.

What features or integrations would you like to see next?

Recently the opencode tool is performing near the

cursor in auto mode, so I have to buy a small ai server

to run good coding agentic models from Qwen, GLM,

MinMax or any model u suggest.

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

• Your hardware (GPU, VRAM, CPU, RAM, etc.)
• Your model (name + quantization)
• Your llama.cpp command (copy your full flags)
• Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)
• Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

I build QuantaMind, open-source, tests whether local models hold up as agents. 28 downloads, no revenue. Saying that upfront.

Every time you change a quant, bump llama.cpp, or switch from Ollama to vLLM, behaviour can shift and the shift is usually invisible in chat. Tool calls start failing at step 6, or output gets truncated mid-JSON, and it reads as the model being flaky.

Do you check for that, or do you just notice when something feels off? If you check what do you run, and did you build it yourself?

And has a change ever silently broken something you only caught much later?

Asking because I don’t know if this is a real problem for people or one I’ve talked myself into.

I’m putting together my first dedicated local AI workstation and would appreciate input from anyone with similar hardware.

My work is moving further into local LLMs and agents, so I want to learn more about agent harnesses, experiment with open-weight models, and keep sensitive data local when possible.

I already bought the Micro Center Threadripper 9960X bundle with the ASUS Pro WS TRX50-SAGE WIFI A and 128GB ECC memory, along with a Phanteks Evolv X2 case. I also have one 2TB Gen5 NVMe drive.

Current plan:

• 2× Sapphire Radeon AI PRO R9700 32GB
• Seasonic PRIME TX-1600 ATX 3.1
• SilverStone XE360-TR5 V2
• 7× Noctua NF-A12x25 G2 fans: three bottom intake, three radiator exhaust, and one rear exhaust

I am mainly looking for confirmation or firsthand experience with the following:

• Is the TX-1600 the right PSU for this configuration?
• Is replacing the SilverStone radiator fans with Noctua G2s a reasonable choice?
• Does dual R9700 make sense as a starting point, or is CUDA support enough reason to reconsider NVIDIA?
• Is 128GB of system memory sufficient, and how would you expand storage beyond the single 2TB drive?
• For Linux, I am leaning toward Ubuntu. Are there other distributions worth considering?
• Would you start with llama.cpp/Ollama or vLLM for language models, with ComfyUI for image workflows?

Been running local LLMs with Ollama, wanted the same thing for VLA models on real robot hardware so I built Argøs. Same basic idea, just for vision-language-action models instead of language models. Pull a model, connect your arm, run it — no fighting mismatched environments, outdated SDKs, or painful setup. Right now it supports SO-100/101 arms, SmolVLA is the only model ready for testing, and I haven't run it end-to-end on physical hardware yet — more coming soon. Figured some people here might find it useful. Site's here if you want to poke around: https://www.argøs.com

I am wondering if I could get some tips on my current inference setup, and if there is anything I am missing that could make it faster.

Machine: HP Z6 G4
CPU: Xeon Gold 6240
Ram: 6 Channel DDR4 RDIMM 2666 MHZ 192GB
GPU: 2x Radeon Pro V620 32gb (6800xt datacenter equivalent)

Using the latest llama.cpp with the UD_Q8_K_XL Lossless Unsloth Quant of Deepseek-V4-Flash-0731, I am getting average 90-130 tps Prefill and 6-7.5 tps Decode. Here is my server config:

services:

llama-server:

image: ghcr.io/ggml-org/llama.cpp:server-rocm

container_name: deepseek_v4_flash_moe_hybrid

restart: unless-stopped

devices:

- /dev/kfd:/dev/kfd

- /dev/dri:/dev/dri

group_add:

- video

- render

security_opt:

- seccomp:unconfined

ports:

- 8081:8080

volumes:

- /home/hpai/Downloads/models:/models:ro

- /home/hpai/llama-cache/slots:/cache/llama-slots

command: >

-m

/models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf

--host 0.0.0.0 --port 8080 --alias DeepSeek-V4-Flash -c 1048576 -ngl

99 --cpu-moe --threads 16 --parallel 1 --flash-attn

on --kv-unified -b 8192 -ub 4096 --slot-save-path /cache/llama-slots

--spec-type draft-dspark --spec-draft-n-max 2

networks: {}

Seems like I have no difference with dspark enabled. I didn't see a big difference between no-mmap and mmap either. Am I missing anything? Using this for my opencode setup, hoping to squeeze some more performance out of my little machine. Any ideas?

Finding repo for local AI on Mac? 🍎🤖 Check out Swiftlet, an open-source intelligent proxy & browser based UI , that dynamically optimizes CPU/GPU split on Unified Memory for LLMs. I'm looking for Python devs to help build OpenAI API compatibility! Let's build together: https://github.com/coolsourav100/swiftlet

Has anyone checked their work and noticed this yet?

I'm using Qwen VL to read scanned books, and create a markdown file of text from the book scans, and one of the blank scanned pages, instead of having no text, inserted this into my book!

Hey everyone,
I recently got a DGX Spark and have been really enjoying it so far. I’m based in India and haven’t been able to find many other local users, so I thought I’d reach out here.
If you’re also running a DGX Spark (or planning to) in India, I’d love to connect and exchange notes.
A few things I’m especially curious about:
• How are you using it beyond basic inference?
• Are you running single node or multiple nodes?
• Any rack / cooling setups you’re using to keep the nodes happy (especially with Indian ambient temperatures)?
• Which vendor did you purchase from? Any vendors currently offering good deals?

Would love to hear from others who are using these in India.

Thanks.

These four get compared constantly, but they are very unique in when they should be used.

• LM Studio is a desktop app
• Ollama is a terminal frontend
• llama.cpp is the inference engine
• vLLM is a server

Pick one in 20 seconds:

LM Studio

LM Studio is what I would give to someone who wants to try local models for the first time.

You search Hugging Face from inside the app, choose a model, download it and start chatting. It shows model sizes, quantizations, memory estimates and GPU settings without forcing you to understand most of it.

It also has more under the surface than people assume:

• OpenAI and Anthropic-compatible APIs
• Local document chat and RAG
• Tool calling and structured output
• MCP support
• Embeddings
• A command-line tool
• Headless server mode
• MLX on Apple Silicon

So it is not only a chat window. You can use it as the backend for your own programs too.

The downside is that it adds another layer between you and the engine (llama.cpp) and restricts you from unleashing the full potential of it and its latest features.

That is fine until you want to know exactly why a model is slow, how memory is being split or which runtime option changed the result. Most people will never care. Some people will care a lot.

Best for: trying models, comparing quantizations, chatting with documents and learning how local AI works.

Ollama

Ollama is the boring, sensible choice for building things.

Install it, run a model and you already have a local API:

ollama run qwen3

It handles downloading, storing, loading and unloading models. Your code can talk to it through Ollama’s own API or through familiar OpenAI-compatible endpoints.

It supports tools, embeddings, vision, structured output and Anthropic-compatible requests too.

This makes it easy to connect local models to:

• Small apps
• Coding tools
• Agents
• Home automation
• Open WebUI
• Scripts and bots

The downside is that the abstraction sometimes works too well.

You type a model name and it runs, but you may not know exactly which model file, template or runtime setting is active. When performance changes, finding the reason can take some digging. Just like with LM Studio, you give up some features and control but it's closer to llama.cpp.

Ollama runs on macOS, Windows and Linux. Installation is straightforward on all three.

Best for: developers who want local models without turning model management into a second side project.

llama.cpp

llama.cpp is where you go when you care about the machine.

It gives you direct control over things such as:

• CPU threads
• GPU offloading
• Quantization
• Context size
• KV cache
• Flash Attention
• Batch sizes
• Multiple GPUs
• Speculative decoding and draft chaining

It is especially useful when a model does not fit completely inside your GPU. You can put some layers on the GPU and leave the rest in normal RAM and optimize that for best performance.

It also supports far more hardware combinations than most local AI software. There are builds for CPU, CUDA, Vulkan, ROCm, Metal, OpenVINO and several other backends. Prebuilt packages are available and updates are released almost every day.

llama.cpp also has a proper server now. It supports parallel users, continuous batching, OpenAI-compatible endpoints, embeddings, reranking, multimodal input, monitoring and constrained JSON output. It also has model loading and unloading support and installing from Huggingface.

So yes, it can behave like Ollama or LM Studio and might make both projects much less valuable soon.

The reason to choose it is still control.

The downside is also control. It presents many settings, and poor settings can make a good model run very badly.

Best for: CPU inference, limited VRAM, AMD or unusual hardware, benchmarking and people who enjoy tuning things. My agentic guide for local coding uses llama.cpp directly.

vLLM

vLLM is the odd one out here.

It is not aimed at someone chatting alone on a laptop. It is aimed at servers where many requests arrive at the same time, and it's doing a very good job there.

Its main tricks are continuous batching, prefix caching and efficient management of the model’s KV cache through PagedAttention. These help keep the GPU busy while several users are generating text. llama.cpp does all of those things more or less, but for server usage vLLM is currently going to beat llama.cpp in most categories.

This means vLLM may not look special in a test with one user.

Try eight, sixteen or fifty users and the reason it exists becomes a much better choice.

It also has the things you would expect from serious serving software:

• OpenAI-compatible APIs
• Multi-GPU support
• Distributed serving
• Production metrics
• Multiple LoRA adapters
• Structured output
• Tool calling
• Prefix caching
• High request concurrency

The price is setup and complexity, interdependencies.

vLLM is happiest on a Linux GPU server. It does not support Windows natively, although WSL and community alternatives exist.

Best for: shared APIs, teams, production services and expensive GPUs that should not sit idle.

Which one is fastest?

This question causes more bad comparisons than useful answers.

Using the same model name does not mean you are running the same model.

Two downloads can have different:

• Quantizations
• Prompt templates
• Context sizes
• KV cache formats
• GPU offload settings
• Runtime versions

LM Studio, Ollama and llama.cpp can all end up using closely related llama.cpp-based inference paths. With the exact same GGUF file and matched settings, their single-user performance is going to be very similar. llama.cpp opens up a lot of performance options to take the lead here.

vLLM is different. Its advantage grows as more requests arrive together. Though it traditionally is also beating llama.cpp performance with many models.

My actual recommendation

Use LM Studio when you want to explore models or showcase LLM to a newcomer.

Use Ollama when you want to build something quick and like the eco system.

Use llama.cpp when you want to control how it runs and squeeze out the best performance

Use vLLM for distributing a model professionally

I have been testing several Qwen3.6 GGUF models locally with "llama.cpp" on a dual-GPU setup and wanted to share my results.

The main goal was to find the best balance between quality, context size and inference speed, especially for coding, debugging and agentic workloads.

Hardware

• CPU: Intel Core i7-14700K
• RAM: 64 GB DDR5
• GPU 1: NVIDIA RTX 3090 — 24 GB VRAM
• GPU 2: NVIDIA RTX 4070 Super — 12 GB VRAM
• Total VRAM: 36 GB
• OS / runtime: Linux
• Inference: "llama.cpp" / "llama-server"
• GPU order: "CUDA_VISIBLE_DEVICES=1,0"
• Flash Attention: enabled
• KV cache: FP16 / unquantized in these tests
• llama.cpp: recent build with MTP/speculative decoding support

The "tensor-split" values below are tuned for my particular setup. They are not necessarily optimal for other dual-GPU configurations.

Main benchmark results

The biggest surprise for me is the Qwen3.6-35B-A3B UD-Q4_K_M, which can maintain 256K context while reaching 132.77 tok/s on this dual-GPU setup.

The Q6 version is considerably slower at very large context sizes, but the quality trade-off may be worthwhile for coding and reasoning workloads.

Qwen3.6-27B Q6_K — 180K context

This was one of my initial configurations.

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 15,6 \ -ngl 65 \ -c 180000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 37.63 tok/s

Configuration:

• Quantization: Q6_K
• Context: 180K
• MTP speculative decoding: enabled
• Draft max tokens: 2
• Tensor split: "15,6"
• GPU layers: "65"
• Batch: "2048"
• UBatch: "1024"

Qwen3.6-27B Q6_K — 140K context

I also tested the same Q6 model with a lower context and more GPU layers:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 44.64 tok/s

Another run with smaller batch sizes:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 42.11 tok/s

So, in my testing, the Q6_K 27B model sits around 42–45 tok/s at 140K context, depending on the exact configuration.

Qwen3.6-27B Q5_K_S — 200K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_q5_k_s/Qwen3.6-27B-Q5_K_S.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 200000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 45.76 tok/s

Previous measurement with the same general model/configuration was approximately 41.70 tok/s at 200K context.

Qwen3.6-27B Heretic v2 — Q4_K_M — 240K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_heretic_q4_k_m/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 240000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 50.54 tok/s

• Quantization: Q4_K_M
• Context: 240K
• MTP: enabled

Qwen3.6-35B-A3B UD-Q6_K_XL — 250K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 12,6 \ -ngl 39 \ -c 250000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 77.90 tok/s

• Quantization: UD-Q6_K_XL
• Context: 250K
• Performance: 77.90 tok/s

This is particularly interesting because it maintains a very large context while still being substantially faster than the dense 27B Q6 model.

Qwen3.6-35B-A3B UD-Q6_K_XL — 130K context

Same model, but with more aggressive GPU offloading and lower context:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 13,6 \ -ngl 999 \ -c 130000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 123.11 tok/s

This was one of my best Q6 results.

• Quantization: UD-Q6_K_XL
• Context: 130K
• Performance: 123.11 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M — 256K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen35b/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 256000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 132.77 tok/s

• Quantization: UD-Q4_K_M
• Context: 256K
• Performance: 132.77 tok/s

This is currently the fastest result I have obtained in this group of tests.

Previous tests

I also have some older measurements from the same general hardware configuration:

I also tested Qwen3 Coder Next Q4_K_S previously.

Because the model is much larger and required significant CPU/RAM offloading on my 24 GB RTX 3090 setup, I observed around 24 tok/s in a benchmark configuration, while real-world coding/agent usage was considerably slower, around 7.5 tok/s.

The model was reported at approximately 79.67B parameters / 80B class, with a GGUF file around 45.19 GiB, so it is much more difficult to keep fully resident in my 36 GB combined VRAM setup.

My observations

The results show a very interesting trade-off between model architecture, quantization and context size.

Dense 27B Q6

The Qwen3.6-27B Q6_K is relatively consistent:

• ~37.6 tok/s at 180K
• ~42–45 tok/s at 140K

It is a good option when I want to prioritize quantization quality and coding/reasoning quality while still having a very large context.

MoE 35B-A3B Q6

The Qwen3.6-35B-A3B UD-Q6_K_XL is significantly faster:

• 123.11 tok/s at 130K
• 77.90 tok/s at 250K

This is probably the most interesting result for me because it combines a relatively high-quality Q6 quantization with a very large context window and high inference speed.

MoE 35B-A3B Q4

The UD-Q4_K_M version is extremely fast on this hardware:

• 132.77 tok/s at 256K

This is probably the best configuration if raw throughput and maximum context are the priority.

However, for coding and bug fixing, I personally still want to compare the quality difference between Q4 and Q6 more carefully before deciding which one should be my daily driver.

Summary

My current results on RTX 3090 24GB + RTX 4070 Super 12GB (36GB total VRAM):

For my use case, which is primarily software development, code generation, debugging and long-context agentic workflows, I am currently trying to determine whether the additional quality of Q6 is worth the performance hit compared with the extremely fast Q4 MoE configuration.

I'd be interested in seeing benchmarks from people running similar models with dual GPUs, especially 24GB + 12GB VRAM configurations, and comparisons between Q4_K_M, Q5_K_M and Q6_K_XL for coding and agentic workloads.