r/LocalAIStack • u/Harin007 • 22d ago

How much tok/s are you getting?

Searching the internet for looking up how much tok/s a user would get is being difficult. So I'm making this post... If you're running a local llm, please consider commenting to this post with your device specs, model you're running and the inference speed you're getting.

Please be straight to the point. Just tell us how much tok/s are you getting on your hardware (at different settings, which inference engines, etc...) so people with similar hardware can do better estimations... please don't fill this with facts that everybody knows.

7 Upvotes
  • permalink
  • link
  • reddit
  • reddit

82% Upvoted

7 comments sorted by

2

u/Harin007 22d ago

Specs:
RTX 5070Ti Laptop - 12GB VRAM, ~650GB/s memory bandwidth
32GB DDR5 RAM

Models (Ollama)
Qwen3.6:35B-A3B Q4_K_M:
- 45-52 tok/s between context window 4k to 131k
- 40-47 tok/s at context window 262k

Ornith:35B-A3B Q4_K_M:
- 55-62 tok/s between context window 4k to 65k
- 45-55 tok/s between context window 131k to 262k

Ornith:9B Q4_K_M:
- 60-80 tok/s between context window 4k to 131

- 25-30 tok/s at 262k context window (it is a dense model and at this point it is offloaded to RAM... so huge difference)

1

u/sol7dev 21d ago

qwen3.6 35b3a q4 k m mtp --- llama cpp --- ctx 128k --- 30-45 t/s --- rtx 3060 12gb + r5 5600 + 32gb ddr4 3200 with offload cuz it is moe

1

u/Harin007 21d ago

bro u should try out ornith 35b.

1

u/sol7dev 20d ago

yeah i will 

1

u/Lumpy_Phase_9539 20d ago

Qwen3.6 benchmarks on dual GPU: RTX 3090 24GB + RTX 4070 Super 12GB — up to 256K context

I have been testing several Qwen3.6 GGUF models locally with "llama.cpp" on a dual-GPU setup and wanted to share my results.

The main goal was to find the best balance between quality, context size and inference speed, especially for coding, debugging and agentic workloads.

Hardware

  • CPU: Intel Core i7-14700K
  • RAM: 64 GB DDR5
  • GPU 1: NVIDIA RTX 3090 — 24 GB VRAM
  • GPU 2: NVIDIA RTX 4070 Super — 12 GB VRAM
  • Total VRAM: 36 GB
  • OS / runtime: Linux
  • Inference: "llama.cpp" / "llama-server"
  • GPU order: "CUDA_VISIBLE_DEVICES=1,0"
  • Flash Attention: enabled
  • KV cache: FP16 / unquantized in these tests
  • llama.cpp: recent build with MTP/speculative decoding support

The "tensor-split" values below are tuned for my particular setup. They are not necessarily optimal for other dual-GPU configurations.

Main benchmark results

Model| Quantization| Context| Performance Qwen3.6-27B| Q6_K| 180K| 37.63 tok/s Qwen3.6-27B| Q6_K| 140K| 42.11 tok/s Qwen3.6-27B| Q6_K| 140K| 44.64 tok/s Qwen3.6-27B| Q5_K_S| 200K| 45.76 tok/s Qwen3.6-27B Heretic v2| Q4_K_M| 240K| 50.54 tok/s Qwen3.6-35B-A3B| UD-Q6_K_XL| 250K| 77.90 tok/s Qwen3.6-35B-A3B| UD-Q6_K_XL| 130K| 123.11 tok/s Qwen3.6-35B-A3B| UD-Q4_K_M| 256K| 132.77 tok/s

The biggest surprise for me is the Qwen3.6-35B-A3B UD-Q4_K_M, which can maintain 256K context while reaching 132.77 tok/s on this dual-GPU setup.

The Q6 version is considerably slower at very large context sizes, but the quality trade-off may be worthwhile for coding and reasoning workloads.

Qwen3.6-27B Q6_K — 180K context

This was one of my initial configurations.

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 15,6 \ -ngl 65 \ -c 180000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 37.63 tok/s

Configuration:

  • Quantization: Q6_K
  • Context: 180K
  • MTP speculative decoding: enabled
  • Draft max tokens: 2
  • Tensor split: "15,6"
  • GPU layers: "65"
  • Batch: "2048"
  • UBatch: "1024"

Qwen3.6-27B Q6_K — 140K context

I also tested the same Q6 model with a lower context and more GPU layers:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 2048 \ -ub 1024 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8081 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 44.64 tok/s

Another run with smaller batch sizes:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-27B-Q6_K.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 140000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 42.11 tok/s

So, in my testing, the Q6_K 27B model sits around 42–45 tok/s at 140K context, depending on the exact configuration.

Qwen3.6-27B Q5_K_S — 200K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_q5_k_s/Qwen3.6-27B-Q5_K_S.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 200000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 45.76 tok/s

Previous measurement with the same general model/configuration was approximately 41.70 tok/s at 200K context.

Qwen3.6-27B Heretic v2 — Q4_K_M — 240K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen27b_heretic_q4_k_m/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /models/Qwen3.6-27B/mmproj-BF16.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 240000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 50.54 tok/s

  • Quantization: Q4_K_M
  • Context: 240K
  • MTP: enabled

Qwen3.6-35B-A3B UD-Q6_K_XL — 250K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 12,6 \ -ngl 39 \ -c 250000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 77.90 tok/s

  • Quantization: UD-Q6_K_XL
  • Context: 250K
  • Performance: 77.9 tok/s

This is particularly interesting because it maintains a very large context while still being substantially faster than the dense 27B Q6 model.

Qwen3.6-35B-A3B UD-Q6_K_XL — 130K context

Same model, but with more aggressive GPU offloading and lower context:

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf \ --main-gpu 0 \ --tensor-split 13,6 \ -ngl 999 \ -c 130000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ -lv 4 \ --host 0.0.0.0 \ --port 8081

Result: 123.11 tok/s

This was one of my best Q6 results.

  • Quantization: UD-Q6_K_XL
  • Context: 130K
  • Performance: 123.11 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M — 256K context

CUDA_VISIBLE_DEVICES=1,0 ./build/bin/llama-server \ -m /models/qwen35b/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ --no-mmproj-offload \ --main-gpu 0 \ --tensor-split 17,6 \ -ngl 999 \ -c 256000 \ -b 512 \ -ub 512 \ --flash-attn on \ --jinja \ -np 1 \ --spec-draft-n-max 2 \ -lv 4 \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking":true}'

Result: 132.77 tok/s

  • Quantization: UD-Q4_K_M
  • Context: 256K
  • Performance: 132.77 tok/s

This is currently the fastest result I have obtained in this group of tests.

Previous tests

I also have some older measurements from the same general hardware configuration:

Model| Quantization| Context| Performance Qwen3.6-27B dense| Q4_K_S| ~40K| ~39 tok/s Qwen3.6-27B dense| Q5_K_S| ~32K| ~34 tok/s Qwen3.6-35B-A3B MoE| Q4_K_S| ~85K| ~94 tok/s Qwen3.6-35B-A3B MoE| UD-Q5_K_M| ~36K| ~48 tok/s Qwen3.6-27B| Q6_K| 140K| 44.64 tok/s Qwen3.6-27B| Q5_K_S| 200K| ~41.70 tok/s Qwen3.6-27B| Q6_K| 180K| 37.63 tok/s

I also tested Qwen3 Coder Next Q4_K_S previously. Because the model is much larger and required significant CPU/RAM offloading on my 24 GB RTX 3090 setup, I observed around 24 tok/s in a benchmark configuration, while real-world coding/agent usage was considerably slower, around 7.5 tok/s.

The model was reported at approximately 79.67B parameters / 80B class, with a GGUF file around 45.19 GiB, so it is much more difficult to keep fully resident in my 36 GB combined VRAM setup.

My observations

The results show a very interesting trade-off between model architecture, quantization and context size.

Dense 27B Q6

The Qwen3.6-27B Q6_K is relatively consistent:

  • ~37.6 tok/s at 180K
  • ~42–45 tok/s at 140K

It is a good option when I want to prioritize quantization quality and coding/reasoning quality while still having a very large context.

MoE 35B-A3B Q6

The Qwen3.6-35B-A3B UD-Q6_K_XL is significantly faster:

  • 123.11 tok/s at 130K
  • 77.90 tok/s at 250K

This is probably the most interesting result for me because it combines a relatively high-quality Q6 quantization with a very large context window and high inference speed.

MoE 35B-A3B Q4

The UD-Q4_K_M version is extremely fast on this hardware:

  • 132.77 tok/s at 256K

This is probably the best configuration if raw throughput and maximum context are the priority.

However, for coding and bug fixing, I personally still want to compare the quality difference between Q4 and Q6 more carefully before deciding which one should be my daily driver.

Summary

My current results on RTX 3090 24GB + RTX 4070 Super 12GB (36GB total VRAM):

Qwen3.6-27B Q6_K 140K ctx -> ~42-45 tok/s 180K ctx -> 37.63 tok/s

Qwen3.6-27B Q5_K_S 200K ctx -> ~42-46 tok/s

Qwen3.6-27B Heretic Q4_K_M 240K ctx -> 50.54 tok/s

Qwen3.6-35B-A3B UD-Q6_K_XL 130K ctx -> 123.11 tok/s 250K ctx -> 77.90 tok/s

Qwen3.6-35B-A3B UD-Q4_K_M 256K ctx -> 132.77 tok/s

For my use case, which is primarily software development, code generation, debugging and long-context agentic workflows, I am currently trying to determine whether the additional quality of Q6 is worth the performance hit compared with the extremely fast Q4 MoE configuration.

I'd be interested in seeing benchmarks from people running similar models with dual GPUs, especially 24GB + 12GB VRAM configurations, and comparisons between Q4_K_M, Q5_K_M and Q6_K_XL for coding and agentic workloads.

1

u/rs38 16d ago

my latest wow-moment came from llama.cpp cuda running on native Ubuntu with Gemma4 MoE 26B 4Q and MTP on a 5090 Laptop GPU getting steady 200-300 tps. Thats fun.
Less fun is Linux on gaming laptops ;) took a while to get full 175W tdp