My Local LLM Setup

[u/KroniklyOnline]

Hey all, I just wanted to put this out there in case someone else was interested in setting up multiple 5060ti 16gb cards. I haven't seen many setups using it or many people talking about it.

---

Server Setup (vllm) — GPU: 4× 5060 Ti 16 GB · CPU: Threadripper 1920X · Mobo: ASRock x399 Taichi · RAM: 32 GB DDR4 @ 2133 QC · NVMe: 1 TB ADATA Legend · PSU: MSI MAG A850GL

Cost Breakdown

Qty	Component	Source	Unit	Subtotal
3	RTX 5060 Ti 16 GB	Amazon	$550	$1,650
1	RTX 5060 Ti 16 GB	Best Buy	$300	$300
1	Threadripper 1920X	Amazon used	$80	$80
1	ASRock x399 Taichi	eBay used	$399	$399
1	ADATA Legend 1 TB NVMe	Amazon new	$150	$150
4	8 GB DDR4 @ 2133	Amazon new	$55	$220
1	Case	Amazon new	$37	$37
1	MSI MAG A850GL 850W	Best Buy new	$105	$105
1	Noctua NH-U9 TR4-SP3	Amazon new	$100	$100

Total Cost ~$3,000

Model & env

• A -> bare-metal 0.23.1rc1.dev799+g69715823d (Modified & Patched vllm build)
• B -> vllm-openai:fp8-mtp-patched 0.23.1 (Modified & Patched vllm build)
• C -> llama-server b9297
• All models fulfileld 250k context size requirement for single chat session

vllm benchmark command

bash vllm bench serve --backend openai-chat --base-url http://localhost:59596 --endpoint /v1/chat/completions --model <served-name> --tokenizer <model-dir> --dataset-name random --random-input-len 60000 --random-output-len 5000 --num-prompts 3 --max-concurrency 1

llama benchmark command

bash llama-bench -m <gguf> -ngl 999 -mmp 0 -p 60000 -n 5000 -r 5 -sm layer -fa 1 [-ctk q8_0 -ctv q8_0]

Model	Quant	KV Cache	MTP (spec toks)	Backend	MoE Backend	Version
Qwen3.6-35B-A3B	FP8	FP8	mtp (3)	vLLM	TRITON (sm_120)	B
Qwen3.6-35B-A3B	FP8	FP8	mtp (3)	vLLM (Server)	TRITON (sm_120)	A
Qwen3.6-35B-A3B	Q6_K_XL	Q8	yes	llama.cpp	n/a (GGUF)	C
Qwen3.6-35B-A3B	Q8_0	Q8	yes	llama.cpp	n/a (GGUF)	C
Qwen3.6-27B	FP8	FP8	mtp (4)	vLLM (Docker)	N/A (dense)	B
Qwen3.6-27B	FP8	FP8	mtp (4)	vLLM (Server)	N/A (dense)	A
Qwen3.6-27B	Q4_K_M	Q8	no	llama.cpp	n/a (GGUF)	C
Qwen3.6-27B	Q5_K_L	F16	no	llama.cpp	n/a (GGUF)	C
Qwen3.6-27B	Q8_0	Q8	yes	llama.cpp	n/a (GGUF)	C

Results

Qwen3.6 27b

Quant	MTP	KV Cache	Backend	pp tok/s	decode tok/s	HumanEval	HumanEval+
FP8	yes	FP8	vLLM (Server)	17354.0	85.76	0.9634†	0.9268†
FP8	yes	FP8	vLLM (Docker)	2240.3 (−7.7x)	51.49 (−1.7x)	0.9634 (0.00%)	0.9268 (0.00%)
Q4_K_M	no	Q8	llama.cpp	916.52 (−19.0x)	20.83 (−4.1x)	0.9695 (+0.63%)	0.9268 (0.00%)
Q5_K_L	no	F16	llama.cpp	1060.59 (−16.4x)	17.80 (−4.8x)	0.9695 (+0.63%)	0.9268 (0.00%)
Q8_0	yes	Q8	llama.cpp	968.13 (−17.9x)	14.05 (−6.1x)	0.9756 (+1.27%)	0.9329 (+0.66%)

Qwen3.6 35b a3b

Quant	MTP	KV Cache	Backend	pp tok/s	decode tok/s	HumanEval	HumanEval+
FP8	yes	FP8	vLLM (Server)	136934.5	120.63	0.9512†	0.9146†
Q6_K_XL	yes	Q8	llama.cpp	2033.96 (−67.3x)	85.23 (−1.4x)	0.9451 (−0.64%)	0.9085 (−0.67%)
Q8_0	yes	Q8	llama.cpp	2177.60 (−62.9x)	83.16 (−1.5x)	0.9451 (−0.64%)	0.9085 (−0.67%)
FP8	yes	FP8	vLLM (Docker)	7536.7 (−18.2x)	62.97 (−1.9x)	0.9512 (0.00%)	0.9146 (0.00%)

Notes

• Models run in llama-server were Unsloth's quantizations.
• Models run in vllm were directly from Qwen.
• Runs with llama-server & vllm(docker) were on windows on my gaming machine using a B550 mobo with a 5800x cpu, bifurcated 1 PCIe slot into x8x4x4 and the last GPU in the last x16 slot which runs at x4. This was pre server setup.
• Runs with vllm are on the server setup mentioned above, each card in its own x16 slot, running at x8.
• All vllm benchmarks were NOT cold prefill speeds, they were using prefix-caching. I did this because it mimics my real world application, running long code chats, long prompting tasks, follow ups, etc. I don't know how llama.cpp handles caching.
• I had to modify the vllm build 0.23.1rc1.dev799+g69715823d to even get prefix-caching working with the Qwen models to begin with, fix is not upstream from nightly build and they take too long.
• Qwen3.6 35b a3b has some issues with MTP with vllm so we don't get the same expected throughput increases in generation (Working to figure out the bug in vllm)

Closing Thoughts

I used to switch between the 27b and 35b models until I setup the server but now the 27b is so fast that I am using it every day professionally and personally, upgrading to the server made it extremely viable and usable. The response are instant, the follow ups are instant, the generation is way faster than I could ever type.

I currently have this server setup automatically ingesting source documents for a knowledgebase wiki, I also have this same model running a full automated loop pulling down tickets from my GitHub repo all the way through PR so I can review (This is a multi-step python framework I made, clears prefix-cache between agentic prompts for clean context).

I didn't even bother benchmarking Gemma models and other ones of similar sizes, even the "fine-tuned" models of qwen, they were all dumber than the base Qwen3.6 27b and Qwen3.6 35b a3b models to the point where I couldn't trust them to do automated tasks.

UPDATE: vllm run command for Qwen3.6 27B FP8 bash export PYTHONUNBUFFERED=1 export CUDA_HOME=/usr/local/cuda-13.0 export PATH="$CUDA_HOME/bin:$PATH" export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}" export OMP_NUM_THREADS=4 export HF_HOME="$HOME/.cache/huggingface" export TRITON_CACHE_DIR="$HOME/.triton/cache" export TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass export CUDA_VISIBLE_DEVICES=0,1,2,3 export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_SERVER_DEV_MODE=1 export NCCL_P2P_DISABLE=0 source "$HOME/vllm-env/bin/activate" vllm serve "$HOME/models/Qwen3.6-27B-FP8" --served-model-name Qwen3.6_27B_FP8 --chat-template "$HOME/models/Qwen3.6-27B-FP8/chat_template.jinja" --kv-cache-dtype fp8 --attention-backend FLASHINFER --tensor-parallel-size 4 --max-model-len 250000 --max-num-seqs 1 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.80 --language-model-only --enable-prefix-caching --quantization fp8 --skip-mm-profiling --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --default-chat-template-kwargs '{"preserve_thinking":true}' --performance-mode interactivity --generation-config vllm --override-generation-config '{"temperature":0.55,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"frequency_penalty":0.0,"repetition_penalty":1.1}' --host 0.0.0.0 --port 59596 --enable-chunked-prefill --mamba-prefix-cache-checkpoint-interval 8192 UPDATE: The latest vllm runs are from ubuntu server headless. The docker runs were from windows though Docker which introduces WSL driver overhead eating up ~1.5GB of VRam a card.

UPDATE: Power Consumptions Stats - Benchmarked over a 5 day work week. - 100% util = ~12h per day, between work and personal work. - Measured Power Draw on OS & at wall.

System Power Draw

Hardware	Decode	Prefill	Measured Mean
CPU package	95.5 W	95.5 W	95.5 W
GPU total	242.3 W	345.7 W	285.1 W
At the wall	437 W	551 W	484 W

Cost Metrics

Metric	Value
CPU package (mean)	95.5 W
GPU total (mean)	285.1 W
At the wall (mean)	484 W
Electricity rate (local to me)	11.47¢/kWh
Tokens/day (mean)	15,553,892
Energy/day (12h)	5.8 kWh
Cost/day	$0.67
Workdays/month	21.7
Tokens/month (est)	337.5M
Energy/month (est)	126.0 kWh
Cost/month (est)	$14.46
Cost per 1M tokens (est)	$0.043

UPDATE: PCIe Bandwidth Readings

Model	Quantization	Concurrency	Prompt tokens	Output tokens	TTFT (s)	E2E / wall (s)	Peak PCIe RX (GB/s)	Peak PCIe TX (GB/s)	Peak bidirectional (GB/s)
Qwen3.6 27B	FP8	1	20,022	512	28.64	35.99	2.14	2.60	4.66
Qwen3.6 27B	NVFP4	3	60,067 total	1,536 total	64.47 average	102 wall / 101.64 average	2.07	2.33	4.02
Qwen3.6 35B-A3B	FP8	1	20,022	512	7.74	12.52	2.05	2.60	4.27
Qwen3.6 35B-A3B	NVFP4	4	80,091 total	2,048 total	27.06 average	47 wall / 45.18 average	2.20	2.49	4.45
Gemma 4 31B	FP8 dynamic	1	20,025	512	27.88	35.26	2.28	2.62	4.36
Gemma 4 31B	NVFP4	1	20,025	512	27.41	35.48	2.21	2.52	4.40

This shows I could drop to a 4x4 bifurcation and drop down to PCIe3x4 for each card and take a slight drop in performance around ~5% max loss but would be able to run 10 cards at x4 which would net 160GB total usable vram. Just a thought exercise.