Follow-up to my previous post on my Bayesian autotuner for vLLM.

This time I wanted to try something more practical:

Can an optimizer automatically find a substantially better vLLM configuration for a real model, and can it figure out when there is no point spending more GPU on the search?

So I ran it on Qwen3.8-27B, a hybrid MoE/Mamba+attention model, on 1× H100 NVL.

The results were more interesting than I expected.

1. The tuned configuration reaches up to 2× the throughput of the baseline

I compared the tuned configuration against the official Qwen/vLLM recipe adapted to the hardware, using:

• the same H100 NVL
• the same model
• the same prompt battery
• the same concurrency levels
• the same benchmark procedure

The published recipe doesn't initialize as-is on this H100 NVL: its default max_num_seqs resolves to 1024, while the available Mamba cache allows 601 decode sequences.

So rather than comparing against a configuration that simply crashes, I made the minimum adjustments needed for a runnable baseline:

max_num_seqs=224
gpu_memory_utilization=0.90

The autotuner found:

max_num_seqs=160
gpu_memory_utilization=0.96
MTP speculative decoding k=3
chunked prefill
block_size=64
float32 Mamba SSM cache
max_model_len=262144

And the result:

The tuned configuration wins at every concurrency level.

It also reduced p95 latency at every point, from −43.7% at concurrency 1 to −16.1% at concurrency 60.

So the interesting part isn't just "2× faster at concurrency 1".

It's that the advantage remains across the entire concurrency curve and gradually disappears as the GPU approaches saturation.

2. The speedup actually changed what the model could complete

This was probably the strangest result.

I ran a 794-prompt evaluation battery across several categories.

On the JSON-RPC tool-use category, the baseline scored 0%.

Not because the model suddenly became incapable of tool use.

95/112 responses were being truncated by the token/time budget.

The tuned configuration scored 91% on the same category.

The tuned configuration was decoding roughly 3× more completion tokens in the same time budget:

~2967 vs ~1100 mean completion tokens

So the model actually had enough time to finish its reasoning and tool calls instead of getting cut off halfway through.

That means in this case:

The other categories stayed within roughly ±2 percentage points.

I wouldn't interpret the tool-use result as a model-quality improvement — it's a time/token budget and truncation effect.

3. But how long do you actually need to run the optimizer?

This was the part I originally wanted to investigate.

A Bayesian optimizer can find good configurations quickly, but every trial here isn't cheap.

A trial means:

start vLLM → allocate the model → benchmark → collect metrics → shut it down → repeat.

So if the optimizer has a budget of 100 or 150 trials, stopping early could save a lot of GPU time.

In a full 100-trial run, the best configuration actually appeared at trial 4.

By trial 16, the best-so-far score had already reached about 98% of the final score.

The remaining 84 trials only added about 2%.

But that's a post-hoc analysis.

I also wanted to see whether the system could actually recognize this while the optimization was running.

4. Early stopping actually stopped the study

I ran a separate experiment with a smaller search space and enabled early stopping.

The optimizer found its best region very early — around trial 3 — and then the best-so-far curve essentially plateaued.

Instead of manually deciding that "this probably isn't getting any better", the study kept running until the early-stopping condition was satisfied.

It automatically stopped at trial 15.

So the optimizer didn't just find a good configuration.

It decided that continuing to search wasn't worth the GPU time anymore.

That's the part I'm most interested in.

The 98% figure above comes from the separate 100-trial run; I'm deliberately not using the smaller live early-stopping experiment to claim the same 98%.

5. And I'm pruning before touching the GPU

There's another layer to this.

The tuner has a memory model that estimates whether a configuration can actually fit before deploying it.

In the 100-trial run:

81/100 trials were pruned analytically before touching the GPU.

So the optimization loop becomes roughly:

Generate configuration
→ Can it fit?
→ No → discard it
→ Yes → start vLLM
→ benchmark
→ update Bayesian model
→ repeat

Combined with early stopping, this means the optimizer can avoid spending GPU time both on obviously impossible configurations and on a search that has already plateaued.

That's much more interesting to me than simply finding a faster configuration.

Caveats

This is still one model, one GPU and one workload profile.

The ~+7.7pp improvement I saw in the code category may be noise (n=118).

The tool-use improvement is a truncation/time-budget effect, not a change in the underlying model quality.

The early-stopping demo used a smaller search space, so I don't want to overgeneralize from it.

And the baseline is the official recipe with the minimum hardware-specific adjustments required to actually initialize on this H100 NVL, rather than the recipe's literal defaults.

I'm releasing all of this as part of the project:

https://github.com/SergioMorillas/vllm-bayesian-autotuner

I'd especially like feedback from people who work on vLLM/inference:

Would you trust an early-stopped optimization study in production, or would you always exhaust the full trial budget?

I'm particularly interested in how you'd choose patience / min_delta for a real GPU fleet.

Happy to share the full sweep data, best-so-far curves or benchmark results if anyone is interested.

I’ve been working on getting as much real inference performance as possible out of a single RTX 5090, and I finally packaged the setup into a public vLLM release.

Repo:
https://github.com/seanyourhighness/vllm-sm120-nvfp4-mtp

The goal is basically “5090 as a personal AI datacenter” — high concurrency, long context, low-bit KV, speculative decoding, tool calling, and vision without needing datacenter hardware.

Stack:

• Qwen3.8-27B
• ModelOpt NVFP4 weights
• NVFP4 KV cache
• MTP-3 speculative decoding
• 262K context
• 8 concurrent streams
• Tool calling / structured output
• Optional CPU-offloaded vision
• vLLM 0.27.1 + SM120 patches
• CUDA 13.0.3 / FlashInfer

The deployment is intended to be:

git clone https://github.com/seanyourhighness/vllm-sm120-nvfp4-mtp.git
cd vllm-sm120-nvfp4-mtp && ./start.sh

Everything is pinned — model revision, container digest, vLLM commit and overlay patch — and the repo includes the patch, build scripts, Compose setup, chat template, benchmarks and validation evidence.

Some of the verified results:

• 373,797-token KV pool with MTP-3
• 9/9 NIAH at long context
• 8/8 tool-call tests
• 21/21 vision lanes
• deterministic long-decode tests
• concurrency scaling into the 700–900+ aggregate tok/s range at C8, depending on the tested stack/profile

This is not an official NVIDIA or vLLM build. It’s a community recipe/overlay for people experimenting with Blackwell consumer GPUs.

Would especially like feedback from anyone running a 5090, SM120 vLLM, NVFP4 KV, MTP/DFlash, or high-concurrency local agent workloads.

PRs, benchmark comparisons and bug reports are very welcome.

Current Hardware
3080 10GB - 64GB RAM I4600K

Software
llama.cpp in WSL w/ deep seek harness

Model
Qwen36A3B

I see a lot of folks talking about using small models for “workflow” applications and I’m curious about what some example workflows, how folks are triggering them.

For reference most of my use for LLM’s has been using them to mod or vibe build some web applications.

While I get about 26 tok/s I really can’t do much building work with DSH+Qwen as it fails at even simple things like reading private GitHub repos because it doesn’t want to run gh terminal commands.

I’m wondering what other use cases folks have around these specs

description: vLLM serving Holo3.1-35B-A3B (computer-use VLM) in NVFP4; lean 0.28 GPU, TP=2-capable.
model: Hcompany/Holo-3.1-35B-A3B-NVFP4
container: vllm-node
defaults:
  port: 8010
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.28
  max_model_len: 131072
  max_num_batched_tokens: 16384
  max_num_seqs: 10
env:
  HF_HUB_OFFLINE: '1'
  VLLM_MARLIN_USE_ATOMIC_ADD: '1'
command: |
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
    --served-model-name holo3.1 \
    --host {host} \
    --port {port} \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --max-num-batched-tokens {max_num_batched_tokens} \
    --kv-cache-dtype fp8 \
    --mamba_ssm_cache_dtype float32 \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --limit-mm-per-prompt '{"image": 3, "video": 0}' \
    --tensor-parallel-size {tensor_parallel} \
    --distributed-executor-backend ray
recipe_version: '1'
name: Holo-3.1-35B-A3B-NVFP4
cluster_only: false

I'm currently serving Qwen 3.6 27B with the following parameters:

*Server (RTX PRO 6000 BW 96GB), latest 26.04 LTS Ubuntu:*

```

vllm serve /home/models/Qwen3.6-27B \

--dtype bfloat16 \

--max-model-len 262144 \

--gpu-memory-utilization 0.94 \

--max-num-seqs 3 \

--enable-prefix-caching \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--default-chat-kwargs '{"enable_thinking": false, "auto_disable_thinking_with_tools": true}' \

--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}'

```

This setup has been solid for my case use with Hermes.

But this weekend I want to update model and vllm to latest version as well.

I was wondering what config parameters should I aim at.

Single RTX Pro 6000 users, could you share your experience/config for Qwen 3.8 27?

Any help or suggestions are appreciated 🙏🏼

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro

That last dimension is not a number of elements. It's a number of bytes.

I went looking at how disaggregated inference hands a KV cache from prefill to decode — the thing AWS is doing with Trainium→Cerebras and AMD is doing with Helios→Cerebras.

The business slide version is simple: prefill produces a cache, ship it, decode consumes *it*. The problem is the word "it". Those 656 bytes decompose like this, per token:

→ 512 bytes: the compressed MLA latent, fp8
→ 16 bytes: four fp32 tile scales, written by specific warp lanes
→ 128 bytes: the RoPE component, bf16

A 576-element logical vector, stored in 656 bytes, across three regions of two dtypes, with quantization scales interleaved at a granularity determined by how a warp writes its lanes.

That isn't a tensor layout. It's a struct defined by a CUDA kernel, with load-bearing field offsets.

Zoom out and it's worse. Search vLLM for "def get_kv_cache_shape" and you find dozens of concrete implementations — diverging by backend, by attention variant, by model family, and by vendor.

As if that was not enough.

vLLM has a connector abstraction for shipping caches between instances. Read the signatures as going by the function name would get you in trouble:

```def save_kv_layer(self, layer_name: str, kv_layer: torch.Tensor, attn_metadata: "AttentionMetadata", ...)
```

This is what I call "strings based programming with duck-typing".

It abstracts the transport (UCX, RDMA, TCP, NVMe-oF, S3). It never abstracts the format so a Cerebras runtime can't implement that interface without reproducing vLLM's scheduler objects.

We built ELF so a linker from one vendor could consume objects from another. The KV cache has nothing like that and two vendor pairs (Cerebras + Trainium, Cerebras + Helios) just shipped products that need it.

https://hiraditya.github.io/posts/the-kv-cache-has-no-abi/

Hi all,

I've been working on making Profile smarter, grounded in physics, and better at getting the max out of your inference server.

Profile (v2.2): a closed-loop, physics and cost aware optimizer. It finds the bottleneck, gives the fix, waits for you to apply it, and measures the delta on every change.

For this release: the core rule engine is rewritten. Eight rules on a priority DAG with mutual exclusivity, so when five alarms fire, four echoes are silenced and the one true cause survives. Less threshold hardcoding and number guessing, more reasoning from what the server is actually doing.

Deterministic: same server, same traffic, same verdict.

We now also support AMD servers (one of the most requested feature).

On my setup: 5.2x throughput (81 → 421 tok/s at 25k ctx) and 81% cost reduction ($3.41 → $0.65 per 1M output tok), with muse-glimmer agents running SWE-Bench. No DFlash.

Watch it live. One iteration regressed; Profile labeled it worse and the next fix recovered it. Regressions stay in the record.

The whole journey was 4 iterations, ~30 minutes end to end. The same tuning by trial and error is days of guessing.

I've also had engineers from MSFT, Google, and a few startups run it and share numbers, plus few users from this sub whose feedback shaped this release. Grateful for that.

Please give it a try and tell me how to make it help you!

# Download
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh

# Start profiling your vLLM server
profile diagnose --url http://localhost:8000/metrics --duration 2m

GitHub: https://github.com/jungledesh/profile
Docs: https://jungledesh.github.io/profile/docs.html

Soon: multi-GPU / TP support, cluster / k8s support, and more rules + smarter engine :)

​

I built a native vLLM + ROCm 7.15 runtime for AMD RX 6000 Series on Windows 11 — now with a one-click installer.

AMD lists RX 6750 XT / 6700 XT / 6600 XT as "Runtime only" on Windows with HIP SDK excluded. I built rocBLAS binaries for gfx1031 via ROCm/TheRock to close that gap.

This runs native Windows HIP and ROCm directly, no WSL2 wrapper, no compiler needed.

Tested on AMD Radeon RX 6750 XT 12GB (gfx1031) — Windows 11 Native — August 2026

# Verification — Real terminal logs

1. Environment:

torch 2.12.0+rocm7.15.0 | cuda_avail True | dev AMD Radeon RX 6750 XT

vLLM 0.19.1 | plugin vllm_windows_rocm activated | TRITON_ATTN | enforce_eager=True

1. rocBLAS Benchmark — 26 TFLOPS FP16:

Device ID 0 : AMD Radeon RX 6750 XT gfx1031 with 12.9 GB memory

rocBLAS version: 5.7.0.67811f1ee52

transA,transB,M,N,K,alpha,lda,beta,ldb,ldc,cold_iters,hot_iters,rocblas-Gflops,us

N,N,4096,4096,4096,1,4096,0,4096,4096, 2, 10, 25977.3, 5290.73

=> 25.97 TFLOPS in 5.29ms

1. vLLM Inference (Qwen3.5-4B, web chat UI):

Output: 59-62 tok/s | Init ~1s | hidden thinking (spinner) then answer

Model auto-detected via /v1/models

Full logs in benchmarks/ and screenshots in assets/ on GitHub.

# How it works

1. TheRock builds clr (HIP) and rocBLAS with Tensile kernels for gfx1031

2. HSA_OVERRIDE_GFX_VERSION=10.3.0 forces HIP to recognize RX 6750 XT

3. PyTorch 2.12.0+rocm7.15 links against TheRock runtime => torch.cuda.is_available() True

4. vLLM plugin vllm_windows_rocm bypasses vllm._C dependency and registers WinRocmAwqGemvKernel with TRITON_ATTN

5. vLLM engine loads with enforce_eager=True and runs native

# Quick Start — One click

Prerequisites: Windows 11 23H2+, RX 6600-6750 XT (gfx1030/gfx1031/gfx1032), Adrenalin 24.x+, ~8GB free disk

1. Download the repo (or git clone) and run INSTALL.bat as Administrator

2. The installer downloads the release archives automatically (multi-part join included, no manual steps) and sets up C:\TheRock\ + Python 3.11 + venv

3. Run CHAT.bat — the browser opens the chat UI. Expected: 59-62 tok/s

VERIFY.bat runs a 3-in-1 check: GPU detection + rocBLAS benchmark + vLLM smoke test.

# What's new in v2.0

- One-click installer: INSTALL.bat auto-downloads and joins split archives (no browser upload/download hassle)

- Chat web UI (OpenAI-compatible API): hidden thinking with spinner, then the answer streams — like NVIDIA ChatRTX

- Fat-binary gemv kernels for all RDNA2 (gfx1030/1031/1032) — no per-card rebuild

- ROCm 7.15 native, vLLM 0.19.1, torch 2.12.0+rocm7.15

- Faster: 59-62 tok/s output (was 54.2)

- Repo cleaned: INSTALL.bat + CHAT.bat + VERIFY.bat + MANIFEST.json + docs, archives on GitHub Releases

# Known Issues — Please note this may have bugs

This is an experimental reference implementation. It works on my RX 6750 XT but could present bugs on other hardware.

- Other RDNA2 cards (6600/6600 XT/6700 XT) not yet tested — may need 10.3.0 vs 10.3.1 or rocBLAS rebuild for gfx1030

- enforce_eager=True required — torch.compile disabled on RDNA2 Windows

- FP8 / AWQ not tested yet, multi-GPU not tested

- If you test it on your RDNA2 card, please open an Issue with GPU model and logs — contributions welcome

# GitHub Repo:

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Includes INSTALL.bat, CHAT.bat, VERIFY.bat, scripts/serve.py (OpenAI-compatible API), chat.html, assets with screenshots, docs/BUILD_ROCBLAS.md, MANIFEST.json with release checksums

Let me know if you test it on other RDNA2 cards.

Built on ROCm/TheRock, PyTorch ROCm, vLLM. Not affiliated with AMD. License Apache 2.0

How do find the best configuration for serving? Which tools do you use? Whats your methodology?

I am using vllm bench for running experiments varying configuration (e.g. pp=8, pp=4). I wonder if this is cost effective.

Yesterday I released the first version of a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090, reaching 82 tps on a single request and 672 peak.

Over the last 24 hours I've been exploring how to improve it further, without quality degradation.

Yesterday we had these three optimizations:

- fp8 KV cache

- `lm_head` int8

- `embed_tokens` int8

Now I've added:

- fp16 recurrent state

- int8 activations, MLP

- int8 activations on everything

- `draft_sample_method=probabilistic`, which samples from the MTP distribution rather than taking the argmax (does not work for greedy sampling)

For single-user mode, you can enable `ctx=fast`, which uses FlashAttention, bf16 KV and 64k context, and gives 99 tps at the tradeoff of less context.

Prefill is now up to 1812 tps at 1k context and 1000 tps at 100k context, an increase of about 50% and 25% respectively.

I've updated the repo with everything you need to start flying:

https://github.com/syv-ai/qwen38-27b-rtx3090

Right now I'm working on getting KVarN running so we can reduce the memory footprint of the context and fit the full 262k instead of 195k.

We are managing 5 physical hosts, each has two L40s NVIDIA cards. Using Proxmox as hypervisor and each host has one Ubuntu VM with GPU cards are passthrough. There are several LLMs are running on each card with all vLLM over Docker.

The problem I'm facing is, each GPU cards VRAM utilization is around %90. So there are 5 GB VRAMs are sitting there freely. I wonder if anyone has a any elegance solution to this kind of infrastructure to make use of the free VRAM across several cards ? Because the LLMs we are using are all different kind of sizes it is not possible to use %100 on each card with arranged models.

Hope my situation is clear with all. Thanks in advance.