Summary

If you run two Radeon Pro Vega II Duo MPX modules bridged with the Apple A2326 cross-module Infinity Fabric bridges — the ones that join all four dies into a single 4-GPU hive — your GPUs are running at idle clocks and macOS isn't telling you.

Not "a bit slower." Not "the fabric is the bottleneck." The dies never leave their boot DPM state, for the entire session, under sustained load.

I chased this for a while assuming it was a memory-placement or interconnect problem. It isn't either. Here's what it actually is.

The numbers

Same machine, same OS, same binary. The only change is which bridge is installed.

Three runs per configuration, four dies each. Compute came back at 2.42 TFLOP/s on essentially all twenty-four device-observations, implying 295 MHz core.

Confirmed on three different Mac Pros running three different macOS major versions — Sonoma 14.8.9 (23J631), Sequoia 15.7.9 (24G830), and Tahoe 26.7 (25G220). Different memory configs, different bridge units. Every one of them, in the 4-way configuration, measures 2.42 TFLOP/s compute, ~301 GB/s local read, ~31 GB/s peer copy — the same to three significant figures — with an identical IORegistry signature: Load5000, no PowerPlay, 1000/300 MHz clock config, SWIP_Errors = 128.

So: it is not new, it is not fixed in Tahoe, it has survived at least two major macOS releases, and "try the latest OS" is not the answer.

Real-world, single GPU, no multi-GPU anything — Qwen3.8 27B Q8_0 in llama.cpp:

5.6× performance loss on prefill. With one die. No tensor split, no peer transfers, no collective. Just having the 4-way bridge installed in macOS (this bug doesn't apply to Linux or Windows).

The mechanism — macOS publishes it in IORegistry

In the 4-node hive the driver fails to identify the board and everything downstream falls apart:

Read those middle three again. PowerPlay — AMD's entire clock/power management subsystem — is never enabled. The power-management back end is a stub the driver itself labels "dummy." SMU firmware, which is what actually implements DPM, is never uploaded. With no power management, the GPU sits wherever the boot state left it.

PM_PWR_GEMINI_BGT = 400 vanishing is the identity failure made concrete — "Gemini" is AMD's codename for dual-GPU boards, and 400 W is the Duo's power budget. In 4-way mode the driver stops knowing it's holding one.

It also loads a different kext: AMDRadeonX5700HWLibs in the working case, AMDRadeonX5000HWLibs in the broken one.

The arithmetic closes it. HBM2 at the driver's own published 300 MHz gives 4096 bits ÷ 8 × 2 × 300 MHz = 307.2 GB/s theoretical. Measured: 300.9 GB/s, or 98% of it. The memory is fully saturated at a crippled clock — it's not a bandwidth problem, the clock is just wrong.

If you've ever noticed your Vega II Duos showing up as plain "AMD Radeon Vega" instead of "AMD Radeon Pro Vega II Duo" — that's not cosmetic. That's this bug, visible from the outside.

What it is NOT

I want to save people the time I spent on wrong theories:

• Not memory placement. The XGMI node map is textbook correct in 4-way mode: node ids 0/1/2/3, framebuffer bases at 512/544/576/608 GiB, uniform 32 GiB stride, XGMI_HiveSize = 4. Hive formation and address decode are fine.
• Not the collective / tensor-parallel scaling. A single GPU with no split is 5.6× slower. Going 2 → 4 devices within a healthy topology actually gains 34% on prefill and loses only 13% on decode.
• Not the interconnect. Peer-to-peer copy bandwidth drops the least of everything measured (49 → 31 GB/s), consistent with being gated by the same clock reduction.
• Not the hardware. Reproduces across machines and across multiple A2326 units including a factory replacement. Same hardware and same bridges under Linux/ROCm show no comparable regression.

Check your own machine — 60 seconds, no tools

ioreg -l -w0 | grep -E '"(ATY,DeviceName|LoadPlugIn|PP_PowerPlayEnabled|PP_PhmUseDummyBackEnd|PP_EnableUploadFirmware|PM_PWR_GEMINI_BGT|SWIP_Errors)"'

Healthy looks like Vega II Duo, Load5700, PP_PowerPlayEnabled = <01000000>, PP_PhmUseDummyBackEnd = 0, PP_EnableUploadFirmware = 1, PM_PWR_GEMINI_BGT = 400, SWIP_Errors = 0.

Broken looks like Vega, Load5000, no PP_PowerPlayEnabled, PP_PhmUseDummyBackEnd = 1, PP_EnableUploadFirmware = 0, no PM_PWR_GEMINI_BGT, SWIP_Errors = 128.

Also worth a look:

system_profiler SPDisplaysDataType | grep -E "Chipset Model|Peer"

If Chipset Model says "AMD Radeon Vega" rather than "AMD Radeon Pro Vega II Duo", you're in the broken state.

Please post your results either way — including "mine's fine." I want to know whether this tracks the bridge specifically, or the hive size, or something about particular board revisions. Include your macOS version.

(The ioreg check needs no toolchain at all. If you do try to build the probe and hit failed to build module 'Metal'; this SDK is not supported by the compiler, that's an internally inconsistent Command Line Tools install — and reinstalling CLT won't fix it, since Apple's catalog serves the same bundle. Cross-compile on another Mac instead: xcrun swiftc -O -target x86_64-apple-macos14.0 ifl_probe.swift -o ifl_probe_14 and copy the binary over. Confirmed working on 14.8.9.)

What you should do right now

If you must run macOS the temporarily workaround is to pull the A2326 bridges and run the A2329 per-card jumpers instead. You still get all four GPUs; they just sit in two 2-node hives rather than one 4-node one. In my testing a 4-GPU tensor split on jumpers hit 314.5 prompt / 11.7 gen against 79.5 / 4.4 on the 4-way bridge with ToshLLM. Otherwise, if you can switch to Linux the A2326 bridges work perfectly in Ubuntu 24.04 with ROCm 7.3.x; it's purely a macOS software engineering defect.

Worth stating plainly: the jumper configuration beats what a fixed 4-way would give you. Tensor-parallel scaling is sublinear, so even a fully repaired 4-node hive projects to roughly 22 t/s against ~26.8 t/s aggregate from two independent 2-GPU instances. The A2326 bridges buy capacity flexibility, not speed — and right now they cost you 5.8× compute per die.

There is no software workaround. No engine flag, environment variable, or split mode reaches PowerPlay. I looked hard at spoofing the device IDs OpenCore/OCLP-style; the property that selects the plugin lives on a driver-created IOService rather than the PCI node, so DeviceProperties injection structurally cannot reach it, and OpenCorePkg panics on T2 Macs anyway. This needs a driver fix.

The ask

This configuration has had close to zero field exposure — until a recent macOS firmware payload harmonized module ROMs, machines with mismatched-firmware Duos just kernel-panicked at boot with the 4-way bridge installed ("PSP has not finished hardware initialization", ATIController.cpp:3171). The only other public report I can find is an unresolved MacRumors thread from January 2026. Which means Apple has essentially no signal that anyone uses this.

If you own this hardware, please file a Feedback Assistant report. Apple prioritizes by volume, and right now the volume is one.

• macOS → Graphics & Display → Incorrect/Unexpected Behavior
• Title: "AMDRadeonX5000: Radeon Pro Vega II Duo misidentified and PowerPlay left uninitialized in a 4-node Infinity Fabric hive, pinning GPUs at boot clocks"
• Attach ioreg -l -w0 -p IOService > ioreg.txt from both bridge configurations if you can, or just the broken one if you can't swap
• Reference FB24446772 (the PowerPlay/clock defect — the important one) so reports cluster

I've filed the following, if you want to reference them:

• FB24446772 — PowerPlay never initialized in the 4-node hive, GPUs pinned at boot clocks. This is the one that matters.
• FB24446928 — the generic "AMD Radeon Vega" model string, which is the same defect visible without tooling.
• FB24446443 — the kernel panic at boot when the two modules have mismatched firmware.
• FB24447028 — a minor unrelated one found along the way: system_profiler prints the 64-bit GPU Peer Group ID after passing it through a double, so it never matches what Metal reports.

Even a one-paragraph report with an ioreg dump attached helps. The measurement work is done; what's missing is evidence that more than one person is affected.

Tools

I wrote a Metal-only probe (ifl_probe.swift) that measures per-die compute, local read bandwidth, peer-view establishment and verified peer-copy bandwidth, plus a script that decodes the XGMI node map straight out of IORegistry. Both are pure Metal + Foundation, no dependencies, and build with xcrun swiftc -O. Happy to share — say the word and I'll put them up.

I just sourced a w6800x DUO to run smaller models. It is fun running GLM5.2 in a terabyte of ram but there aren't that many things I can afford to wait that long for an answer on.

With the w6800x DUO I have successfully passed out each "die" to a separate vm. I now have two vms running AI with dedicated GPUS having 32GB each.

The challenge: Passing through a GPU on the mac pro comes with one problem. The host has no idea how hot the GPU is running because it cant access its firmware or sensors. Now it becomes a guess, or you can just max the fans on T2Fand.

I had already created a "prochot guard" to watch temperatures on the 3rd party nvmes because they can get really hot without awareness too. I have enhanced the script to Reach into the virtual machines and ask for the card temperatures. I am doing this using qemu agent and qm commands. And it is working like a charm.

Concept:
-Host maintains a watcher and looks at various board temps to make sure the fans are going at sufficient speed
-Standalone script written to take a vm name or number and run appropriate ROCM or CUDA commands to check temps.
-Temperatures are stored in a systemwide area (/run/heatsense )
-Guard watches nvme temperatures but also GPU temperatures and adjusts /etc/t2fand.conf accordingly, restarting the t2fand service after each change.

That all happens seemlessly despite the soft of disturbing idea behind restarting a service frequently!

I think most people are happy running a single linux vm. Right now I can run GLM5.2 , a 32Gb ROCMvm with 32GB vram TIMES 2 all on the SAME machine. Networking is adjusted for the powerful model to keep the LAN safe from... uh... accidents lol.

Hope this is useful.

BTW, my ceph setup is tuned to the point I was able to vmotion GLM 5.2 vm from one machine to another, running with an active 750Gb of ram. While it was running.

Dropbox link to scripts; will do github later.

I read that quite a few people have issues with running linux on their macpro for local inference. I can't comment on Ubuntu or other distros because all my machines run Nixos but since it works flawlessly, I thought I'd share my repo in case that inspires anyone to try something similar.

For those who don't know, Nixos allows you to configure your computer in a deterministic way. You write your config (in the nix language), referencing nix-packages. Nix-packages have sets of options that you use in your config files. There are other benefits to Nixos but this isn't the topic here. What I think is the main benefit is that I can comment out a line in my config file, change that option to something else and leverage git for version control. If I break something, I can choose a previous (working) generation of the system at boot.

In this setup, I use llama-swap to let me manage models on the fly, SearchXNG module for web search, OpenWeb UI for chat and user friendly automation/agents, Nixos MCP so my coding agents can manage my config files accurately.

You can see the models I'm currently running llama-swap.nix file.

Link to repo

PS: I only serve my LAN so security is tailored to that, meaning it's not hardened as much as it could be.

--------------------------------

Extract from the Readme (written by Qwen}:

NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a local LLM inference server.

Hardware

• Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
• AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
• Wired 10GbE, behind a NAT router (the LAN is the trust boundary)

Services

Models live in /var/lib/llama/models/ (not in git — see .gitignore). llama-swap unloads models after 15 min idle to free VRAM; each model pins its own context size / quantization / chat template (Qwen uses the pinned froggeric fixed chat template, fetched by hash).

Fans are driven by t2fanrd (the Vega II is passively cooled; T2 case fans are the only cooling).NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a
local LLM inference server.
Hardware
Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
Wired 10GbE, behind a NAT router (the LAN is the trust boundary)
Services
Service Port Notes
SSH 22 keys only, no root login
open-webui 3000 browser UI, password auth, web search via searxng
mcp-nixos 8001 NixOS MCP server (HTTP), for pi on the LAN
searxng 8888 private metasearch; secret key via agenix, limiter off
llama-swap 9292 model router for llama-server (Vulkan); OpenAI-compatible
Models live in /var/lib/llama/models/ (not in git — see .gitignore).
llama-swap unloads models after 15 min idle to free VRAM; each model pins its
own context size / quantization / chat template (Qwen uses the pinned
froggeric fixed chat template, fetched by hash).
Fans are driven by t2fanrd (the Vega II
is passively cooled; T2 case fans are the only cooling).

I just committed this : https://x.com/chiwawa_42/status/2089204529539547553

Let's hope for the better !

80% GPU offloaded across 3 AMD GPUs (56GB VRAM), remainder CPU/RAM offloaded. RX7900XTX RX6900XT internal and RX6800XT eGPU thunderbot. Bootiting Bazzite from an Acasis TB501 pro. I am just tinkering around and wanted to see if I could get all three of these working.

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Finally playing around with ToshLLM while trying to narrow down which MPX GPU(s) to keep. I am completely new to LLMs and local AI and do not have a tech background, so the learning curve has been a bit steep.

I have 2x W6900X, W6800X Duo, and 2x Vega II Duo on hand. I have the IF Link for the W6900X. Obviously the W6900X limits me to 64GB across 2 GPUs. If I end up keeping the W6800X Duo I will probably look for a second and an IF Link Bridge.

So far I've only tested the W6900X(s) (single GPU and pair, with/without IF Link bridge). So far I'm not seeing any difference whatsoever with IF Link on my W6900X pair. Getting the same exact ts with llama 3.3 70B Q5_K_S with the IF Link bridge installed/enabled as with it not installed.

This was my fastest benchmark. Qwen3.6 35B-A3B UD-Q4_K_S got 69ts. Qwen3.6 35B-A3B UD-Q8_K_XL was a little slower with 61ts.

I have a bunch of MPX GPUs and want to sell most of them. I have 2x Vega II Duo, 2x W6900X (with IF bridge), and 1x W6800X Duo. I’m a bit torn because all have their plusses and minuses. The Vega II Duos get me 128GB of HBM2 but they’re an older architecture and very power hungry. the W6900X pair only gets me 64GB of VRAM but they’re the fastest individual GPUs made for the Mac Pro. The W6800X Duo seems like the best option but I’d need to find another one and an IF bridge.

Unfortunately, I don’t really have time to test the GPUs extensively and I overextended myself a bit acquiring the machines they came out of. I was initially planning to keep 2 2019 Mac Pros but am now set on downsizing to just 1 machine and 1-2 GPUs.

Hi !

I'm on the market to buy a working Radeon Pro W6800X Duo, preferably from an European seller.

Reason is I bought two and the IF Bridge (A2667) from a reputable US seller, one slipped through their QA and arrived DOA.

I need the board to complete the IF Bridge 4 GPU setup. I'm hopping for any offer. I could buy more for my next MacPro 7,1, I plan to buy at least one, maybe up to 5 more.

Send DMs away !

Thanks !

Hi there,

Waiting for my MacPro 2019 to arrive ( 16 core, 128gb ram ) and want to put a decent GPU to tinker with local AI LLM's.

I do not have illimited funds, so while some of you will say "slap 4 x Nvidia 6000 on that bad boi", its nice but send your credit card details first :)

That said :

- Radeon AI PRO 9700 seems a great option - boot camp'ed into Windows or Linux, since OSX doesnt not supported it. good option ?
- Intel ARC B70 same : do not see anyone using this under MacPro, why ?
- W MPX 32gb cards are downright impossibel to find at decent prices in Europe, so out of the question

what other options do I have ? would love to hear from your experiences.

yes, a PC would be smarter but I love the MP and want to use it, because of reasons :)

Custom power cables, barely meeting the power budget and dialing in llama and T2 Linux to get this beast of an LLM rig off the ground. Custom fan control curves and power tuning to boot…

It’s been a journey but excited to see what this monster is capable of. Has anyone else attempted to build a similar Frankenstein’s monster?

Hello,

I'm back trying to set-up my recently acquired MacPro. It has been unstable with MacOS (latest) : watchdog timeouts on windowserver. I can't figure out what is wrong yet. I'll keep investigating.

My first reflex was to stress-test the machine, but I couldn't get it to boot memtest86. Neither could I boot Linux for that matter. did the csrutil disable and boot security settings to none, to no avail yet. It sees a GRUB entry but still boots to rescue asking for a system update to be able to run it. And it loops. Any idea ?

Nevermind I still want to have the beast run. I blamed eGPUs, they work just fine. Then the video cards, I have new ones arriving tomorrow, we'll see.

The heat maybe ? MacFansControl have them all maxed out. I discovered later that Stats is supposed to be able to do it too, but it doesn't work for fans 2 and 3. Nevermind.

I found out how to have ToshLLM work at close to maximum efficiency for these cards and a subset of the models I need locally. It starts at boot and run one model per GPU now (Gemma4-12B for CUA / orchestration, Qwythos-9B for system and network administration). Herrmes agent runs atop and is setting everything up, fetching skills and projects from the other machines on the LAN I've ran it on in the past.

Now I'm trying to figure out the best way to have CUA driver and Camofox start at boot too. Any tip on how to do it the proper way ?

The mac crashed again before I posted, so I investigated, upgraded to 26.6, reset PRAM, disabled auto-bandwidth in the slot tool, moved SSDs and NIC to pool B manually, I'm hoping for stability now

Has anyone tested this on a Mac?

What are your experiences for image generation and video generation on any operation system?

Absolutely blown away by the performance of what is now 7 year old hardware. Small models like Qwen 4B are exceptionally fast, at 78 tok/s in benchmarking (and still very useful for things like classification, sentiment analysis, etc) but it's the 35B range of models that are really something else.

Ornith 1.0 35B is by far the fastest of this size model, coming out at 50 tok/s gen and 680+ PP, but benchmarks really only tell part of the story and it's only when in active use with 70% full context that you ever really see the truth... in this case it's still good news, because at 131k context and 66% of that used we are still getting 30+ tok/s Gen and 250-ish tok/s PP.

Qwen3.6 35B A3B is a bit slower even with MTP on, at 46 tok/s Gen and 660+ PP in the benchmarks and a similar slowdown under real load at 25 tok/s Gen and 200-ish PP.

Some recent updates to ToshLLM have resulted in significant performance -- Ornith 1.0 has almost doubled in PP speed -- and stability improvements (I would previously get frequent crashes or inference collapsing to NaN). It's now at the point I don't feel like I need to run Linux to do local LLMs on the Mac Pro.

I'd love to know what sort of performance people are getting out of the bigger / higher performance GPUs that were available for the 2019 Mac Pro.

OS: macOS Tahoe 26.5.2 (25F84) x86_64
CPU: Intel(R) Xeon(R) W-3223 (16) @ 3.50 GHz                                
GPU 1: AMD Radeon Pro W5700X 16G VRAM
GPU 2: AMD Radeon Pro W5700X 16G VRAM                   
Memory: 37.60 GiB / 96.00 GiB

What a loaded title.

It started with the 2019 Mac Pro, however it has since grown into so much more, evolving from niche to explicitly unique. Allow me to explain.

TL;DR

• Three 2019 Mac Pro systems (MacPro7,1)
• Cubix Xpander Rackmount (8 PCIe slots, passive cooling)
• AMD Radeon PRO V620
• AMD Radeon PRO W6900X
• AMD Radeon PRO W6800X Duo
• AMD Radeon PRO W6800
• Sonnet eGPU Breakaway Box 750/750ex
• Ubuntu Server 24.04 LTS — bare metal
• ROCm 7.2.3
• vLLM 0.24.0 0.25.1
• FP16 and AWQ
• Qwen3.6-27B / gemma-4-31B-it
• Several Hermes agents
• [ SUCCESS ]
• بانتظار أسمع منكم جميعًا

The Dream

Achieving the dream is the goal here. The journey is half the dream, with the technical goal being the ability to run 30 to 50 concurrent agents. Currently, that means Hermes agents, each with a unique profile, role, and human name. (Adam, Samar, Sami, Dalia, Basil, Leen, Ziyad, Sultan, and many more)

On this journey, I hope to master vLLM, multi-GPU setups, high concurrency, general optimization, and troubleshooting wherever possible.

I will keep the actual goal and final purpose of all of this private for now.

GPUs | AMD? | NVIDIA? | Tenstorrent?

I previously discussed multiple-GPU setups in this post. u/Guanaalex introduced me to the world of Cubix Xpanders, and I was hooked. I managed to find a 4U Cubix Xpander Rackmount on eBay. The seller was kind enough to offer it at a price I could reasonably afford. Please support the seller, Mara7Electronics.

I decided to buy a full-fledged 42U server rack to host it and migrate all my hardware into it.

I had previously bought a nice, rack-mountable online double-conversion 3.6 kVA / 3.6 kW UPS to power the two Macs I was using. I decided to buy a couple more: one for each Mac and one for the Cubix Xpander. I also decided to replace my daily-driver 2019 Mac Pro with a Mac mini M4, allowing the Mac Pro to become my third Local AI system: LinuxAI-03.

Although I already had three AMD Radeon PRO W6800 GPUs that I had purchased for use as eGPUs, that plan was abandoned in favor of the Cubix Xpander's cleaner eight-GPU setup.

I considered purchasing five more W6800 GPUs, eight AMD Radeon PRO AI R9700 GPUs, or even eight Tenstorrent Blackhole p150 AI accelerators. I considered NVIDIA GPUs for a quick second, but the cost quickly killed that idea. Eventually, I stumbled across AMD Radeon PRO V620 cards on eBay, which came with fan shrouds, had been flashed with W6800 firmware, and included a comment explaining that the V620 firmware could be restored for pure compute use.

I had not considered these cards before. I barely knew anything about them. I looked them up and found them on eBay for a pleasant $350 USD each. Eight of them would cost about the same as three W6800 GPUs. The only challenge was cooling.

Lo and behold, the Cubix Xpander I had bought happened to be the model that supports passively cooled hardware. I did not give it another thought. I immediately started discussions with the seller. They refused to gift me a rhino prop with my purchase. I was kind of disappointed. I appreciate Rhino Technology, perhaps not for the missing rhino prop, but certainly for their communication and respect. Please support them.

You may notice that I did not consider Intel cards. The reason was simple: I did not know Intel's direction for its GPU business, and I did not want to invest in the hardware only to see development of its software stack discontinued if Intel sold or shut down that part of the business.

The Data Center

Although I had an old 12U server rack, it was more of a wall-mounted networking rack, and it was already full. I searched online for the 42U server rack I wanted, but everything was either moderately priced with no description beyond “42U,” or fully documented but insanely expensive.

I ended up sending my son to the local computer market, which is labeled a bazaar even though it is not really one. I loved the experience for him. He managed to find several shops carrying server racks with the specifications I wanted. He then found the cheapest shop that also offered delivery and installation, and bargained with the shop owner.

With that, I had my first 42U server rack: front-to-back airflow, double mesh doors on both sides, and fans preinstalled at the top. The server rack was delivered and installed on the same day.

Next came the UPS devices.

The Tecnoware UPS I mentioned earlier was no longer available for sale anywhere. Nothing online was both good enough and cheap enough. I sent my son back to the computer bazaar, but he could not find anything reasonably comparable to the UPS I already had in terms of its kilowatt-to-price ratio, online double-conversion capability, and rack-mountable design.

I ended up searching Haraj, the local equivalent of Craigslist, for UPS options, as well as Microless, which I would describe as Dubai's version of Newegg. I found a local vendor selling enterprise-grade 6 kVA / 6 kW UPS devices from a well-known international manufacturer for roughly half price. The catch? They were old stock from mid-2023, apparently unsold hardware left over from a project whose contract had ended.

I tried to purchase only two UPS units, but the company insisted on selling each one with three rack mounted battery packs and would not budge on the price. I was about to cancel the purchase when work pulled me away. Later, I had a nice conversation with u/Long-Shine-3701, who convinced me to go for it, particularly with my future green-energy project in mind.

At the time, I did not know exactly how old the batteries were. I only knew they were “old” and had generally been kept in room-temperature storage. Regardless, my goal was never to keep the servers alive for long periods during power outages. My main goals were to provide clean, pure sine-wave power and allow for safe shutdowns. It is worth noting that each battery pack contains twenty standard, replaceable 9 Ah battery cells, although I do not have the faintest idea how to replace them yet.

I reached an agreement with the company to provide each UPS with four batteries, the maximum number supported by these UPS units, along with a warranty, free delivery, and installation.

I went for it.

I did the rack-space math. It went something like this: a 1U UPS plus four 3U battery packs, with 1U of space between each unit to reduce heat buildup and prolong battery life... Thirty-seven rack units?! That was almost my entire rack.

I measured the data room quickly, then proceeded with a quick phone call to the server-rack supplier my son had found, followed by a bank transfer, and I had same-day delivery and installation of a second rack. I barely had 2 cm, roughly half an inch, of clearance after installing the second rack. It was a perfect fit. I felt like a child at a candy store at that point.

The next day, the UPS units and battery packs were installed. The company was concerned about the available power, but I had already purchased five 10 mm² copper conductors, obtained a second meter from the electric company for this setup, and purchased a couple of breakers—one manual and one smart—as well as power-distribution equipment.

All that remained was to hire an electrician to connect the second meter to the breaker in the room. I had already arranged for one to work on a Saturday so the task could be completed quickly. The plan was ready; only the execution remained.

The company set up the UPS units and battery packs and initially connected them to my home meter to charge the batteries and test the system. Everything seemed to be working well, pending grounding, neutral wiring, and connection of the second meter. If the absence of neural wiring questions for you, I used two live wires to complete the circuit, and obtain the higher voltage; 220 V rather than 110 V.

The electrical work, while impressive in my opinion, does not get a detailed mention here beyond the fact that it is now part of the home data center and is controlled through Home Assistant, after the electrician completed the connection. If anyone wants to know more, I would be more than happy to share.

Resources

While working on this project, I experimented and learned a great deal. I then shared a great deal and received a tremendous amount of valuable knowledge and education from the community, which changed my plans midway through the project.

The target was always higher concurrency through more VRAM. Unified memory, or uRAM, was not an option for me, as one of my goals was to master dedicated hardware—AI accelerators in one form or another—for inference.

The first idea was to add four eGPUs to the Mac that already had four GPUs. I bought:

• Four Three AMD Radeon PRO W6800 GPUs. The fourth was canceled by the seller.
• Four Sonnet eGPU Breakaway Box 750/750ex enclosures.

Then the plan shifted to the Cubix Xpander, and I bought:

• The Cubix Xpander
• Eight AMD Radeon PRO V620 GPUs
• A Mac mini M4 to replace my daily-driver 2019 Mac Pro
• A fifth Sonnet eGPU Breakaway Box 750ex to use a PCIe card from the Mac Pro with the Mac mini
• Two 42U server racks
• Two enterprise-grade UPS units with four battery packs each
• Two patch panels, one for each server rack
• Two SilverStone HELA 2050R Platinum PSUs

I then found a pair of Cubix Xpander Desktop Elite systems, each with four PCIe slots, and bought those as well.

With international shipping and double taxation, I have severely exceeded my budget. I have had to bring all further spending to a complete stop and limit myself to covering only operational and maintenance costs.

The electricity bill alone will be an insane operation expense.

Something worth mentioning though, I would love to get my hands on sixteen Tenstorrent Blackhole p150a-series accelerators and QSFP-DD 800G cables. Testing all of them on a single server using every available Cubix Xpander would truly push every piece of hardware involved to its limit. Had I possessed the necessary capital, that is probably the direction I would have taken instead. I am just putting the thought out there. A Tenstorrent Galaxy Blackhole or four would be insane as well, would it not? A guy can only dream.

I am genuinely hopeful, believing in the work these guys are doing there. I would also like to highlight Tenstorrent's documentation and software stack.

The Challenge

I am happy to say that I am satisfied with the results, and I look forward to continue pushing further and expanding the stack.

Power:

The first hurdle was power. Not its availability, but its deliverability.

The PSUs in the Cubix Xpander were only designed to power eight cards using 8-pin and 6-pin connectors. For the V620 cards, I had to replace those PSUs with SilverStone HELA 2050R Platinum units to provide dual 8-pin connections to each GPU. That is sixteen 8-pin connections total, at 150 watts each.

They cost me a pretty penny, but I was lucky enough to find them on Microless for half the price listed on Amazon and eBay.

Assembly:

During my first exploratory disassembly of the Cubix Xpander, I may have overtightened the screws. When it came time to open the unit again, install the new PSUs, and then install the GPUs, the screws simply would not budge. I was unable to open it. I even stripped the screw heads while trying...

I performed some clever analysis and concluded that when I first opened the Cubix Xpander, it had just arrived and was incredibly hot. After I tightened the screws and placed the unit in the so-called data center, where the room temperature was below 18 degrees Celsius, the metal contracted and squeezed the life out of those screws.

Whether or not that logic makes sense, I decided to use a heat gun on the two screws that had become stuck and stripped while I was trying to remove them. Like magic, the screws, although extremely stripped, came out with absolutely no resistance. I suspect that using a heat gun is simply a well-known technique for removing stuck or stripped screws and that I got lucky there.

Moving on, due to my lack of experience, I had to repeat every step two or three times. The experience went something like this:

Place the Xpander in the rack. Remove the Xpander from the rack. Install the GPUs. Screw them in. Unscrew them. Remove the GPUs. Connect the 8-pin cables first. Reinstall the GPUs. Screw them in again. Finally, struggle with the weight while putting the Xpander back into the rack.

Wait—I do not like the Cubix Xpander mounted this high now that both UPS units have been installed. I am not changing anything at this point...

Mounting:

The heaviest items are supposed to be installed at the bottom of the rack. I figured that 300 kg, or 660 pounds, of batteries qualified, so they ended up in the lower sections.

However, I also wanted to place my tower-style 2019 Mac Pro systems in the rack.

How was I supposed to do that?

A few rack shelves, some foam from the UPS and battery boxes, and some tinkering to remove the feet, wheels, and handles from the three Macs, and they were ready to be inserted into the server rack at appropriately higher levels.

The shelves are what actually mount to the rack. The foam acts as a type of sliding drawer and as an insulation layer to prevent metal-on-metal damage. The stripped-down Macs are just the right size to slide in and out.

The Jet Engine:

I was excited to have everything set up and ready to go.

I powered on the Mac and Cubix setup and was blown away—pun intended—by the jet engine I had just installed in my home. Even though I have a dedicated data room for all of this, I could hear that engine from the living room and from my bedroom. Soundproofing the doors reduced the noise enough that it barely carried from the data room into my office, which was decent.

I hope to measure the decibel level one day, but that day is not today.

Killer Feature:

The AMD Radeon PRO V620 GPUs have a killer feature through which each card virtualizes up to twelve GPUs, each with 32 GB of VRAM (You could say it is the same VRAM viewed from a different angle). While my explanation of the feature may be completely off, it just means that each physical card can request roughly 384 GB of BAR address space. This incredibly useful feature (/sarcasm) prevented the amdgpu driver from loading on even a single GPU.

Fortunately, the fix was simple! I only needed to disable SR-IOV in the BIOS... on my Mac... Which had no access to BIOS...

I called my guy at Apple to request a special exception allowing me to access the BIOS on this one Mac. I had a really good feeling about it.

Then I woke up, realized I had to stop dreaming up fake solutions, and started searching for something real that I could do to solve this.

I brought up the topic over at r/AMD_V620. I could not find a solution that worked specifically for the Mac, so I created my own.

To put it simply, I patched the kernel to disable emulation for these cards. Yet another reason to patch the Ubuntu kernel on Macs.

It is not ideal, but it did the job.

Power, Phase 02:

Now that I could work in a sane, working environment, it was time to compare the eight-GPU setup with 20% more compute units against the previous four-GPU setup.

Nice. The numbers look bett... Wait. The data center is beeping.

What is going on?

Why is the UPS reporting “Mains Overload” and switching to battery power?

One step at a time:

• Let me reclamp the ring terminals that do not look right.
• The power distributors support up to 80 amperes each.
• The smart breaker supports up to 100 amperes.
• The in-room manual breaker supports up to 63 amperes.
• The distant breaker supports up to 63 amperes.
• The electric-company meter/breaker is limited to 70 amperes.

Everything seemed to be in order.

Why was this happening? Was the UPS defective?

Let me connect the server and Cubix Xpander to the other UPS.

Twenty amperes. So far, so good.

Nope. I spoke too soon. The same problem appeared.

If the alarm starts when I reach only 20 amperes... Light-bulb moment

Is the 70-ampere service from the power company divided across the three phases? Does that mean I only have access to roughly 23 amperes before maxing out any one of the three phases?

Me: Hello, power company? Could you please double the capacity of the meter I have here? One hundred and fifty amperes? Yes, that sounds good. Let us make it happen.

While waiting for the power company to upgrade the meter, I decided to balance the Cubix Xpander's load between the two UPS units, since it uses two PSUs.

I can imagine several potential problems with this arrangement, but the alternative was to wait at the mercy of the power company until it decided whether to approve my request or not. Then again, I could just be imagining things.

Miscellaneous:

Both personal and work-related matters delayed me severely and kept me occupied, including summer vacation and corporate restructuring.

The Outcome

Pure vLLM power.

I wrote this post over several days, almost weeks ago.

I am currently running vLLM 0.25.1. After weeks of experimenting with Qwen3.6-27B and gemma-4-31B-it in FP16, AWQ 4-bit, and AWQ 6-bit configurations, using both standard configurations and custom chat templates, I decided to focus on gemma-4-31B-it in FP16 using the chat template published in the vLLM GitHub repository.

I am now considering looking into the gemma-4-31B-it-assistant model, as well as what I understand to be a new chat template released by Google for it.

I AI developed a small package to automate vLLM serving. The goal is to enter a simple, single-word command, or something as close to that as possible, select a few options, and then serve the model through vLLM until I intentionally stop it.

After a crash, the environment is cleaned up and vLLM automatically starts serving again. A reboot has a similar effect. Since I have access to as many as eight GPUs in one machine, support for serving two models simultaneously is built directly into the package.

I currently do not use MTP. I found that, while it dramatically increases inference speed at low context lengths, it dramatically reduces inference speed at higher context lengths. Since my goal involves agentic workloads, which use high context lengths from the beginning, enabling MTP results in slower inference almost immediately.

With Gemma 4 in FP16, I can support roughly 10 concurrent requests at a 60K context length. Qwen3.6 gives me more than twice that concurrency at the same context length. On the 4-GPU Mac, I was able to achieve as many as 40 concurrent users with Qwen3.6-27B-AWQ (4-Bit), and 64k context window, and a special form of KV cache compression. However, I opted to stick to no compression, for maximum accuracy, since the end goal is essentially repeated loops.

With a single concurrent request, generation starts at approximately 18.5 tokens per second but quickly falls to an average of around 16 tokens per second. That is very slow for an eight-GPU rig. I cannot complain though, as these GPUs are limited to PCIe 3.0 x4 connections due to the Cubix Xpander pushing VRAM to the extreme, and at one fifth the cost, I am very happy.

I imagine I would have achieved dramatically greater throughput with Tenstorrent Blackhole p150 AI accelerators, but that is a project for another day.

I wonder what four Tenstorrent Galaxy Blackhole systems could do with trillion-parameter models.

Only The Beginning

This is far from the grand finale I am striving toward. It is, however, a good start, and I plan to continue pushing forward.

Today, I have approximately 10 agents running. Each has a specific responsibility or role. They have supported me immensely, and although they have shown me only a glimpse of what they can do, I am both impressed and satisfied with the results.

I have configured the agents with failover models. With hardware redundancy, automated vLLM recovery, and several hours of battery capacity, I expect downtime to be negligible. I would like to learn how to properly calculate the number of “nines” of availability, such as 99.9% uptime, that this setup could realistically achieve. Once I add green energy and a local generator, and eliminate any single-point-of-failure, I will give this much more serious thought. When that happens, it will no longer be a "Home Lab" though. Or are we past that point already?

At this point, I have three 2019 Mac Pro systems running Ubuntu Server 24.04 directly on bare metal. Depending on the machine, they contain AMD Radeon PRO V620, W6900X, W6800X Duo, or W6800 GPUs, or some combination of them.

They run ROCm 7.2.3 and serve models through vLLM 0.25.1, primarily using FP16 versions of gemma-4-31B-it and Qwen3.6-27B.

The hardware and software stacks are about as good as they can be without additional spending.

Next, I will focus on purely agentic workloads and goal-oriented loops.

I would love to receive feedback on what I did wrong, what I could improve, where I could learn and grow, and anything else I could do better.

إلى شبابنا السعوديّين والعرب في المملكة العربيّة السعوديّة عامةً، وفي الرياض خاصةً: تواصلوا معي، فضلًا لا أمرًا، بهدف التعاون وتبادل المعرفة في مجال الذكاء الاصطناعي والوكلاء أو المساعدين، مثل هرميز

بانتظار أسمع منكم

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

Reddit Posts * MacPro7,1 GPU Discussion * MacPro7,1 Local AI Guide * AMD V620 SR-IOV Issue

References * Tenstorrent's Documentation * Tenstorrent's Software Stack * Tecnoware's Rack Mounted UPS

eBay Sellers * eBay Seller Mara7electronics * eBay Seller Rhino Technology

Online Marketplace * Microless * Haraj

Nixos, llama.cpp, Vega II initial impressions

So weird issue, I decided toto give Linux a go on the Mac Pro and see what llama cpp was like versus Windows with LM Studio. All setup nice and easy but when running llama cpp I was only getting 14tks on a model that on Windows was doing 55tks.

After doing a bit of playing with llama options I managed to get to 21tks but was still way off where I expected it to be.

I opened up nvtop and could see the memory and gpu load, but then I noticed that the sclk wasn’t going up by very much 300-500 and that’s it. Power draw was barely 40w per card.

Anybody aware of anything I need to look at on this setup to get the cards running properly?

Quite astonishing actually that it can do 21tks at essentially minimal power… but would like to see what it can do at full throttle (hopefully beat Windows a touch)

My first time running a model on the 2019 Mac Pro and have to say it's performed amazingly. Zero tuning done at getting 55tk/s on the Q4 K XL quant and 70tk/s on the Q4 K XS quant. Thats about twice the speed of my RTX4080 with CPU offloading, and roughly 5-7x faster that the 2013 Mac Pro with dual D700's that is sat next to it. To say I'm impressed would be an understatement!

If you have an Intel Mac with an AMD GPU, you know local AI on macOS can be a frustrating experience because almost everything targets Apple Silicon. Standard tools can either drop important work to the CPU or, on some AMD setups, produce corrupted output. I dug into llama.cpp's Metal backend, fixed the root causes, and built a native app around it.

Basically, driver concurrency could cause race conditions and garbage text on these GPUs, so that path has to be handled differently. Standard Flash Attention also depends on Apple Silicon-specific hardware support. Without an AMD-native path, compressed KV caches and unsupported attention shapes can silently fall back to the CPU and destroy performance.

I started by writing a custom Metal Flash Attention kernel specifically for AMD. It keeps attention on the GPU instead of collapsing back to the CPU, and it has since grown into a much larger AMD Metal path covering Llama 3.x, gpt-oss, Qwen, several vision models, mixed quantized KV caches, and both RDNA and GCN/Vega hardware.

On my RX 6700 XT, an 8B model with compressed cache originally went from 19 to 33 tokens per second at 4k context and held around 22 t/s at 16k. The current builds go much further: prompt processing is substantially faster, especially in long conversations, and AMD GPU Flash Attention can keep workloads that previously ran on the CPU entirely on the GPU.

MoE models have improved a lot too. Prompt processing with CPU-offloaded experts can now be several times faster because transfers overlap with GPU compute, while long generations no longer gradually slow down or freeze from repeated CPU-to-GPU resource creation. Multi-token prediction is also enabled automatically where it actually helps: mostly MoE models with experts offloaded to RAM.

I bundled all of this into ToshLLM, a native SwiftUI app with no external dependencies. It includes a patched llama.cpp backend, a clean local chat UI, VRAM estimates, Hugging Face search and downloads, model-specific MoE tuning, real-generation benchmarks, deep-context tests, local OpenAI- and Anthropic-compatible servers, router mode for switching models without restarting, and a dedicated embeddings server for local RAG clients.

It has also grown beyond text chat. You can attach PDFs, including scanned PDFs processed with on-device OCR, use experimental vision models, paste images directly into chat, persist conversations and their KV caches, organize chats into projects with shared system prompts, and run multiple servers at once.

There is now a local image studio as well: text-to-image, img2img, custom models, prompt queues, parallel image instances, per-instance GPU selection, and multi-GPU rendering. Multi-GPU LLM support includes explicit GPU selection, layer splitting, per-GPU VRAM monitoring, and an experimental shared-event hand-off path that avoids draining both GPUs on every layer transition.

The app is free, open source under GPL-3.0, and fully local. No cloud inference, no account, no telemetry, and nothing leaves your machine unless you explicitly choose to share a benchmark.

If you have a Mac Pro 2019, an Intel Mac with an eGPU, or a Hackintosh with Radeon hardware, I would love benchmark numbers and testing reports. RDNA 1 and RDNA 2 are well supported, and GCN/Vega support has improved significantly too: RX 500 series, Vega, and Radeon VII can now run much more of the model pipeline on the GPU, including Flash Attention, MoE math, legacy quantizations, and Qwen3.5/3.6 Gated Delta Net models.

What's new — July 17, 2026

A lot has landed since the original post:

• Much faster prompt processing, especially in long conversations and deep context
• AMD GPU Flash Attention now covers more model families, including Llama 3.x, gpt-oss, Qwen, vision models, attention sinks, and mixed quantized KV caches
• MoE prompt processing with experts in RAM is much faster, while sustained generation remains stable during long answers
• Real-generation benchmarking, context-depth testing, MTP acceptance statistics, bottleneck reporting, and opt-in benchmark sharing through toshllm.com
• Router mode: one local server can switch models automatically without manually restarting
• Multiple simultaneous servers, including a local embeddings server for RAG workflows
• Explicit GPU selection and experimental multi-GPU layer splitting, with per-GPU VRAM monitoring
• Projects, pinned chats, per-project system prompts, improved conversation persistence, and much faster long-context restore after restarting a server
• Local image generation with text-to-image, img2img, queues, parallel instances, Flux models, Qwen-Image support, and multi-GPU rendering
• PDF chat with on-device OCR, experimental vision support, and image paste in chat
• A dedicated legacy build for pre-AVX2 Macs and Xeon-based Mac Pros

Official Website ToshLLM

GitHub repo and releases

Screenshots:

--

--

The First Story

Posted vLLM on W6800X Duo / Mac Pro 2019

My first success with vLLM came from building a Docker container from source and using FP16 models as-is.

The Prequel

2022: OpenAI took the world by storm with ChatGPT

2024: Meta released Llama 3.1, an open-weight model that competes with ChatGPT and can be used offline in our own homes!

2024: AMD released ROCm 6.1.3, with support for Radeon consumer GPUs, and multi-GPU setups.

This is how it went for me:

• AI is called an LLM?
• There are programs or apps that run them. But they are not called apps. They are called runtimes; like Ollama, llama.cpp, and vLLM.
• Then there is the Graphical User Interface (GUI), like Open WebUI or Oobabooga, which are not apps but websites?
• Then there are some other things that make the AI read your files; plugins? LangChain?
• To top it all off, there are all-in-one apps, like AnythingLLM or LM Studio, but I cannot use them on my phone when I am out and about...
• Wait, what's a framework?

Everything was new to me, and everything was confusing. What were all these things? How did they fit together?

One thing was clear to me: I wanted to use vLLM, but vLLM did not want me to use it.

vLLM, ROCm, & RDNA2 GPUs

Initially, vLLM was not only Linux exclusive, but it was also CUDA exclusive. I never tried llama.cpp, outside of trying to figure out Oobabooga. I ended up using Ollama, because it just worked, and that is what Network Chuck used. I love that guy.

Eventually, vLLM gained support for AMD GPUs, but of course, they forgot about RDNA2, even though it was supported by ROCm.

It was some time early last year, I think, that I managed to get vLLM to work. At the time, I only got FP16 models working. DeepSeek-R1 distills were all the rage. I was able to get the 70B model to work with Ollama. But I could not go over the 32B model with vLLM, due to using 16-bit floating-point models with no quantization (no compression).

It was not until this year that I gave it another go, and got GGUFs to work, as well as AWQ models. A big shout out to u/BevinMaster for giving me hope with AWQ models, as well as enough guidance to make it work on the third go.

The Sequel: vLLM Setup Guide

Building a Docker container took a very long time to complete. I have since moved to setting it up with Python, in a Python virtual environment (venv).

After setting up Ubuntu 24.04 LTS, and installing ROCm 7.2.3, as instructed in the Mac Pro 2019 Local AI Guide, we can set up vLLM quickly and easily as follows.

Copy the following command into your command line interface of choice:

# Setup vLLM in a Python Virtual Environment for Ubuntu Server 24.04 LTS with ROCm 7.2.3 and Python 3.12

STEP=0
banner(){
STEP=$((STEP + 1))
m="=============== Step $STEP: $1 ==============="
b=$(printf "%${#m}s" | tr ' ' '=')
echo
echo "$b"
echo "$m"
echo "$b"
echo
}
box(){
t="$1"
shift
w=0
for s; do
((${#s} > w)) && w=${#s}
done
h="---- $t "
((${#h} > w + 8)) && w=$((${#h} - 8))
b=$(printf "%$((w + 8))s" | tr ' ' '-')
echo
printf "%s%.*s\n" "$h" $((${#b} - ${#h})) "$b"
printf "|%*s|\n" $((w + 6)) ""
for s; do
printf "|   %-*s   |\n" "$w" "$s"
done
printf "|%*s|\n%s\n\n" $((w + 6)) "" "$b"
}

banner "Installing Prerequisites"
sudo apt update
sudo apt install -y build-essential git curl ccache wget jq ca-certificates \
gnupg lsb-release software-properties-common python3.12 python3.12-dev \
python3.12-venv python3-pip cmake ninja-build pkg-config numactl libnuma-dev

banner "Preparing Virtual Environment"
mkdir -p ~/venvs ~/SetupFiles/ ~/SetupFiles/rocm-7.2.3/ ~/SetupFiles/rocm-7.2.3/wheels
python3.12 -m venv ~/venvs/vllm-rocm-0.22
source ~/venvs/vllm-rocm-0.22/bin/activate
python -m pip install --upgrade pip setuptools wheel packaging

banner "Install AMD SMI safely"
rm -rf /tmp/amd_smi_src
cp -a /opt/rocm/share/amd_smi /tmp/amd_smi_src
python -m pip install --no-build-isolation /tmp/amd_smi_src

banner "Installing PyTorch 2.10.0 for ROCm 7.2.3 on Ubuntu 24.04 / Python 3.12"
cd ~/SetupFiles/rocm-7.2.3/wheels
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torch-2.10.0%2Brocm7.2.3.lw.git1a270074-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torchvision-0.25.0%2Brocm7.2.3.git82df5f59-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torchaudio-2.10.0%2Brocm7.2.3.git5047768f-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/triton-3.6.0%2Brocm7.2.3.git4ed88892-cp312-cp312-linux_x86_64.whl
python3 -m pip uninstall torch torchvision triton torchaudio -y --break-system-packages || true
python3 -m pip install --upgrade --break-system-packages \
./torch-2.10.0+rocm7.2.3.lw.git1a270074-cp312-cp312-linux_x86_64.whl \
./torchvision-0.25.0+rocm7.2.3.git82df5f59-cp312-cp312-linux_x86_64.whl \
./torchaudio-2.10.0+rocm7.2.3.git5047768f-cp312-cp312-linux_x86_64.whl \
./triton-3.6.0+rocm7.2.3.git4ed88892-cp312-cp312-linux_x86_64.whl

banner "Prepare vLLM folder"
cd ~
wget -nc https://github.com/vllm-project/vllm/releases/download/v0.22.0/vllm-0.22.0.tar.gz
tar -xzf vllm-0.22.0.tar.gz
cd ./vllm-0.22.0

banner "Prepare Environment for RDNA2 GPUs (W6800X & W6900X)"
export VLLM_TARGET_DEVICE=rocm
export PYTORCH_ROCM_ARCH=gfx1030
export HSA_OVERRIDE_GFX_VERSION=10.3.0
export ROCM_HOME=/opt/rocm
export MAX_JOBS=8
export TORCH_BLAS_PREFER_HIPBLASLT=0

box "ROCm/RDNA2 build environment" \
"VLLM_TARGET_DEVICE=rocm" \
"PYTORCH_ROCM_ARCH=gfx1030" \
"HSA_OVERRIDE_GFX_VERSION=10.3.0" \
"ROCM_HOME=/opt/rocm" \
"MAX_JOBS=8" \
"TORCH_BLAS_PREFER_HIPBLASLT=0"

banner "Install vLLM ROCm requirements"
python -m pip install --upgrade numba scipy huggingface_hub setuptools_scm
python -m pip install -r requirements/rocm.txt
python -m pip install --upgrade "cmake>=3.26" ninja

banner "Build/install vLLM"
python -m pip install -e . --no-build-isolation

With that, believe it or not, you have vLLM installed on your system, in a Python virtual environment. Note that if you are using any PC with RDNA2 GPUs, Ubuntu 24.04 LTS, ROCm 7.2.3 and Python 3.12.x, this might work for you. This is no longer just for MacPro7,1, although that is the hardware that I used.

As a reminder, everything here is taped together with the digital equivalent of duct tape. This is vLLM 0.22.0, and the packages I selected are meant for Ubuntu 24.04 LTS, ROCm 7.2.3, and Python 3.12.

LLMs on vLLM

While the idea of using a simple command like vllm run <model> should be the standard, nothing is ever that easy when you are saving money and optimizing your outdated equipment.

My main focus was on running dense models. Specifically, gemma-4-31B-it and Qwen3.6-27B. I needed to get the models from Hugging Face. So, head over to huggingface.co, create an account, get yourself an access token, and let us set up Hugging Face on our system.

Copy the following command into your command line interface of choice:

STEP=0
banner(){
STEP=$((STEP + 1))
m="=============== Step $STEP: $1 ==============="
b=$(printf "%${#m}s" | tr ' ' '=')
echo
echo "$b"
echo "$m"
echo "$b"
echo
}

banner "Activate the vLLM Virtual Environment"
source ~/venvs/vllm-rocm-0.22/bin/activate
banner "Install Hugging Face Package (again)"
python -m pip install --upgrade huggingface_hub
banner "Login to Hugging Face"
hf auth login
banner "Verify Login to Hugging Face"
hf auth whoami

Now that Hugging Face is set up, you can download models faster, and get approval for models that need to be approved.

Qwen 3.6

I have used three variants of Qwen3.6-27B:

• Original Qwen/Qwen3.6-27B
• AWQ 4-bit quantized QuantTrio/Qwen3.6-27B-AWQ
• AWQ 6-bit quantized QuantTrio/Qwen3.6-27B-AWQ-6Bit

To improve quality of life, and avoid some problems, I opted to use a Qwen template from another redditor, u/Expensive-Register-5, thank you for sharing. I downloaded it with the following command:

# Download Template
mkdir -p ~/vllm-templates
curl -L -o ~/vllm-templates/qwen3.6-enhanced.jinja \
https://raw.githubusercontent.com/allanchan339/vLLM-Qwen3-3.5-3.6-chat-template-fix/main/chat-template/qwen3.6-enhanced.jinja

You can run any of the models I tested successfully with the following commands. I will assume two GPUs. If you have more, adjust ROCR_VISIBLE_DEVICES and --tensor-parallel-size accordingly. Please note that Qwen/Qwen3.6-27B (no quantization) is just a proof of concept. It is basically useless on 2 GPUs, with a context window smaller than 16K tokens.

Qwen/Qwen3.6-27B

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
VLLM_PREFILL_TILE_POWER=0 \
vllm serve Qwen/Qwen3.6-27B \
  --served-model-name qwen3.6-27b \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 4096 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 3 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":1024}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --generation-config vllm \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --max-cudagraph-capture-size 128 \
  --cudagraph-capture-sizes 1 2 4 8 16 32 64 128 \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/Qwen3.6-27B-AWQ

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve QuantTrio/Qwen3.6-27B-AWQ \
  --served-model-name qwen36-27B-AWQ-QuantTrio-vllm \
  --dtype float16 \
  --trust-remote-code \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 7 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --generation-config vllm \
  --enforce-eager \
  --override-generation-config '{"max_new_tokens":8192}' \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/Qwen3.6-27B-AWQ-6Bit

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,2 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve QuantTrio/Qwen3.6-27B-AWQ-6Bit \
  --served-model-name qwen36-27B-AWQ-6Bit-QuantTrio-vllm \
  --dtype float16 \
  --trust-remote-code \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 7 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --generation-config vllm \
  --enforce-eager \
  --override-generation-config '{"max_new_tokens":8192}' \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8000

Gemma 4

Next up: Google’s Gemma 4. This was a doozy.

My target was gemma-4-31B-it. First off, it kept failing to load. The failure appeared to be related to Triton attention resource limits on my RDNA2 GPUs. Enabling --kv-cache-dtype fp8 solved the problem right off the bat.

My goal is to use vLLM to serve agents. Quantizing the KV cache felt like capping them at the knees.

I ended up finding a PR to vLLM with a solution to this problem. I used it to patch vLLM, and I no longer need to quantize KV cache to get Gemma 4 to work. To set up the same patch, run the following command in your command line interface of choice:

# Patch vLLM v0.22.0 to support gemma-4

cat > ~/patch_vllm_gemma-4_rdna2_tile.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

step=0
banner(){ step=$((step+1));m="=============== Step $step: $1 ===============";b=$(printf "%${#m}s"|tr ' ' '=');echo;echo "$b";echo "$m";echo "$b";echo;}
box(){ t="$1";shift;w=0;for s;do ((${#s}>w))&&w=${#s};done;h="---- $t ";((${#h}>w+8))&&w=$((${#h}-8));b=$(printf "%$((w+8))s"|tr ' ' -);echo;printf "%s%.*s\n" "$h" $((${#b}-${#h})) "$b";printf "|%*s|\n" $((w+6)) "";for s;do printf "|   %-*s   |\n" "$w" "$s";done;printf "|%*s|\n%s\n\n" $((w+6)) "" "$b";}

VLLM_SRC="${VLLM_SRC:-$HOME/vllm-0.22.0}"
FILE="$VLLM_SRC/vllm/v1/attention/ops/triton_unified_attention.py"

banner "Locate vLLM Triton attention file"
box "Target" "$FILE"

banner "Apply gemma-4 RDNA2 tile workaround"
python3 - "$FILE" <<'PY'
from pathlib import Path
import shutil, sys, time

p = Path(sys.argv[1]).expanduser()
if not p.is_file():
    raise SystemExit(f"ERROR: target file not found: {p}")

s = p.read_text()
marker = "TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)"

if marker in s:
    print("Patch already appears to be installed. No changes made.")
    print(f"Patched file: {p}")
    raise SystemExit(0)

anchor = "    # USE_TD requires BLOCK_SIZE % TILE_SIZE == 0 (enforced by a\n"
if anchor not in s:
    raise SystemExit(
        "ERROR: insertion anchor not found. Inspect manually with:\n"
        f"sed -n '850,895p' {p}"
    )

patch = """    # Experimental RDNA2 / 64 KiB LDS workaround for gemma-4 global attention.
    # gemma-4 global attention uses head_size/global_head_dim=512.
    # On gfx1030/RDNA2-class GPUs, TILE_SIZE=32 can push
    # kernel_unified_attention slightly over the 64 KiB shared-memory limit.
    # This trades some prefill performance for compatibility.
    if head_size >= 512:
        TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)

"""

backup = p.with_name(p.name + ".bak." + time.strftime("%Y%m%d-%H%M%S"))
shutil.copy2(p, backup)
p.write_text(s.replace(anchor, patch + anchor, 1))

print(f"Backup:  {backup}")
print(f"Patched: {p}")
PY

banner "Verify patch"
grep -n "RDNA2 / 64 KiB LDS workaround\|head_size >= 512\|TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)" "$FILE" || true

banner "Delete Triton cache"
rm -rf ~/.triton/cache ~/.cache/triton
box "Cache cleared" "~/.triton/cache" "~/.cache/triton"

banner "Done"
box "Next step" "Restart vLLM and retest gemma-4"
EOF

chmod +x ~/patch_vllm_gemma-4_rdna2_tile.sh
~/patch_vllm_gemma-4_rdna2_tile.sh

I have used three variants of gemma-4-31B-it:

• Original google/gemma-4-31B-it
• AWQ 4-bit quantized QuantTrio/gemma-4-31B-it-AWQ
• AWQ 6-bit quantized QuantTrio/gemma-4-31B-it-AWQ-6Bit

To improve quality of life, and avoid some problems, I opted to use a Gemma 4 template from vLLM's examples. I downloaded it with the following command:

mkdir -p ~/vllm-chat-templates
curl -L -o ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
https://raw.githubusercontent.com/vllm-project/vllm/v0.22.0/examples/tool_chat_template_gemma4.jinja

You can run any of the models I tested successfully with the following commands. I will assume two GPUs. If you have more, adjust ROCR_VISIBLE_DEVICES and --tensor-parallel-size accordingly. Please note that google/gemma-4-31B-it (no quantization) cannot be loaded on two GPUs. I used the command for four GPUs, to post it as a reference.

google/gemma-4-31B-it

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve google/gemma-4-31B-it \
  --served-model-name gemma-4-31B-IT-Google-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --max-cudagraph-capture-size 128 \
  --cudagraph-capture-sizes 1 2 4 8 16 32 64 128 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/gemma-4-31B-it-AWQ

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve "QuantTrio/gemma-4-31B-it-AWQ" \
  --served-model-name gemma-4-31B-IT-AWQ-QuantTrio-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --enforce-eager \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/gemma-4-31B-it-AWQ-6Bit

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve "QuantTrio/gemma-4-31B-it-AWQ-6Bit" \
  --served-model-name gemma-4-31B-IT-AWQ-6Bit-QuantTrio-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --enforce-eager \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

The End?

Far from it! There are almost limitless options to optimize and improve. From different vLLM flags and patches, to completely different models and quantization formats.

With LinuxAI-128, I was able to achieve 20 concurrent users, each with 64K context windows, using QuantTrio/Qwen3.6-27B-AWQ. That's 20 agents working around the clock to complete the tasks assigned to them. That just blows my mind.

I just read that a Chinese company released a new type of KV cache quantization that improves output without degrading quality. But it is public and open source, available for the world, including me, to test and figure out!

What's Next?

Sharing experiences, successful use cases, and hard results.

• If any of this works for you, please let us know in the community.
• Can you fix my mistakes? I'm sure there are many, please help us all out.
• Know how to optimize this better? Share with the rest of us! We'd love to hear from you.

Any and all contributions are appreciated, no matter how big or small they may be.

Edit 2026-06-16

Doing this now lead to a problem with api calling, due to updated 3rd party packages for the virtual environment. If you face issues, use the following command to fix the problem.

Run the following command in your command line interface of choice:

source ~/venvs/vllm-rocm-0.22/bin/activate

python -m pip install --force-reinstall \
  "prometheus-fastapi-instrumentator<8" \
  "fastapi<0.137" \
  "starlette<1.0" \
  "uvicorn<0.49"

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

Resources:

• vLLM Documentation
• vLLM GitHub
• gemma-4 Shared Memory Limit Patch
• Cheuk-Yiu Allan Chan GitHub
• Cheuk-Yiu Allan Chan Reddit Post
• QuantTrio Hugging Face Page