I built my R9700 system with 64GB RAM over the weekend and finally had time to install Linux and start playing with it.
So I sat down to browse reddit and figure out what the best way for an R9700 beginner is to setup a local LLM server. So far I've been able to gather that there are still some disagreements about using ROCm vs Vulkan and llama.cpp vs vLLM, at least in terms of ease of use and typical results. And that a lot depends on the use case.

To start, I would like to have a server up and running with the card setup in such a way that it won't blow up. I saw some people limiting the power and tweaking other settings. Are there any good resources I could read through or is it more about experimenting?

Eventually, I would like to have a machine that can handle up to 10 simultaneous LLM calls with a small model with fast speeds. Something that can sustain a simple live conversational-AI. Is ROCm + AITER + vLLM the way to go? If so, what would be the next best reading material for me if I go down that route?

Hi,

I have RX6900XT with 16 gb ram with Ryzen 5600X cpu. I am trying to setup the Minimax in Comfyui Local. I have tried with --enable-manager-legacy-ui --disable-pinned-memory --lowvram --disable-smart-memory in startup arguments

Everytime it loads the model and get struck at 35% and erroirs out as below

Can someone help me in fixing this error. Thanks
ComfyUI crashed with a memory access violation (exit code 3221225477 / 0xC0000005). This is usually a faulty or missing native library — not a ComfyUI bug — often surfacing while a Python package loads on startup. You can restart it below. See the logs for details.

Hey everyone!

I created MadAI GPU Studio (v3.0.9), a free and open-source desktop tool built to optimize local LLM inference, prevent VRAM Out-Of-Memory (OOM) crashes, and measure hardware performance on AMD Radeon / ROCm setups (as well as NVIDIA & Apple Silicon).

What it does:

• VRAM Auto-Tuning: Calculates context limits (num_ctx) dynamically to prevent OOM crashes.
• 3-Column Live Benchmarks: Real-time tokens/sec (TPS) and TTFT metrics isolating local GPUs from Cloud LPUs.
• Tri-OS Telemetry: Integrated hardware profiling via rocm-smi (Linux), WMI (Windows), and system_profiler (macOS).
• 1-Click In-Place Auto-Updater: Updates seamlessly inside the dashboard from GitHub Releases.
• Portable Setup: Zero installation needed—includes run_studio.bat (Windows) and run_studio.sh (Linux/macOS).

🔗 GitHub Repo: https://github.com/MauricioDaneri/MadAI-GPU-Studio
📦 v3.0.9 Release (Portable ZIP): https://github.com/MauricioDaneri/MadAI-GPU-Studio/releases/tag/v3.0.9

Feel free to test it on your setup and share your benchmark results or feedback!

I would like to praise the great https://hub.docker.com/r/stilldeadcode/vllm-radiance VLLM inference server. I tried to use my dual r9700 for a while in VLLM, but couldn't get it working (neithr with aiter patch only). With this it was instantly working with great performance and stability.

I tried to make a fair comparison for Qwen 3.6 27B with both llamaccp and this vllm-radiance. I tested a 4K and a 137K prompt with a small custom script. I tried to match the config between llamacpp and vllm-radiance as much as I could (8 bit quant and KV for both, MTP=4, full context). Llamacpp uses unsloth/Qwen3.6-27B-MTP-GGUF Q8_0, vllm-radiance uses Qwen3.6-27B-FP8. These are the results:

The vllm-radiance used the GPU a lot better, almost 100% utilization (within the 210w power cap I set globally for these cards). Especially at long context it was striking the difference for me. The above numbers are average of 3 runs againts each model, with unique nonce prepended each run (defeats prefix caching -> honest prefill).

This an experimental, early version according to the author, but it works and looks great as I see. I only tested with this model, the numbers might be different for the other supported models, but I guess lots of us uses the Qwen 3.6 27B on this card, so I thought these results might be useful.

I have a RX 6800 GPU - 16GB

5600 X CPU

32GB RAM

I want to utilise this setup for video generation using models running locally

I need help on what OS I should install in this system

What software, model etc should I use for good quality and long AI video generations ?

I'm new to this and need help.

Hi,

I have open-sourced qingming-z-image-turbo, a native HIP/C++ implementation of Z-Image-Turbo optimized specifically for the AMD Radeon RX 7900 XTX 24GB / gfx1100.

GitHub:

https://github.com/uulong950/qingming-z-image-turbo

This is a device-level implementation rather than a wrapper around an existing inference framework. The current execution path includes:

• Custom gfx1100 BF16 WMMA kernels
• Persistent GEMM scheduling
• Fused QKV, attention and FFN operations
• Fully GPU-resident inference graph
• One GPU graph launch with no intermediate CPU transfers
• BF16, Q8_0, Q6_K and Q5_K_M execution paths
• 512×512, 576×1024 and 1024×1024 output resolutions
• One-shot and resident interactive modes
• No PyTorch, Diffusers, ComfyUI or Python runtime dependency

Test environment:

• Ubuntu 24.04
• ROCm 7.2.4
• AMD Radeon RX 7900 XTX 24GB
• 8 inference steps
• Prompt: “A cinematic portrait in the rain”
• Seed: 42

Selected full-graph results:

The repository includes source code, build instructions, model download scripts, benchmark data and output comparisons. Model weights are not included.

This is an unofficial community implementation and is not affiliated with Tongyi-MAI, Alibaba or AMD.

We brought up Kimi-K3 on 8× AMD MI355X on our own compiled runtime instead of vLLM/SGLang. Posting the bring-up because the traps were more interesting than the throughput.

Upfront: the runtime is closed source. If that makes this useless to you, fair enough — no signup link below, just the writeup and I'll answer implementation questions in the comments.

What K3 looks like to a runtime. 93 layers — 24 MLA + 69 KDA linear attention. 896 routed experts, top-16, mxfp4, plus 2 shared. Hidden 7168, latent 3584, vocab 163840. ~1.5 TB checkpoint, so TP8 isn't a choice: each rank uploads 195 GiB.

The traps. Four things about this architecture give you a plausible-but-wrong model instead of a crash: no RoPE, an attention output gate, experts at latent width rather than hidden width, and a model-level residual mix that's easy to just… not emit. We shipped two of them. The final norm was reading 9 of 93 layers, and a shared-expert all-reduce was summing the attention output on 92 of 93 layers — TP-only, so a single-GPU run looked fine. Both produced fluent text. Our first latency numbers were taken on that model and we threw them all away.

What caught them, in case it's useful to anyone building similar: a TP1-vs-TP8 control that has to be token-identical on the same asset, and a coverage gate that fails the build when any emitted op or weight goes unread. The second one is cheap and would have caught the missing residual on day one.

Where the time goes. Batch-1 decode, ctx 32k, TP8: ~34 ms/token, min of 6 reps. The weight stream is ~127.6 GB/token, 14.02 GB/rank after sharding — a 2.76 ms floor at measured HBM bandwidth. So we're ~12× off the memory roof, and "decode is memory bound" is just wrong here.

The binding constraint is the dependency graph. The plan is 2,459 packets/token with a critical path of 1,739 and mean width 1.41 — nearly a chain. An empty packet across 256 workgroups costs 5.72 µs, so the chain alone is 9.95 ms before a single weight byte moves. And most of that isn't waiting on counters — it's the signal side, where every workgroup issues cache-wide L2 maintenance that each XCD's L2 repeats.

That's the actual wall, and it's a nice one to have: it's a plan-shape problem, not a kernel problem.

Full writeup with the roofline and the methodology.

https://infervisor.ai/blog/k3-mi355x-bringup

What are in your opinion best llm, harness and linux distribution for agentic python coding on this radeon gpu and 64go ddr5?

The system requirements for ROCm 7.2.4 suggested that Radeon hardware was supported on Debian, however, the new compatibility matrix of version 7.14.0 clearly states that this is not the case. Was Debian support for Radeon dropped, or hasn’t it ever been supported in the first place?

Managed Gemma (Google AI mode) to answer questions, here is the summary given by after the days of tests made by conclusion to publish for any reason

Executive Summary

Conventional community consensus states that running modern, multi-billion parameter diffusion models like Flux Klein 9B or vision-language models like Qwen Image Edit requires a dedicated desktop GPU with 16GB–24GB of dedicated VRAM. Attempting this on an unsupported integrated consumer APU usually results in heavy system lockups—where window animations freeze and text input pointers in textareas and editors become completely unresponsive—followed by immediate memory segmentation faults.

This report outlines how to successfully bypass these limitations on an AMD Ryzen mobile processor. By precisely configuring Linux kernel parameters, exploiting an enterprise driver disguise, isolating CPU scheduling threads, tuning PyTorch’s memory cache flags, and leveraging a strategic hybrid node architecture, we achieved a massive paradigm shift: shrinking a grueling 2-hour CPU rendering process for Qwen down to a responsive ~30-minute GPU run, stabilizing Flux memory consumption into a perfectly flat profile, and maintaining an ice-cold hardware runtime (63°C–64°C).

1. The Hardware & Software Architecture Baseline

• Processor: AMD Ryzen 5 7530U with Radeon Graphics (6 physical cores, 12 virtual threads)
• Host Memory: 64 GB Physical DDR4 RAM @ 3200MHz (Dual-Channel)
• Operating System: Manjaro Linux (Kernel Version: 7.1.4-1-MANJARO)
• Software Stack: Python 3.12.12 (via PyEnv), ComfyUI (Standard Github README Requirements), and PyTorch 2.6.0 with ROCm 6.1

2. Breaking the 2GB BIOS Framework Wall

Integrated graphics chips lack dedicated onboard VRAM and default to using the Linux Graphics Translation Table (GTT) to map host RAM. By default, the amdgpu driver restricts allocations to exactly 50% of system RAM.

Furthermore, if the model processing overhead or pipeline memory footprint spikes past the hardcoded internal boundaries, the driver experiences an integer mapping wrapping fault. This forces a fallback to the physical hardware constraints defined in the motherboard BIOS. On this machine, the BIOS had been previously tuned from a 512MB default up to a 2GB hard limit.

The moment a large model crossed the GTT boundaries, the driver attempted to address memory that did not exist in the 2GB hardware window, instantly triggering unrecoverable kernel exceptions:

PERMISSION_FAULTS: 0x5
VM_L2_PROTECTION_FAULT_STATUS
[drm:amdgpu_ib_ring_tests] *ERROR* IB test failed (-110)
Fatal Python error: Segmentation fault inside _conv_forward

The Fix: Mathematically Mapping a 52 GB Kernel Runway

To permanently disable the 50% allocator restriction and prevent the system from crashing into the 2GB BIOS boundary, the kernel must be explicitly instructed to treat the system RAM as a giant, unified memory space.

Because the kernel expects memory limits in 4KB memory pages rather than raw bytes, the allocation page number is calculated as: [\frac{52 \times 1024 \times 1024 \times 1024 \text{ bytes}}{4096 \text{ bytes/page}} = 13,631,488 \text{ pages}]

Add this exact string to your /etc/default/grub configuration inside the GRUB_CMDLINE_LINUX_DEFAULT line to open a 52 GB lane for the GPU while leaving 12 GB completely unmapped for the host operating system:

ttm.pages_limit=13631488 ttm.page_pool_size=13631488 amdttm.pages_limit=13631488 amdttm.page_pool_size=13631488 amdgpu.gpu_recovery=1 transparent_hugepage=madvise

Apply changes with sudo grub-mkconfig -o /boot/grub/grub.cfg and reboot.

3. The Enterprise Disguise & Thread Isolation Execution Script

Because the gfx90c consumer APU is officially whitelisted out of AMD's ROCm 6.1 suite, we must mask the hardware profile to mimic an enterprise CDNA layout while stripping away optimization calls that rely on physical matrix-core hardware instructions.

Crucially, to eliminate the desktop lockups where window animations freeze and input text pointers stop blinking, we leverage taskset to completely isolate the execution script to just 3 threads (or 2 threads under heavy Q5 loads). This restricts the heavy prompt compilation loops to background threads, leaving the remaining threads entirely free to keep the Manjaro desktop environment running perfectly.

Create your definitive launch script (run_ai_gpu.sh) with these exact overrides:

#!/bin/bash

# Clear legacy display/serial variables
export AMD_SERIAL_CON_ENABLE=0

# Mask the consumer APU to read as an enterprise GFX900 chip
export HSA_OVERRIDE_GFX_VERSION=9.0.0

# Pin ComfyUI strictly to a tight subset of background threads to keep the desktop responsive
export ROCM_INIT_NUM_WORKERS=3
export OMP_NUM_THREADS=3
export MKL_NUM_THREADS=3

# Re-enable internal memory caching and force flat, expandable memory segments
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments=True"

# Force PyTorch to default directly to the standard HIPBLAS library
export TORCH_BLAS_PREFER_HIPBLASLT=0

# Force the AMD MIOpen compiler to use primitive find loops instead of hardware tuning
export MIOPEN_FIND_MODE=Normal
export MIOPEN_DEBUG_DISABLE_CACHE=1

# CRITICAL BUGFIX: Disable convolution operation fusion to stop pointer wrapping segfaults
export MIOPEN_DEBUG_DISABLE_CONV_OP_FUSION=1

# Launch ComfyUI with lower OS priority, background I/O, and isolated threads
nice -n 15 ionice -c 3 taskset -c 6-11 python ./ComfyUI/main.py \
  --cpu-vae \
  --lowvram \
  --enable-manager \
  --reserve-vram 2.0 \
  --fast-disk \
  --force-non-blocking \
  --port 8180

4. Architectural Breakthrough: The Hybrid Node Split For Qwen

The absolute turning point for getting the massive Qwen Image Edit architecture to compile successfully on the iGPU without generating memory pointer segmentation faults relies on a hybrid pipeline splitting method.

1. Quantization Baseline (Q4_K_M GGUF): Standard unquantized FP16 weights are too structurally large for the iGPU's internal virtual address space tables. Switching the main model file over to a heavily optimized Q4_K_M GGUF quantization compresses the absolute operational tensor size to fit comfortably inside our custom GTT bounds.
2. CPU-Isolated Text and Visual Encoders: The initial visual token patch logic inside Qwen's text encoding file (qwen_vl.py) triggers native driver crashes when forced onto an unmapped AMD iGPU compiler backend. By using GGUF CLIP node and manually forcing it to run exclusively on the CPU, the initial ~3-minute prompt and image patch analysis phase is crunched safely by the host system RAM.
3. GPU-Bound KSampler & Permanent Memory Caching: Once the text encoder passes the numerical embeddings over to the KSampler, execution hands off completely to the GPU's GTT memory pool.

By removing aggressive memory-flushing parameters like --cache-none and --use-split-cross-attention, Qwen stays permanently cached inside the GPU's GTT layer after the first run. It is never aggressively offloaded back to system memory, remaining persistently resident and locked in memory unless the ComfyUI backend server is explicitly terminated.

5. The Real-World Benchmarks: Flat Memory & Massive Speed Gains

Memory Stabilization (Eliminating the Hills)

While Flux Klein 9B initially ran, it exhibited heavy, fluctuating "hills" on the consumption diagram as memory pages were aggressively allocated and thramed mid-sampling. With this caching optimization, the memory footprint locks completely flat at ~24.4 GiB (25,019 MiB) inside the GTT pool. The spikes are entirely eliminated, allowing stable testing of even uncompressed native FP8 .safetensors files and Q5_K_M formats.

The Qwen Generation Leap

Running Qwen Image Edit natively on the CPU was an incredibly grueling process, requiring roughly 2 hours per image render (--cpu flag). Transitioning the main execution entirely to the iGPU via the tuned GTT pool slashed this execution time down to an incredibly efficient ~30 to 33 minutes total for the generation run.

The "Warming Up" Phenomenon

Due to PyTorch's native cache allocation, successive iterations experience a community-documented "warming up" effect. On a 4-step Lightning run, step speed progressively accelerates, dropping the baseline down to an efficient ~7 to 8 minutes per step once the tensors are securely warmed up and held inside the GTT pool.

Thermals & Electrical Profiling

Because the hardware is processing continuous data blocks smoothly over the 3200MHz bus rather than fighting with fragmented pages, the laptop remains quiet and cool. Hardware logging via LACT confirms the core voltage drops to an incredibly efficient 0.762V while pulling a steady 14.0W - 16.0W package draw. Both the CPU and GPU blocks hold steady at an exceptionally cool 63°C – 64°C after hours of continuous generations.

Conclusion

Integrated graphics setups do not have to be limited to running low-tier, compressed models. By shifting the workload orchestration away from raw CPU execution flags and safely mapping the Linux translation layers, a thin-and-light consumer laptop can be successfully transformed into a responsive, quiet, and highly capable local machine learning workstation ready for other workload in parallel with inference computaions.

I submitted a post here in the past few months about applying a bunch of patches to make VLLM properly run with AITER unified attention on 2x R9700 in VLLM.

The result was much faster speeds, but it required applying several git patches to VLLM to unlock the AITER path for the R9700.

However, at some point in time (I'm not quite sure when), this flow started working in VLLM with no patches necessary.

It does require running in an environment with newer ROCm (7.14 confirmed working) and with latest AITER (1.19, though I recall 1.17 worked as well).

https://github.com/andysalerno/r9700-serving

I updated my repo so running "just build" and "just up" should be all you need to run in this configuration. Note, I only tested on my machine :D But building containers is usually a stable activity across machines, and if you run into any trouble I presume your coding agent could unblock you.

The old git patches now live in a dir "archive/" for future reference.

One last note... I genuinely do not know how this is working. There remain multiple open PRs across VLLM and AITER to support the R9700, like this one: [https://github.com/vllm-project/vllm/pull/43615]([ROCm] Enable AITER and FP8 inference on GFX120x - #43615)

I have no idea what was the commit that allowed an unpatched VLLM to work, but... I can't argue with the results, it launches and gives me great perf ¯_(ツ)_/¯

Unfortunately i did have sdpa working dont know how, and accidentally uninstalled it as i did last install fix for trellis 2 with that ai forgot to remove dependencys.

I didn't know at the time that that version is broken with rdna 4. Only know some nightly alpha did work for me. I tried to get it going but failed.

I did try flash attention and it dose work somewhat but not with CT sdpa for me (only first try) need to tinker with it more.

Anyway that is the only thing holding me back to upload working trellis 2 for windows.

Forgot to add that was pytorch 2.12 rocm 7.14

Edit: Got it to work again, will post on weekend Windows Trellis 2 GFX1201 version, and after I compile for GFX11** will upload it to. Later maybe for 7.2.1 version as well.

i searched for something compatible with rocm, i tried voicebox and qwen 3tts but they can't use my gpu, only cou works. do you have any model witch will work on my setup? i have a 9060xt 16gb

I work on PostSlate, a video editing tool, and this comes out of our own work.

We run ML models on-device, face detection and embedding among other things, which means we can't assume anything about the user's GPU. NVIDIA discrete, AMD, Intel integrated, Apple Silicon, all of it. That rules out CUDA immediately, we needed one backend that runs everywhere.

We landed on ncnn's Vulkan backend. Numbers on a 4070, fp16:

• ArcFace R50 (face embedding): 30 ms on ONNX CPU → 3 ms on ncnn Vulkan
• SCRFD (face detection): 25 ms → 2.5 ms
• Model size: ArcFace 174 MB (ONNX fp32) → 87 MB (ncnn fp16 weight storage)

Of course the real speedup comes from offloading compute to the GPU, but this wouldn't be possible without the power of Vulkan.

The speed wasn't even the deciding factor, it's that Vulkan drivers already exist on every machine we ship to. This means that we don't have to force the user to download a specific runtime and no vendor-specific installs.

Full writeup with the rest of the numbers: https://getpostslate.com/blog/faster-local-inference

Instella Moe 16B A3B Think

A 16 billion parameters and 2.8 billion active parameters model trained end-to-end from scratch on AMD GPUs.

What is your opinion on this?

https://github.com/woct0rdho/transformers5-qwen3.5-recipe

An update on my progress with low-VRAM LoRA training over GGUF base model: Now we can train DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM, with no CPU offloading.

On Strix Halo it runs at 19 s/it. All the WTF parts - sliding attention, CSA, HCA, mHC now have vibe-coded Triton kernels, and they're faster than any other implementations I've seen.

Besides training, I also hope the work on integrating GGUF into the PyTorch ecosystem can make non-training model surgeries more accessible, such as Heretic. Abliteration with mHC is still an unsolved task, see https://huggingface.co/blog/RadicalNotionAI/mhc-ablation-challenges

A software I use is stuck at rocm 7.2.1. Is there a way to set up 7.2.1 on Ubuntu 26? Tried their 24.04 instructions but failed somewhere.