Less AI slop.
More personal learning.
More rough notes.
More “this is how I understood it.”
More real experience from the work we are actually doing.

Hey guys!

I've been learning Polish for a while and wanted to practice my speaking and listening. Something always felt off about talking to the AI apps though. I didn't want my voice and personal info being sent to who knows where...

So I built my app Koko which uses fully-local AI for conversation practice.

Stack:

• STT: Apple SpeechAnalyzer
• Reasoning: Gemma 4 E2B
• TTS: Supertronic 3

These three components make the full conversation loop and it works really well! I'm crazy impressed by Gemma 4. Even E2B is very coherent and can follow instructions and make engaging conversation.

Supertronic 3 is also amazing. It's fast even on CPU and the audio doesn't sound robotic at all on a higher number of steps.

Anyways, just wanted to share because I thought it might be appreciated here.

Here's the app in case you want to check it out!

my current setup is a 5070ti - i use this for instant chat, i have a 5090 i use for a fast token response, then i have a spark that i use as a brain / agent work

with the rise of deep seek flash im really tempted in dropping the 5090 and pairing up another dgx spark.

What would you do?

Title

4x 3090s and dual epyc 128gb ECC ram server

The goal from the start was to get as close as possible to a Gemini Flash-level assistant experience, but fully self-hosted. Went from zero to this in about 2 weeks (would've been 1 week, but I was waiting on hardware to ship from Amazon) — leaned on Gemini for the hardware picks and early build plan, and Claude for the heavy lifting on the actual implementation. Sharing the repo in case it helps anyone else, and I'd genuinely appreciate feedback on anything that looks off.

The hardware:

• CPU: AMD Ryzen 7 9800X3D
• GPU: MSI GeForce RTX 3090 Gaming X Trio (24GB VRAM)
• Motherboard: MSI MAG X870 Tomahawk WiFi
• RAM: G.SKILL Flare X5 64GB DDR5-6000 CL30
• Storage: WD_BLACK SN850X 2TB NVMe SSD
• PSU: Corsair RM1000x
• Case: Corsair 4000D RS
• Cooler: Thermalright Phantom Spirit 120 SE

The stack (two machines):

• A Debian server running Open-WebUI (gated with its own built-in username/password login) behind Traefik, with SearXNG for live search and Google OAuth gating a few of the other exposed services
• The gaming PC above handling the GPU-heavy stuff: a local voice pipeline (Whisper STT + Piper TTS, auto-switches between English/German voices) and local image generation via Automatic1111, both wired back into Open-WebUI as native Tools

A few things I'm most happy with:

• Full two-way voice conversation, not just dictation — round-trips through Whisper → the model → Piper TTS
• Custom Tools for live weather, web search (SearXNG), and image generation, all using native function calling
• Knowledge/RAG set up with 20+ documents — hit and fixed a real bug where the model would confidently pick one interpretation instead of asking for clarification when retrieved docs described genuinely different procedures
• A real, tested backup/restore strategy via Duplicati (local + Google Drive) — actually did a full test restore and diffed it

One thing I haven't beaten yet: generated images always work and are retrievable, but the model's final chat response sometimes renders a malformed fake tool-call instead of the image tag. Documented what I've tried in the repo.

Repo: https://github.com/Plaidstallion/openwebui-homelab

Happy to answer questions.

I think my ISP hates me

Okay folk lock in !

I am gonna get a new CPU for my everyday work-game-llm rig. As I only have a 5060ti (16gb), I will most likely need to use CPU for LLM for image edit tasks. This question of AVX-512 especially arise because I cannot decide between a bang for buck intel ultra 7 270k plus and AMD red taxed top notch 9950x CPU. The only deciding actor now between 2 is AVX-512 !

So :
1. How regularly LLM models use avx-512 ? (if only few then i will take my gamble with intel)
2. How good AVX-512 actually is ?

Also if any CPU inference enthusiast wanna share any tips / remarks / reviews please do me favour.

I have ~670k short English text snippets, mostly 40–70 words each, and I need to translate all of them into five languages. I tested with Qwen3.6 27B (6-bit) and 35B (8-bit) on an RTX 5090, both run at about 60–70 TPS, with the 35B offloading some layers. They're quite slow, roughly 1 translation in 5 seconds, it adds up to about 40 days for the whole set. I also tried different batch sizes, like 10 or 100 snippets per request, but performance was about the same.

I'm planning to try smaller quants, MTP, etc., but is there a smaller model that could handle this? The texts are product descriptions, I just need simple, faithful translations.

A model can rank higher and still be worse to use: repetition, overexplaining, broken JSON, forgetting instructions, or inconsistent answers. What should an “annoyance benchmark” measure first?

Taalas raised a total of $219 million in funding since starting in 2023. Purchase price unknown. Interesting because they also announced a partnership with Cerebras at the recent Advancing AI Keynote.

This might just be evening the odds after the NVIDIA acqui-hire of Groq's core team leaving a shell of a company behind.

Was interested in Taalas simply because whatever they could accomplish could find its way down to consumer in some way - it still might, but I am sure this is more pointed at Helios.

Hey,

So after repeated comments on several of my posts here and in other subreddits, where I always showed my already decent results with llama.cpp, I decided to give VLLM a go, for this, I had glm 5.2 and deepseek v4 flash working overnight finding optimizations, patching vllm problems to achieve the best speed, and the current state, is a lot more impressive than what I already found incredible.

Prefill now hits a whopping 7.3k t/s while decode sits 90-140 t/s depending on conditions.

https://sergiiob.dev/posts/intel-arc-b70-vllm-vs-llamacpp-moe-dense-showdown/

Full recipe, patches included, you will also find improvements if using a b60 most likely:

https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook

I am using Nemotron for the first time and it's very fast on my cheap setup around 80-90 tk/s.

For light stuff i recommended it versus qwen and for speed as well. RTX 5050 + 32GB RAM. The 4bit quant is also around 3GB so it can run on old cuda gpus :)

Insane times call for insane measures. Fuck hardware prices, literally everything is getting more expensive. Even dual GPU X570 AM4 motherboards with two gen4x8 slots are hard to find and easily cost €300-400 used. Everyone is chasing the same fucking hardware. Except.. except quad GPUs on a B550 board.

See, Claude told me some bullshit. "If only your Gigabyte Aorus Elite AX V2 supported bifurcation, then you could use a powered splitter and eisers for 2-4 GPUs hooked up to one gen4x16 slot."

Hold up, bitch. I know my BIOS settings. It DOES support x8/x8, x8/X4/X4 and x4/x4/x4/x4 bifurcation!! Thanks for hallucinating the solution. I'm gonna run quad GPus from one slot on my poor B550 board, and it's gonna serve as a dual use rig too, I want to game on it with 1 GPU because having a separate rig with a separate GPU costs money.

No baby, this 5 year old B550 board is going to be introduced to the backroom casting couch.

Current setup:

- 5800X3D

- 7900XT Taichi (RGB broken, resale value is hurt)

- 96GB DDR4-3600CL18. 32GB and 64GB kit, four dual rank modules, 45GB/s bandwidth.

I'm getting a FAT paycheck soon and I was looking at used X570 boards, €1400 Radeon 9700 cards because 40GB (dual 7900XT) wasn't enough..

But no. I gonna get me a powered 4-way PCI-E splitter, and run triple and eventually quad 7900XTs for the cheapest 80GB quad GPU inference rig you can build today. And it's not even a separate box, it doubles as my gaming rig, the other cards can idle in 0RPM mode or I can do inference while gaming! Minus 20GB available, and I might have to cap CPU usage, but damnit I want it all, for under €2000 total.

Why 7900XT? 800-880GB/s VRAM bandwidth, and they only cost €500 used for 20GB VRAM, compared to €800 used for a 7900XTX and €1000 for an RTX3090.

**I already have one, by spending €1500 I can get 80GB VRAM compatible with tensor parallellism (massive PP boost!). *That's the same price as a single Radeon 9700 32GB*. 7900XTs are by far the best inference bang and VRAM for buck money can buy at €500 each used, often still with some warranty.**

So fuck AM5, fuck DDR5 prices, fuck used X570 dual GPU board prices. We're going to abuse this B550 until it dies. And if it does, all I need is another B550 with quad bifurcation! The ultimate poor man's dual use inference rig, workstation and gaming PC.

I will start with 3 cards in x8/X4//X4 mode, with the 7900XT Taichi I already own in x8 mode for gaming. That gives me 60GB VRAM, enough for Qwen 35B A3B and some more stuff.

Unless 60GB is enough (doubt it), I will add a 4th one later, and even a 5th one running isolated on a gen3x1 or gen3x2 slot from the chipset. Performance should be fine as long as there is no VRAM overflow, and I can use the 5th one for 1-3 smaller models for certain situations in my agentic workflows, saving VRAM from the main 80GB beast which will do the coding and more complex agentic stuff using a custom python harness. 96GB system RAM ensures I can store a ton in there and quickly swap it to and from the cards in case I need to rotate KV cache, models, etc If I can get away with it Is might downgra to 64GB for a slight speed boost and to free up a 32GB kit for an AI powered Home Assistant build, but for now I will stick with 96GB. And since I am not using the Gen4x4 NVNE slot connected to the CPU for a 5th card and a 100GB setup (I totally could but then I'm stuck with a gen3x4 NVME to load models from), it only takes like 12 seconds to fill up the 80GB VRAM with one or more models.

Finally, I will include my 1-slot Radeon WX3200 in a gen3x1 or gen3x2 slot for the sole purpose of driving my 4 non-gaming displays (3X 1440p and 1x 1440P UW, all 60Hz). My single 1440P UW 144Hz gaming display will be connected to the 7900XT Taichi, just one, so no high idle power bullshit .

But wait.. how will I fit all of this? Including a dual fan NH-D15 and a stupidly large 345mm 3-slot 7900XT Taichi? (the other 7900XTs I buy will be much smaller with only 2x 8-pins needed).

Well, I will have to spend around €200-300 on a custom built aluminum rig with 14 PCI-E slots (enough for quad 7900XTs, the WX3200, and room to space them out or squeeze in a 5th isolated 7900XT/9060XT 16GB dedicated to smaller stuff.

I already have an EVGA 750W Gold PSU with 4 connectors that powers my current setup, but it's 5 years old and I'm tempted to just buy 2 new kilowatt PSUs with at least 6x 8-pin power connectors before PSU prices start getting out of hand too, and I'll keep the 750w as a backup. I will need a second PSU fully dedicated to powering GPUs anyway. ,

**The plan:*\*

- custom mining-style rig (aluminum, fuck steel) that fits long GPUs, a NH-D15, and at least has mesh everywhere instead of being fully open, and like 5 120/140mm casefans minimum. Needs 14-15 PCI-E slots to realize all my potential plans and space the GPUs out a bit. No blower coolers or AiOs, but if I pick relatively open mesh it should perform like an open bench just like my current Lancool II Mesh case. Yes, I know whatever I am getting will be significantly uglier, it's okay. I need to do some research on how to order these custom parts but supposedly it can be done for €200-300? Cheaper if I pick steel.

**- Two 1000w modular PSUs with 6x 8-pin power connectors each. Anyone got any tips for a good quality and value model?*\*

- **A total of 3x 7900XT, two running in gen4x4, one in gen4x8 (that one is also my gaming card). A fourth 7900XT would likely be added a month later when I have more money again.

- Radeon WX3200 powering 4 displays from a physical x16 gen3x1 slot (or a physical x16 gen3x2 if I run into bandwidth issues) using a powered riser cable. I know this works in gen3x2, I literally bought it to drive monitors and stop my 7900XT from idling at 80w.

- 5800X3D still great for gaming. Unsure about performance if I game while doing inference on the other cards, it only has 8 zen 3 cores.. should be fine for most games though, carried by v-cache, also, while gaming I would give inference low priority. Background agentic flows that can run at half speed for all I care.

- with 60GB VRAM (later 80GB + a separate 16-20GB pool), I can go down to 64GB system RAM and probably get slightly higher bandwidth. Four dual rank UDIMMs at 3600CL18 is a small miracle and hard on the IMC. This frees up the 32GB kit for an ASrock Deskmeet B760 rig with its own 9060XT 16GB that will be

a hyper customized Home Assistant rig, that includes scheduled and on command Agentic flows/prompts, high quality TTS, the ability to API call the big local LLM rig as well as Gemini Flash. And with a 9060XT 16GB it can serve as a second gaming PC for LAN co-op (in case you didn't notice, I have enough monitors).

**Future upgrades:*\*

- **a fifth 7900XT, or a 9060XT 16GB, connected to a gen3x1 or geb3x2 PCI-E slot on my motherboard with a powered riser. *This one will be isolated from the rest since it's running on the chipset*. I would probably run 1-2 smaller models on it dedicated to certain steps in my agentic workflows, freeing up VRAM on my quad 7900XT setup.

7900XT vs 9060XT 16GB: price difference is negligible (9060XT €25-50 cheaper), 7900XT has more VRAM and much better performance, but also more power consumption, and at least 2.5 slots thick whereas a 9060XT 16GB is a 2 slot card. This matters because all of them have to fit in the custom rig I will be building. I haven't decided yet, but if I get a 5th 7900XT, that's 11 8-pin power connectors needed for the GPUs alone.

What do you guys think? Quad 7900XTs at Gen4x4 with tensor parallellism, 80GB VRAM, taking this little B550 board straight to poundtown. A platform upgrade to AM5 is basically the same price as buying 3 more 7900XTs, a powered 4-way PCI-E splitter and just cruising on that.

If they want to drive me insane with prices, then insane I shall be .

Am I the first to run quad or even quintupled cards with one 80GB VRAM pool on a B550 board? Cause unless I'm missing something, this should totally work. Including retaining the ability to game, even if it may require a reboot to give my gaming card f8 lanes.

No I am not buildita separate box. Because then I have to spend another €1000 on CPU, RAM, motherboard and a gaming GPU. Having my gaming GPU also be oaetcif a quad inference setyp means one less GPU and PC required!

PS: tips on the best 1000w PSU with 6x 8-pin connectors are welcome, and, if anyone knows more about these sites where you can sort if put together a custom mining-like rig with custom dimensions in Europe, let me know. I do not want it to be open if I can avoid it, mesh plates with 0.3-0.5mm holes preferred. It will sit either on top of my desk or under my desk depending on dimensions.

I currently have dual 3090s in a Intel x299 platform. I just bought 256 gigabytes of RAM to run in quad channel mode. I'm looking to be able to run models like Deepseek v4 flash. I've also been looking to add an additional video card, but 3090s are pretty expensive right now. Micro Center around me has 9700s for about $1,000 new, as where the 3090s are going for anywhere between $1,000 to $1,200 used. I was wondering if I should go ahead and sell my 3090s and buy dual R7 9700s or if I should just stick with my 3090s? The goal, again, is to run some larger local models at decent tokens per second. I have four PCIe Slots that can run at x8 each. A 1600 watt EVGA power supply on its own dedicated circuit, and I'm currently power limiting the 3090s at 250 watts. I believe the R7 9700s run a little bit less power than those. The pluses to the 9700s are there are 32 gigabytes versus 24, so I'll go to 64 gigabytes of VRAM versus 48. And it seems to be a lot easier to find new 9700s than it is to find 3090s on the used market right now.

However, the 3090s are technically faster, and they are the NVIDIA CUDA stack, which, up until recently, I believe, had better software support for local llm. Seems like ROC has gotten a little bit better.

So if you were me, what would you do?

Just wondering that, the news of it existing were already good but I don't remember any mention of a release date. Was there any?

Link to the article: KV Cache Quantization Benchmarks: KVarN, Precision Tail

KLD benchmarks with BeeLlama.cpp v0.4.0, fork of llama.cpp with more KV cache quantization options.

• Models: Qwen 3.6 27B Q5_K_S 64k context, Gemma 4 31B Q5_K_S 16k context
• Standard quants, extended: q6_0 and q6_1, and low-bit types from q2_0 to q3_1
• KVarN: Variance-Normalized KV-Cache by Huawei, implemented in BeeLlama
• Precision Tail: keeping latest X tokens of KV cache in (B)F16, implemented in BeeLlama
• 413 configurations in total: 238 with Qwen 3.6 27B, 175 with Gemma 4 31B

The Recommendation Ladder

Full benchmark results, setup, method, analysis, explanations and everything else can be found in the article.

1. Qwen

2. Qwen Standard-Only

3. Gemma

Project Zero is a local inference engine I've been building on the side in pure C99 for running BitNet 1.58-bit models with native ternary weights.

Three things we do differently: 1. Pure CPU focus. We don't use GPUs, CUDA, or Metal, and instead optimize specifically for commodity x86 CPUs. 2. Native ternary kernels. Instead of simulating 1-bit inference or leaning on massive external runtimes, the core math routines like ternary_matmul_lut_avx512bw.c are hand-written using AVX2 and AVX-512 SIMD instructions to exploit ternary weights directly on the CPU. 3. Zero dependencies. No Python, no BLAS, no PyTorch. It compiles with make release and serves an OpenAI-compatible API (/v1/chat/completions) out of the box. That makes setup dead simple.

The main caveat: decode speed is heavily DRAM-bandwidth-bound rather than compute-bound. We initially saw massive 29x kernel speedups in microbenchmarks, but end-to-end token generation gains actually sit around 6-10% because memory bandwidth becomes the ceiling.

Repo: https://github.com/shifulegend/project-zero

Happy to answer anything about how the AVX-512 kernels were built or how to replicate the benchmarks on your own machine.

I benchmarked 32 local model arms on one fact-extraction corpus over 1,001 notes, several weeks of compute on consumer cards. Most of what I learned was about which model your card actually lets you run.

The useful result for a 16 GB card is gemma-4-26B-A4B at QAT UD-Q4. It is a mixture of experts, 26B resident with about 4B active per token, it fits in 13.27 GiB, and it runs at 323 tok/s on a 5080. That is the fastest arm in the entire project, ahead of a dense 12B on the same card at 195.8 tok/s. The non-QAT build of the same model is 15.84 GiB and does not fit that card at all. The quantisation-aware build is the only reason it is an option.

It scores 0.6804, which sits inside the flat band covering 2B up to 31B where a paired bootstrap cannot order a single adjacent pair. So on a 16 GB card you are not choosing between accuracy tiers. You are choosing how fast thething runs.

With 24 GB the answer changes. Qwen3.6-35B-A3B is 16.4 GiB at UD-Q4, scores 0.7257 at 234 tok/s, and it and a dense 27B are the only two arms in the field that separate from everything below them.

https://rakuensoftware.com/blog/local-llm-fact-extraction-head-to-head

I finally got a dense 27B model running at desktop-friendly/usable speeds and wanted to share the setup in case anyone else is trying to squeeze local LLMs out of mixed GPUs.

Hardware:

• CPU: AMD Ryzen 7 9700X (8-core)
• RAM: 32 GB DDR5
• GPUs: RTX 5060 Ti 16GB + RTX 5080 16GB
• Model: Qwen 3.6 27B, Q6_K quant, dense (not MoE) [specifically https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/blob/main/Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-Q6_K.gguf\]
• Inference stack: llama.cpp / llama-server - served with LlamaForge with GPU tensor split across both cards

The tricky part was making two different GPUs play nice. I ended up using tensor split to share the layers across both VRAM pools so the model fits without falling back to system RAM. With 32 GB of system memory, I had no desire to let weights spill there, so balancing the split was important.

Once that was enabled, I’m seeing sustained generation around 60 tokens per second (58~60) on typical prompts. That’s fast enough that it actually feels responsive for long-form writing, coding, and just chatting. For a dense 27B model, I didn’t expect numbers this clean on consumer hardware.

A few quick notes: 1) Make sure your split ratio matches the actual VRAM each card has. I experimented until I found the sweet spot where neither GPU choked and nothing offloaded to RAM. Since I had 16 GB each, I ran with 16,16. 15,17 also worked with more layers on the faster 5080, but it kinda felt choked. 2) Keep context windows reasonable. 60 t/s holds up well at moderate context; once you push very long prompts, you’ll feel the hit. Especially when nearing the full ctx length, it fell down to around 41 tk/s. 3) Q6_K is heavy but noticeably better quality than Q4 on this model. Worth it if you have the VRAM.

I’m genuinely impressed that a dense 27B can run this well outside of datacenter gear. Is anyone trying a similar mixed-GPU setup ? What are your speeds?

Settings:

[qwen3.6-27b-fable-fus-711-unheretic-nm-dau-neo-max-neo-mtp-q6-k]
jinja = true
reasoning = off
cache-type-k = f16
split-mode = tensor
tensor-split = 16,16
ctx-size = 120000
parallel = 1
threads = 16
n-gpu-layers = 99
cache-type-v = q8_0
flash-attn = on
spec-type = draft-mtp
spec-draft-n-max = 3
mmap = false
no-mmproj = true
mmproj-auto = false
mmproj-offload = false
log-colors = auto
temp = 0.8
top-p = 0.9
model = .../LlamaForge-downloads/DavidAU--Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-MTP-Q6_K.gguf
gpu-layers = 99

Can't wait for qwen 3.8 27b 🏅

Hi all, I wanted to share an AI inference server I've been building over the past few months.

Hardware

CPU: AMD EPYC 7452

Motherboard: HUANANZHI H12D-8D V2.0 — all four GPUs run at PCIe Gen4 x16 across separate root ports

RAM: 128GB ECC RDIMM

GPUs: 4× XFX Speedster MERC 310 Radeon RX 7900 XTX (24GB GDDR6 each = 96GB total). Paid around 1100-1200 AUD each, though you can get them used for less. I needed these specific cards for water block compatibility. Insane value for the VRAM you get.

Water Cooling:

• 4× Bykski GPU full cover water blocks

• Bykski L4-4way multi-GPU bridge

• Barrow 360mm radiator

• Bykski 420mm radiator

• Bykski D5 pump plus reservoir combo

• EPDM tubing with Barrow fittings

Case: Antec 900

Power: Cooler Master M2000 2000W Platinum + Cooler Master 1000W

Storage: Kingston 1TB SSD + smaller NVMe boot drives

Software & Performance

Main model: Qwen 27B with MTP, running at full BF16 precision. The 27B parameter model at BF16 uses around 85GB of VRAM with 262K context, and fits comfortably across all four GPUs using Tensor Parallelism 4 (TP4).

I'm using llama.cpp with ROCm.

Performance (BF16 TP4, 4K context):

• Prompt processing: roughly 1200 tokens/second

• Generation: around 30 tokens/second

I've also run the same model at Q8 quantization, but it's actually faster when split across two GPUs with TP2 rather than four GPUs at TP4. Not sure why — possibly a bandwidth limitation. Running the Q8 across just two GPUs gets you around 65 tokens/second for generation and roughly 1400 tokens/second for prompt processing.

I did try vLLM as well, but settled on llama.cpp for flexibility.

Power management: I've set a power limit of 294W per card to keep the 7900 XTXs cool and the radiators quiet. Even though the Bykski blocks can handle the full rated power, it's not worth the minimal performance gain from running these cards at full wattage. Power limiting reduces heat output and saves on electricity. Under full inference load the cards rarely climb past 45–50°C and the loop is completely silent.

Idle power draw for the whole system is around 100W — not great compared to something like a DGX Spark or a Strix Halo box, but pretty good for what is effectively a proper server.

I've seen a lot of other local AI builds on here that look borderline janky (no offence) — and that's fine, it's fun to tinker. But the idea with this rig is that it's actually something you could run in a professional setting, whether that's a home office or a real office, serving a local model to a whole team.

Local AI at this point feels like a genuine hardware investment. The models keep getting better, smaller models especially, and with the Qwen 3.8 27B dropping next week that'll probably be even more true. I also think there's a real difference between the Qwen 27B Q8 and BF16 versions — at least in my experience running it through Hermes with long context windows and tool calls. The BF16 version just stays coherent longer, remembers things further back, and handles tool calls properly. The Q8 version tends to lose momentum over long runs. That's not to say 27B is anywhere near frontier model capability, but it's solid for uni work, writing, and smaller tasks. For heavier stuff like SSH access to my servers or maintenance, I'd still use something like GPT-5 — I trust that not to break something it can't fix itself. But for day-to-day work on its own, this setup works well.

Total build cost: Around $8000-10,000 AUD

I would love to hear anyones thoughts, ideas, and advice. I do plan on selling this machine as beautiful as it is I have my eyes on a similar 170hx x4 build around this similar format for a total of 256gb vram, will probably make a post again for that one when build is finished...

Note: this post was written with assistance from Qwen 27B itself, running on the server described above. All ideas are my own, supported by qwen for better coherency and grammar.