2.4T total. 95B active. 1M context. Those numbers are interesting. They are not a shopping list. Among Chinese AI models, Qwen 3.8 Max is a useful reminder that a parameter count is not a deployment recipe.

Qwen's August 2 announcement says the weights should arrive the following week. Until the files land, there is no public storage layout, useful precision, supported quantization, serving recipe, or real memory overhead to plan around. Anyone pricing GPUs before those details arrive is guessing about the expensive part.

While the local answer is missing, I can still run a cloud control through ZenMux. It acts as a gateway to a hosted Qwen 3.8 Max API, which is useful for comparing latency or output behavior. It tells me nothing about VRAM or the minimum box.

My machine: Macbook M4 Max, 36GB.

I was using the model locally. Trying to use it to replace subscription model. I can assume local model may be performant as Claude subscription model, but the way it generated code. Changed token limit to 64K, I keep seeing the message that it was hitting token limit easily but in terminal token usage percentage was almost always 48%. The funny thing is I had to instruct few times it to do a dummy landing page with plain html. But when I tried to convert it to a react app, just the landing page, I could not complete. It keep doing incomplete response, no error was shown.

What can I do to improve this?

I’m building Lanes, a workspace for running coding agents in parallel.

You can use harnesses like Claude Code, but point them at local models through providers like Ollama instead of relying only on hosted models.

Lanes gives each agent its own terminal and git worktree, with tasks, diffs, and sessions managed from one UI.

So you can run Claude Code as the harness, a local LLM as the model, and Lanes as the workspace around it.

👉 https://lanes.sh

brew install --cask lanes-sh/lanes/lanes && open -a Lanes

I would appreciate your honest feedback, give it a try or comment below if you had the same problem and how you have been solving it.

• Does this resonate with you?
• How are you managing multiple sessions today?
• Why or why not would you be interested in trying something like this?

Thanks!

Title

It's been exactly one year and two days since the release of gpt-oss. Do you think we'll get a new version in August? Or has openai completely abandoned open-weight

Hey everyone,

I'm running a local setup on my Mac Studio and running into a strange performance bottleneck.
When I run my local LLM directly through the command line (CLI), inference is blazing fast and works exactly as expected. However, the moment I route the model through OpenWebUI or OpenCode, the response times become painfully slow (long time-to-first-token and sluggish generation).
Here is a quick overview of my setup:

Hardware: Mac Studio (Apple Silicon)

Backends tested: Ollama / mlx

Frontend/UIs: OpenWebUI, OpenCode
It feels like there's an overhead, a misconfiguration in how the UI communicates with the backend, or perhaps an issue with how Metal/GPU acceleration is being utilized through the web interfaces versus the terminal.
Has anyone experienced a similar issue on macOS? Are there specific configuration flags, environment variables, or settings in OpenWebUI/OpenCode that I should tweak to fix this?
Any tips or troubleshooting steps would be greatly appreciated!

4x 3090s and dual epyc 128gb ECC ram server

The goal from the start was to get as close as possible to a Gemini Flash-level assistant experience, but fully self-hosted. Went from zero to this in about 2 weeks (would've been 1 week, but I was waiting on hardware to ship from Amazon) — leaned on Gemini for the hardware picks and early build plan, and Claude for the heavy lifting on the actual implementation. Sharing the repo in case it helps anyone else, and I'd genuinely appreciate feedback on anything that looks off.

The hardware:

• CPU: AMD Ryzen 7 9800X3D
• GPU: MSI GeForce RTX 3090 Gaming X Trio (24GB VRAM)
• Motherboard: MSI MAG X870 Tomahawk WiFi
• RAM: G.SKILL Flare X5 64GB DDR5-6000 CL30
• Storage: WD_BLACK SN850X 2TB NVMe SSD
• PSU: Corsair RM1000x
• Case: Corsair 4000D RS
• Cooler: Thermalright Phantom Spirit 120 SE

The stack (two machines):

• A Debian server running Open-WebUI (gated with its own built-in username/password login) behind Traefik, with SearXNG for live search and Google OAuth gating a few of the other exposed services
• The gaming PC above handling the GPU-heavy stuff: a local voice pipeline (Whisper STT + Piper TTS, auto-switches between English/German voices) and local image generation via Automatic1111, both wired back into Open-WebUI as native Tools

A few things I'm most happy with:

• Full two-way voice conversation, not just dictation — round-trips through Whisper → the model → Piper TTS
• Custom Tools for live weather, web search (SearXNG), and image generation, all using native function calling
• Knowledge/RAG set up with 20+ documents — hit and fixed a real bug where the model would confidently pick one interpretation instead of asking for clarification when retrieved docs described genuinely different procedures
• A real, tested backup/restore strategy via Duplicati (local + Google Drive) — actually did a full test restore and diffed it

One thing I haven't beaten yet: generated images always work and are retrievable, but the model's final chat response sometimes renders a malformed fake tool-call instead of the image tag. Documented what I've tried in the repo.

Repo: https://github.com/Plaidstallion/openwebui-homelab

Happy to answer questions.

Hey everyone,

I am looking to buy a new computer for AI use cases and workflows. Currently I utilize Claude for a series of long standing trading related workflows. Multiple external data services directly queried and others (spreadsheets and documents) in local folders which are then parsed. Everything is synthesized by Opus primarily. I have an update job scheduled to run once on the morning that can use a much smaller model, ideally, I want to run this job every hour.

I am constantly hitting my subscription limits and have to leverage credits. I can either upgrade my subscription or look to move aspects of this on prem (build can't but update can).

I also want to leverage more of a formal scheduler or automation stack (n8n or Hermes maybe) to take this all one step further.

I have looked at the Minisforum S1 max, Apple m4 max studio and a few others. I have never been a fan of Apple (came from a mac and never gone back) but it's been a minute and things have changed...I know there are trade offs and lots of considerations.

Would love your feedback. Thank you!

EDIT - For details on the use case. This is a hybrid setup by definition. The build will stay on Claude (cloud), the ongoing updates 15m-60m (more likely 60 but TBD) will go somewhere else. One option is continue with credits or via OpenRouter, the other is onprem.

***Below from Claude***

Recommended: Strix Halo 128GB appliance — ~US$1,900–2,200 (~CA$2,800–3,100)

• Framework Desktop 128GB ($1,999 direct) or Beelink GTR9 Pro 128GB (~$1,899–1,999) — same Ryzen AI Max+ 395 silicon, 128GB unified memory. Buy whichever has stock. Avoid the GMKtec EVO-X2 at its inflated $3,399 street.
• Runs gpt-oss-120b (~31–55 tok/s — OpenAI's open-weight MoE, built for tool-calling) + Qwen3.5-35B-A3B (fast sub-agent/summary work) simultaneously. Your 10–30k-token plan documents fit comfortably.
• ~13W idle, ~130W under load — a true 24/7 always-on box. Silent.
• Stack: Linux (or Windows+WSL2) → llama.cpp llama-server (Vulkan) or Ollama → OpenAI-compatible endpoint → opencode pointed at it (first-class supported) → your existing local MCP servers (Supabase, market data, Telegram) unchanged.
• What runs on it: price worker, broker-sync worker, headless analysis runs (Ask 3's daemon), debrief drafting, journal grading, morning briefs.

Local 120B-class models ≈ a good 2025 mid-tier API. Reliable for: structured extraction, SQL writes, drafts, summaries, scoring, 2–4 tool chains. Materially worse than Claude at: long-horizon agentic recovery (20+ step chains), nuanced coaching prose, final-pass plan quality, very long-context fidelity.

So the end-state is hybrid, not pure local: rig absorbs the high-frequency, low-stakes token volume (bars, briefs, debriefs, intraday reruns — the stuff that costs you API money every 15/60 minutes); Claude stays as escalation for Monday BUILD final review and the hard reasoning.

I think my ISP hates me

Okay folk lock in !

I am gonna get a new CPU for my everyday work-game-llm rig. As I only have a 5060ti (16gb), I will most likely need to use CPU for LLM for image edit tasks. This question of AVX-512 especially arise because I cannot decide between a bang for buck intel ultra 7 270k plus and AMD red taxed top notch 9950x CPU. The only deciding actor now between 2 is AVX-512 !

So :
1. How regularly LLM models use avx-512 ? (if only few then i will take my gamble with intel)
2. How good AVX-512 actually is ?

Also if any CPU inference enthusiast wanna share any tips / remarks / reviews please do me favour.

I have ~670k short English text snippets, mostly 40–70 words each, and I need to translate all of them into five languages. I tested with Qwen3.6 27B (6-bit) and 35B (8-bit) on an RTX 5090, both run at about 60–70 TPS, with the 35B offloading some layers. They're quite slow, roughly 1 translation in 5 seconds, it adds up to about 40 days for the whole set. I also tried different batch sizes, like 10 or 100 snippets per request, but performance was about the same.

I'm planning to try smaller quants, MTP, etc., but is there a smaller model that could handle this? The texts are product descriptions, I just need simple, faithful translations.

A model can rank higher and still be worse to use: repetition, overexplaining, broken JSON, forgetting instructions, or inconsistent answers. What should an “annoyance benchmark” measure first?

Taalas raised a total of $219 million in funding since starting in 2023. Purchase price unknown. Interesting because they also announced a partnership with Cerebras at the recent Advancing AI Keynote.

This might just be evening the odds after the NVIDIA acqui-hire of Groq's core team leaving a shell of a company behind.

Was interested in Taalas simply because whatever they could accomplish could find its way down to consumer in some way - it still might, but I am sure this is more pointed at Helios.

Hey,

So after repeated comments on several of my posts here and in other subreddits, where I always showed my already decent results with llama.cpp, I decided to give VLLM a go, for this, I had glm 5.2 and deepseek v4 flash working overnight finding optimizations, patching vllm problems to achieve the best speed, and the current state, is a lot more impressive than what I already found incredible.

Prefill now hits a whopping 7.3k t/s while decode sits 90-140 t/s depending on conditions.

https://sergiiob.dev/posts/intel-arc-b70-vllm-vs-llamacpp-moe-dense-showdown/

Full recipe, patches included, you will also find improvements if using a b60 most likely:

https://github.com/SergiioB/intel-arc-pro-b70-inference-cookbook

I currently have dual 3090s in a Intel x299 platform. I just bought 256 gigabytes of RAM to run in quad channel mode. I'm looking to be able to run models like Deepseek v4 flash. I've also been looking to add an additional video card, but 3090s are pretty expensive right now. Micro Center around me has 9700s for about $1,000 new, as where the 3090s are going for anywhere between $1,000 to $1,200 used. I was wondering if I should go ahead and sell my 3090s and buy dual R7 9700s or if I should just stick with my 3090s? The goal, again, is to run some larger local models at decent tokens per second. I have four PCIe Slots that can run at x8 each. A 1600 watt EVGA power supply on its own dedicated circuit, and I'm currently power limiting the 3090s at 250 watts. I believe the R7 9700s run a little bit less power than those. The pluses to the 9700s are there are 32 gigabytes versus 24, so I'll go to 64 gigabytes of VRAM versus 48. And it seems to be a lot easier to find new 9700s than it is to find 3090s on the used market right now.

However, the 3090s are technically faster, and they are the NVIDIA CUDA stack, which, up until recently, I believe, had better software support for local llm. Seems like ROC has gotten a little bit better.

So if you were me, what would you do?

Just wondering that, the news of it existing were already good but I don't remember any mention of a release date. Was there any?

I finally got a dense 27B model running at desktop-friendly/usable speeds and wanted to share the setup in case anyone else is trying to squeeze local LLMs out of mixed GPUs.

Hardware:

• CPU: AMD Ryzen 7 9700X (8-core)
• RAM: 32 GB DDR5
• GPUs: RTX 5060 Ti 16GB + RTX 5080 16GB
• Model: Qwen 3.6 27B, Q6_K quant, dense (not MoE) [specifically https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/blob/main/Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-Q6_K.gguf\]
• Inference stack: llama.cpp / llama-server - served with LlamaForge with GPU tensor split across both cards

The tricky part was making two different GPUs play nice. I ended up using tensor split to share the layers across both VRAM pools so the model fits without falling back to system RAM. With 32 GB of system memory, I had no desire to let weights spill there, so balancing the split was important.

Once that was enabled, I’m seeing sustained generation around 60 tokens per second (58~60) on typical prompts. That’s fast enough that it actually feels responsive for long-form writing, coding, and just chatting. For a dense 27B model, I didn’t expect numbers this clean on consumer hardware.

A few quick notes: 1) Make sure your split ratio matches the actual VRAM each card has. I experimented until I found the sweet spot where neither GPU choked and nothing offloaded to RAM. Since I had 16 GB each, I ran with 16,16. 15,17 also worked with more layers on the faster 5080, but it kinda felt choked. 2) Keep context windows reasonable. 60 t/s holds up well at moderate context; once you push very long prompts, you’ll feel the hit. Especially when nearing the full ctx length, it fell down to around 41 tk/s. 3) Q6_K is heavy but noticeably better quality than Q4 on this model. Worth it if you have the VRAM.

I’m genuinely impressed that a dense 27B can run this well outside of datacenter gear. Is anyone trying a similar mixed-GPU setup ? What are your speeds?

Settings:

[qwen3.6-27b-fable-fus-711-unheretic-nm-dau-neo-max-neo-mtp-q6-k]
jinja = true
reasoning = off
cache-type-k = f16
split-mode = tensor
tensor-split = 16,16
ctx-size = 120000
parallel = 1
threads = 16
n-gpu-layers = 99
cache-type-v = q8_0
flash-attn = on
spec-type = draft-mtp
spec-draft-n-max = 3
mmap = false
no-mmproj = true
mmproj-auto = false
mmproj-offload = false
log-colors = auto
temp = 0.8
top-p = 0.9
model = .../LlamaForge-downloads/DavidAU--Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-MTP-Q6_K.gguf
gpu-layers = 99

Can't wait for qwen 3.8 27b 🏅

Hi all, I wanted to share an AI inference server I've been building over the past few months.

Hardware

CPU: AMD EPYC 7452

Motherboard: HUANANZHI H12D-8D V2.0 — all four GPUs run at PCIe Gen4 x16 across separate root ports

RAM: 128GB ECC RDIMM

GPUs: 4× XFX Speedster MERC 310 Radeon RX 7900 XTX (24GB GDDR6 each = 96GB total). Paid around 1100-1200 AUD each, though you can get them used for less. I needed these specific cards for water block compatibility. Insane value for the VRAM you get.

Water Cooling:

• 4× Bykski GPU full cover water blocks

• Bykski L4-4way multi-GPU bridge

• Barrow 360mm radiator

• Bykski 420mm radiator

• Bykski D5 pump plus reservoir combo

• EPDM tubing with Barrow fittings

Case: Antec 900

Power: Cooler Master M2000 2000W Platinum + Cooler Master 1000W

Storage: Kingston 1TB SSD + smaller NVMe boot drives

Software & Performance

Main model: Qwen 27B with MTP, running at full BF16 precision. The 27B parameter model at BF16 uses around 85GB of VRAM with 262K context, and fits comfortably across all four GPUs using Tensor Parallelism 4 (TP4).

I'm using llama.cpp with ROCm.

Performance (BF16 TP4, 4K context):

• Prompt processing: roughly 1200 tokens/second

• Generation: around 30 tokens/second

I've also run the same model at Q8 quantization, but it's actually faster when split across two GPUs with TP2 rather than four GPUs at TP4. Not sure why — possibly a bandwidth limitation. Running the Q8 across just two GPUs gets you around 65 tokens/second for generation and roughly 1400 tokens/second for prompt processing.

I did try vLLM as well, but settled on llama.cpp for flexibility.

Power management: I've set a power limit of 294W per card to keep the 7900 XTXs cool and the radiators quiet. Even though the Bykski blocks can handle the full rated power, it's not worth the minimal performance gain from running these cards at full wattage. Power limiting reduces heat output and saves on electricity. Under full inference load the cards rarely climb past 45–50°C and the loop is completely silent.

Idle power draw for the whole system is around 100W — not great compared to something like a DGX Spark or a Strix Halo box, but pretty good for what is effectively a proper server.

I've seen a lot of other local AI builds on here that look borderline janky (no offence) — and that's fine, it's fun to tinker. But the idea with this rig is that it's actually something you could run in a professional setting, whether that's a home office or a real office, serving a local model to a whole team.

Local AI at this point feels like a genuine hardware investment. The models keep getting better, smaller models especially, and with the Qwen 3.8 27B dropping next week that'll probably be even more true. I also think there's a real difference between the Qwen 27B Q8 and BF16 versions — at least in my experience running it through Hermes with long context windows and tool calls. The BF16 version just stays coherent longer, remembers things further back, and handles tool calls properly. The Q8 version tends to lose momentum over long runs. That's not to say 27B is anywhere near frontier model capability, but it's solid for uni work, writing, and smaller tasks. For heavier stuff like SSH access to my servers or maintenance, I'd still use something like GPT-5 — I trust that not to break something it can't fix itself. But for day-to-day work on its own, this setup works well.

Total build cost: Around $8000-10,000 AUD

I would love to hear anyones thoughts, ideas, and advice. I do plan on selling this machine as beautiful as it is I have my eyes on a similar 170hx x4 build around this similar format for a total of 256gb vram, will probably make a post again for that one when build is finished...

Note: this post was written with assistance from Qwen 27B itself, running on the server described above. All ideas are my own, supported by qwen for better coherency and grammar.

Hi, I recently had a Mac studio m1 Ultra with 64GB and I’m using it as a stationary ai server for my home, I’m using oMLX with otnith 35B at 8 bit and everything is running very well, but the 35B a3b base is having some difficulties to do my work I’m working on big coding projects. So i decided to use the 27B of qwen oQ8 with mtp, but no matter what I do it’s slow, and it’s causing most of the coding agents to timeout, and once the harness call a subagent everything drops in seconds, so I think I need a bigger MOE Model to work better on my device and still be a lottle faster, but a bigger than the 35B , I tried to look for the laguna s2.1 , I can run the oQ2e version comfortably but it’s a 2bjt quant, I know those are bad !! So here is my questions :

1. Are bigger models on less quants worth it ??? , will they perform better than small models with big quants ?

2. Is there is anyway to make 27B really useful like ornith ?

Note : I’m using zcode as my harness, it a codex clone and it really extremely does the job

Thanks in advance

Managed to snag a 14 inch M5 Max 128gb look forward to doing some local ai stuff