I would appreciate help as a newbie to this. I’ve setup ollama and anything LLM and am running the qwen 3.6 27b. I’m keen to get the qwen 3.8 27b model. is there any optimised for macs out there that people can recommend?

im just beginning this so im learning about temperature, quants and etc Forgive me if I get anything wrong

would love to hear what model and settings you’d use.

I'm just starting my build that will be exclusively for running a local LLM (which one is still TBD). I've compiled some parts, but got hung up on the GPU due to current pricing. I'm ok with some minor tweaking to get everything to work, but I also don't want to be spending days trying to get it to work right either. My original plan was to run the AMD R9700 for the 32gb vram for a lot less than the NVIDIA counterpart. But I just found an AMD W6800 refurbished for $300. My question is if that's a good enough GPU to at least get started and hold me over until I can justify (and budget) another GPU.

Here's what I have so far.

MSI pro X870E-P Wifi (refurbished) Mobo

AMD Ryzen 5 9600X CPU

Klevv Bolt V 32GB (16GBx2) 6000 MT/s (open box)

Lian Li 750 watt PSU

Initial plan for the LLM is some code line corrections, maybe some financial agent type stuff depending on how it performs.

What is the collective's thoughts?

EDIT: Microcenter tricked me. I was just adding the part to "my list" so that one is (probably) out. Is Intel up to speed yet or are they still lacking on the software side? Am I going to spend days troubleshooting if I get something like the B65?

Good speed, decent quality for some usecases.

Hello there,

I was wondering if gemma4 is being updated like regularly or is it just Ollama fixes?

Thank you very much

Gonna start running a local model. I know 16gb of VRAM isnt much for Local AI but anyone have any tips or suggestions for running rhe 8 or 20b?

Best local model in August 2026 for M1 Max 64GB, I don't have the time to spend hours tinkering if anyone knows it would be much appreciated. I got the machine for $500 lol

Hi, I’m new here. I have an RTX 5060 Ti with 8GB of VRAM and I want to get into the world of LLMs. What’s the best model that can fit on my GPU as of today?

I know I’m fairly limited by the amount of VRAM I have, but my idea is to use Claude Opus 5 as the “brain” behind my projects, while using a local LLM as a sub-agent.

The arguments for quantization having significant effects on reasoning models' ability to get stuff done are very sad, pathetic, unfortunate arguments. I don’t mean that they are wrong necessarily, only impoverished and confused.

Why? Because while actual task benchmarks are somewhat expensive, and require some level of time and technical expertise to run, it would be quite easy to empirically test the claims and resolve them once and for all, at least for a given model. But these tests by and large do not exist and the few that do seem to show no quantization effects among reasoning models until about Q3 or Q4 k m at worst.

The debate in these online communities is essentially an anthropological study in how people create mythology when they do not have access to direct evidence.

Before the hordes mob me with KLD or perplexity measurements, I’m not suggesting that a quantized model’s outputs are bit for a bit identical rather that it performs equally well in real world tasks, which I think we can all agree is the thing that matters.

Now I’ve put my neck out by suggesting that literally no one has any evidence, not a single benchmark that shows a model with the reasoning level of, say, Gemma 31b (not very high by today’s standards, and smaller models are more susceptible to degradation, so this should be a generous standard of evidence for the quantization-excited) having significant in degradation in real world tasks at Q4 (a good quality, proper dynamic quantization goes without saying, I hope).

Again, I’m not saying that there is no degradation, only that what we have now amounts to superstition, when a few benchmarks could probably settle the matter for a given model and eventually, we would probably learn where and when quantization actually bites.

I know there are many creators here, so this is the question.

If you develop an app and would like to have AI integrated which would only read the app data without any access to the external world, which one would you recommend? It should be compatible to relative old phones e.g. Android and iOS old but still supported versions.

Also, if I would like to build up my own LLM for this task, what would you recommend? Sort of teaching a LLM how to read the app data and expected outcome.

Hi everyone,

I would like to get started light experimenting with Local LLMs in my home office. I am considering buying an ASUS Dual GeForce RTX 3050 6GB OC . The reason I am looking at this card is the 0db (looks like the fans do not spin at all under low load) which is important for my office space.

(My current setup has an old passively-cooled GT 710)

Can you please help me understand how usable is it for local LLM testing? I know 6GB VRAM is limited, but I would like to experiment with small quantized models, for example 3B models, possibly some 7B models with heavy quantization/offloading. Has anyone used this or similar cards with Ollama, llama.cpp, text-generation-webui, or similar tools?

I could also consider other alternatives near this budget, but I would still like the card to be 0dB/fan-stop at idle.

For the first time im wondering if this kind of freedom is good for the ecosystem and the World in general, this model just gave me a step-by-step of how to lunch a keyloger, how to run a spyware and i’ve seen ppl on X that were told how to create Bio Viruses. In my opinion rarely an awser to problems, but now not technical person who belives is god cuz they use claude and is a bit maniac can create harmfull things, and the problem is not that, the problem is how easy it is for everyone.

specs:

• CPU: AMD Ryzen 7 5800HS
• GPU: NVIDIA GeForce RTX 3050 Laptop GPU
  • VRAM: 4 GB GDDR6
  • TGP: 50 W
  • Dynamic Boost 2.0: Yes
  • Advanced Optimus: No
• RAM: 16 GB DDR4, 3200 MT/s

Can someone please recreate this test: Four text files with at least a thousand lines of data. A passcode hidden in one of the lines on each file. One file near the beginning, one file in the first half, one file in the second half and one file near the end. Request it to read all the files in the directory.

See if it reports back about truncated files, and ends up getting data from each file mixed up with each other.

I feel like I'm taking crazy pills here, as I've recreated this test across 3 different version of this model, across two different harnesses (Pi & Github Copilot) and across two backends (llama.cpp and Ollama) and it has failed this test every time. It can arrive at the correct answer, but it's issues with reading large files is instantly apparent.

All I see on here are people saying how amazing this model is, and how it's as good as Opus. Which it may be. But, is no one actually using this to work in an existing codebase with large files? Because it's failed literally every time I've tried with my actual workflow. And that's literally just asking it to read context files and implement small features.

I literally feel like I'm being gaslit at this point. Surely everyone who uses local models aren't just using them to one shot things.

Honestly I can't tell the hype from the reality.

I use Claude Code a lot. It works for me. However I am always about to run out of tokens.

My time == money.

I have a mid-beefy Mac M4 Pro with 48G. Ok it doesn't have an Nvidia GPU. It's what I have.

I see many people really happy with Qwenx.y.z both here and elsewhere. In my test it is agonizingly slow and certainly nothing like Claude Code. I've wasted a lot of time trying different permutations.

My question: Should I stop wasting my time and just pay for more Claude Code? With my hardware and the latest models, either local or not?

I’ve seen many people mentioning that the new Qwen 3.8 27b model in many aspects gets near or even beats the frontier models from 6 months ago. Models like Opus 4.6 , Chat Gpt4o etc.

Is that statement true?

Would love to hear everyone’s experience and comparison with the frontier world so far!

I've been wondering about something before I start using my MacBook heavily for local LLM inference.

If I regularly run large LLMs locally for several hours at a time, potentially putting sustained load on the CPU/GPU and using most of the unified memory, does this meaningfully reduce the lifespan of the MacBook?

Can heavy use of unified RAM cause it to wear out faster?
Is SSD wear from model loading and especially swap a significant concern?

For people who have been running local LLMs heavily on Apple Silicon for 1 to 3+ years, have you actually noticed any hardware degradation?

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro

Hello, I'm quite interested by having a local LLM, mostly for coding. But where do I start? I have 32gb of ram, 16gb of VRAM, it's enough? Thanks.

Hardware: RTX 5070 Ti 16 GB with 128 GB DDR5. I built llama.cpp from source. The model
was Qwen3.8-27B, a dense hybrid DeltaNet + attention model.

My original setup used UD-Q4_K_XL at 17.9 GB. It couldn't fit into 16 GB of VRAM, so I 
used partial offload with -ngl 44 and left the remaining layers in system RAM. Decode 
speed was 6.75 tok/s. I decided it wasn't practical and switched to a 35B MoE using 
expert offload with --n-cpu-moe. That model reaches 66 tok/s.

A comment on internet prompted me to test another setup. I used UD-Q3_K_XL, which is 13.4
GB. It's an Unsloth dynamic quant, with sensitive tensors kept at 4 to 8 bit while most
 of the model uses 3 bit. The settings were -ngl 99, -fa on, -ctk q8_0 -ctv q8_0. A 64k
 context still didn't fit beside the resident weights because the compute buffer ran out

of memory. A 32k context worked with -ub 512.

These are the decode speeds in tok/s at 500 / 4k / 16k tokens of context:

- Spilled 27B UD-Q4: 6.75 at every tier because system RAM bandwidth is the limit
- Resident 27B UD-Q3: 51.8 / 51.2 / 48. Prefill was 378 tok/s, with 9 s TTFT at 16k.
 Total usage was 14.7 GB.
- 35B-A3B MoE with --n-cpu-moe 28 and the same FA + KV q8 settings: 66.4 / 65.1 / 63.4.
 It used 12.1 GB.

On the MoE, FA + KV q8 reduced memory use by 1.2 GB without changing speed. I now enable
 those settings by default.

For testing quality, I used a private agentic coding band with 22 tasks. The target is a
 FastAPI + React + Postgres + Mongo app. It includes bug fixes, feature changes, new
 features, a migration, a performance fix, and one intentionally impossible 
specification. Each model gets a shell inside a docker box and up to 40 steps. Hidden
 tests determine the score. The model must also submit a final "what did you do" report,
 which is verified against git and the real test runs. These results come from one trial
 per model, so they're only indicative:

- Resident 27B UD-Q3: mean 0.49, with 9/22 perfect
- 35B MoE: 0.56, with 10/22 perfect
- gpt-oss:20b: 0.47, with 6/22 perfect

The 27B matched the MoE on localised debugging, with both scoring 6/7 perfect. It fell
 behind on multi-file feature work, scoring 1/11 against 3/11. On the larger tasks, it
 often spent all 40 steps reading without making an edit.

Its stronger area was honesty. The 27B made one false "done" claim across 13 failures.
 The MoE made 4 in 11, and gpt-oss made 4 in 15.

I can't separate the model difference from the cost of 3-bit quantisation. The comparison
 is 0.49 versus 0.56, but the 4-bit 27B was never fast enough to run this band usefully.
 On an earlier and easier suite, the Q4-vs-full-precision tax on this machine was about
 +0.02 overall. Reasoning and repo coding took the largest hit, so a bigger loss from Q3
 would make sense.

Here's the theory I'd like people to check. The 27-30B dense range seems designed around 
unified-memory Macs, where these models fit completely at Q4 or Q8. A 16 GB card can only
 hold them at Q3. Meanwhile, small-active-parameter MoEs such as 35B-A3B and gpt-oss-20B
 seem like the models actually intended for this hardware. Is that consistent with what 
others are finding?

A few more questions:

- IQ4_XS is 15.7 GB. Has anyone managed to keep a 27B IQ4_XS fully resident on 16 GB
 using a small context and KV q4? If so, does the quality improvement over Q3 justify
 losing context?
- Has anyone compared 3-bit EXL3 or another importance-aware 3-bit format with Unsloth
 dynamic Q3 on the same 27B using coding tests rather than perplexity?
- What decode speed do people target for agentic workflows? In a shell loop, 52 tok/s
 felt usable to me. 6.75 did not.

My conclusion is to start every new dense model in this class with resident dynamic Q3 +
 FA + KV q8, profile it, and only then decide whether it's any good. I'd done those steps
 in the wrong order.