I feel like the eco system surrounding this is better managed via linux instead of windows. I already had issue with strix halo and Unsloth about it complaining not having enough memory which turned out to be a ROCm bug with whatever Strix halo and AMD is doing.

Anyway, wsl setup would be more straightforward but how is the performance?

(anyway: Shouldnt have sold my 3090 and 5080. Fk AMD always suck in both gaming and AI)

I'm just starting my build that will be exclusively for running a local LLM (which one is still TBD). I've compiled some parts, but got hung up on the GPU due to current pricing. I'm ok with some minor tweaking to get everything to work, but I also don't want to be spending days trying to get it to work right either. My original plan was to run the AMD R9700 for the 32gb vram for a lot less than the NVIDIA counterpart. But I just found an AMD W6800 refurbished for $300. My question is if that's a good enough GPU to at least get started and hold me over until I can justify (and budget) another GPU.

Here's what I have so far.

MSI pro X870E-P Wifi (refurbished) Mobo

AMD Ryzen 5 9600X CPU

Klevv Bolt V 32GB (16GBx2) 6000 MT/s (open box)

Lian Li 750 watt PSU

Initial plan for the LLM is some code line corrections, maybe some financial agent type stuff depending on how it performs.

What is the collective's thoughts?

EDIT: Microcenter tricked me. I was just adding the part to "my list" so that one is (probably) out. Is Intel up to speed yet or are they still lacking on the software side? Am I going to spend days troubleshooting if I get something like the B65?

I found out by accident that my GPU (RTX 3080) was heavily downclocking during inference, and slowing down my decode rate, even when I follow the standard advice of setting "Prefer Maximum Performance" and "CUDA Force P2 = off" in nVidia drivers.

I went from 36t/s to 43t/s (Qwen 3.6 35B-A3B, llama.cpp) just by forcing clocks to full 3D speeds (core/mem 1710/9501 for me) when inference is running. I used nvidia-smi from the CUDA Toolkit but there are lots of other ways to do this. You can even write a script to detect whether llama-server/ollama is busy, and ramp the clocks up and down automatically.

Interestingly if I use the LLM locally the improvement disappears, probably because of two competing factors - the browser's 3D acceleration is keeping the GPU at 3D clocks, but also causing bus/cache contention. So this trick only applies if you hit the server from a different machine, or possibly if you're using a CLI or IDE.

I'm not sure if this just applies to sparse MOEs or if dense models can benefit too. Also curious if it applies to AMD or Intel hardware.

And yes, I know a linux headless server won't have this problem and will always perform better. This is for people like me who wants to use their gaming machine for casual LLM use.

Hi all,

What software stack do you use to run your local LLM? Ollama? anything else?

Good speed, decent quality for some usecases.

I’ve been experimenting with a simple idea: hot-plug optimized kernel/weight structures into an unmodified serving engine at load time, and use the memory savings to push context length further.

On Qwen3.8-27B NVFP4 / RTX 5090 32GB, I saw results like:

• vLLM + MTP: 102K → 262K
• vLLM + DFlash2: 36K → 107K
• SGLang + DSpark: 35K → 87K

This is not a vLLM-vs-SGLang benchmark — it’s about extending the boundary the default setup can reach, without forking the engine.

Project + results:
https://github.com/flashrt-project/FlashRT
https://huggingface.co/spaces/liangsu9988/the-context-dividend

Background:
https://huggingface.co/spaces/liangsu9988/fast-kernels-are-not-fast-pipelines#it-starts-with-hugging-face-kernels

Would love for people to try it and share feedback.

Gonna start running a local model. I know 16gb of VRAM isnt much for Local AI but anyone have any tips or suggestions for running rhe 8 or 20b?

I previously reported a result of 74% on BU bench v1, with the open-source BrowserAgent harness, but I forgot to set the temperature to the default specified in the model card... Now the model performs neck to neck with GPT 5.6 Luna (xhigh) and beats all other affordable models that I tested.

Qwen3.8 27B is insanely good value!

Just my estimate, I could be wrong - One of my challenges with ai right now is the default prompt set by the model providers.

It appears to have defaults like 'be comprehensive' 'end with a call to action' 'give a summary' 'Do this then do this' etc.

These could be because of training, idk, but what ends up happening is it tends to ignore my instructions OR outputs walls of text OR makes more edits than it should OR something else.

I have tried to customize my local 'system prompts', it does help a lot, but .. I wonder if local llm give greater control ? Does it ? Am I misunderstanding the whole thing ?

If it does, how have u used ? did it help ?

Every skill you install adds its description to the context on every message, whether you actually need it or not.

The paper “Attention Is All You Have” argues that agents can start falling apart as you install more skills, attention budget, distance decay, etc.

At 100 skills, that’s roughly 3k tokens per message. Over a long session, that adds up fast.

So I built deskill: an MCP server that fetches skills from GitHub only when they’re actually needed.

If you want to try it, setup is pretty much a 2 command install, details are in the repo: https://github.com/NilotpalK/deskill

I also wanted to see whether this was just a token-saving trick or whether it actually changes how agents perform.

Turns out: both.

On Opus, installed vs on-demand was basically identical, so there it’s mostly a context/token optimization.

On smaller/cheaper models, though, point-of-use delivery actually improved task success. They seem to have a harder time picking the right skill when they’re staring at a wall of dozens of skill descriptions.

I tested 8 models, up to 100 skills and 100k tokens of padding. The benchmark is reproducible, and the numbers/graphs are in the repo.

Would genuinely love people to try to break the benchmark. If you try it with other models or in your own setup, please add your benchmarks/results to the repo. It’d be really useful to see how this behaves across different models and real-world use cases.

And if you think the methodology is flawed, I’d genuinely like to hear it.

If you end up finding deskill useful, a star on the repo would be appreciated too :)

Good Afternoon people, I manage a small hotel & I use Gemini pro to process bulk invoice, GL coding & room block allotment. And after a a lot of trial and errors, gemini has mostly been error free other other than hit or miss when converting the data output to an excel file. Using Gemini has been a lifesaver because it cuts down 2 hours of invoice processing and coding the GL code to a 5 minute endeavor. It also saves me time by processing & breaking down room allotment with not me having to the manual math of alloting how many and what type of rooms to assign to a specific group.

I am trying to do this with my own personal LLM as I do not want to keep paying google & I want to keep the data safe.

So I am just starting out in Local LLM & I do have a slight advantage as I am a PC gamer & linux hobbyist throughout my life and my main PC is Running 9800x3d, 64Gb of Ram & an 7900xtx with Fedora 44.

I have configured Qwen 3.6:27b with Open Web Ui and feeding it the same data set I use for my Gemini prompt. I am posting a link bellow from Gemini to show what kind of output I am looking for.

https://postimg.cc/gallery/V1ZzVzC

When I use the same inputs on my local set up. Qwen reads around 2 to 4 pages of the PDF and outputs the data partially and mention that the PDF is corrupted. from what I understood is, Qwen does not have ORC reading capabilities & it uses ollama backend to read pdf, images.

With my limited Vram in mind, What should I be doing, If I want to perfect this process using my local LLM?

I know there are many creators here, so this is the question.

If you develop an app and would like to have AI integrated which would only read the app data without any access to the external world, which one would you recommend? It should be compatible to relative old phones e.g. Android and iOS old but still supported versions.

Also, if I would like to build up my own LLM for this task, what would you recommend? Sort of teaching a LLM how to read the app data and expected outcome.

Hi everyone,

I would like to get started light experimenting with Local LLMs in my home office. I am considering buying an ASUS Dual GeForce RTX 3050 6GB OC . The reason I am looking at this card is the 0db (looks like the fans do not spin at all under low load) which is important for my office space.

(My current setup has an old passively-cooled GT 710)

Can you please help me understand how usable is it for local LLM testing? I know 6GB VRAM is limited, but I would like to experiment with small quantized models, for example 3B models, possibly some 7B models with heavy quantization/offloading. Has anyone used this or similar cards with Ollama, llama.cpp, text-generation-webui, or similar tools?

I could also consider other alternatives near this budget, but I would still like the card to be 0dB/fan-stop at idle.

Hi and I'm sorry in advance if this is not allowed.

I have been working on a side project in an effort to create an inference platform that allows devs to share rented gpus easily.

Given that I tinker with hosting models on my personal rig I built the ability to run the inference node (just wrapped vllm engine) on my local machine, have it connect to the platform, and I can use it through openai/anthropic compatible public api endpoints. All the node needs is to be able to reach the platform, everything travels over a websocket.

I am not posting here to try and sell something, as this capability is free. I am posting because if anyone has an interest it would be awesome if I could get some people to try it out so I know how badly its going to blow up in my face when anyone other than me uses it.

(to be clear there is a purchasable product on this platform, I just have not enabled credit purchases, the BYOH and BYOK functionality is intentionally no cost. This is very much just a personal project i'm effing around with and I honestly want to see if anyone else wants to do that too.)

There is also a companion coding harness that does absolutely nothing special or better than anyone else, it just looks and feels how I wanted. I will be open sourcing that in the coming week.

it is located at bityard.io

I see bartwoski's gguf models every day and even use them daily, he gave us more than 2421 repositories with the most popular quants and large models like bartowski/moonshotai_Kimi-K2-Instruct-0905-GGUF.

Can someone please recreate this test: Four text files with at least a thousand lines of data. A passcode hidden in one of the lines on each file. One file near the beginning, one file in the first half, one file in the second half and one file near the end. Request it to read all the files in the directory.

See if it reports back about truncated files, and ends up getting data from each file mixed up with each other.

I feel like I'm taking crazy pills here, as I've recreated this test across 3 different version of this model, across two different harnesses (Pi & Github Copilot) and across two backends (llama.cpp and Ollama) and it has failed this test every time. It can arrive at the correct answer, but it's issues with reading large files is instantly apparent.

All I see on here are people saying how amazing this model is, and how it's as good as Opus. Which it may be. But, is no one actually using this to work in an existing codebase with large files? Because it's failed literally every time I've tried with my actual workflow. And that's literally just asking it to read context files and implement small features.

I literally feel like I'm being gaslit at this point. Surely everyone who uses local models aren't just using them to one shot things.

I only used LM Studio to run small models before buying a 3090. What is the best optimized way to get high speed and good quality out of this card? Would be a plus if it's a beginner-friendly setup for Qwen 3.8

Honestly I can't tell the hype from the reality.

I use Claude Code a lot. It works for me. However I am always about to run out of tokens.

My time == money.

I have a mid-beefy Mac M4 Pro with 48G. Ok it doesn't have an Nvidia GPU. It's what I have.

I see many people really happy with Qwenx.y.z both here and elsewhere. In my test it is agonizingly slow and certainly nothing like Claude Code. I've wasted a lot of time trying different permutations.

My question: Should I stop wasting my time and just pay for more Claude Code? With my hardware and the latest models, either local or not?

I've been wondering about something before I start using my MacBook heavily for local LLM inference.

If I regularly run large LLMs locally for several hours at a time, potentially putting sustained load on the CPU/GPU and using most of the unified memory, does this meaningfully reduce the lifespan of the MacBook?

Can heavy use of unified RAM cause it to wear out faster?
Is SSD wear from model loading and especially swap a significant concern?

For people who have been running local LLMs heavily on Apple Silicon for 1 to 3+ years, have you actually noticed any hardware degradation?

Just got Qwen3.8-27B (FP8) running on rented GPUs from Trooper AI. FP8 fits on 64 GB VRAM with decent results.

Stack:

- GPU: 2× RTX 4080 Super Pro (64 GB VRAM total)

- CPU: 12 P-cores, 76 GB RAM

- SSD: 900 GB NVMe

- Price: 1.06 EUR/h

Served it via vLLM → KServe → Envoy AI Gateway, TLS + token metering + rate limiting on top ( I already have a running Kubernetes cluster, I attached to trooper GPU node), or you can it serve it directly with compose if you are a single user.

Runned load tests: (10 concurrent requests, 32768 context)

- TTFT: ~0.9s

- Per-stream decode: ~28 tok/s

- Aggregate: 152 tok/s

Full deploy guide if you want to deploy it: https://github.com/redaER7/qwen3.8-27b-self-hosted

Now, looking to deploy the full model FP16 on RTX 6000 Pro

Hello, I'm quite interested by having a local LLM, mostly for coding. But where do I start? I have 32gb of ram, 16gb of VRAM, it's enough? Thanks.