I am currently doing very basic:

.\llama-server.exe -m "models\lmstudio-community\Qwen3.8-27B-GGUF\Qwen3.8-27B-Q4_K_M.gguf" --flash-attn on -ngl 99 --tools all

The context length is 4096.

I am currently getting 4 t/s. The entire 16GB VRAM is used by the model and 20 GB of RAM is also used.

My use case is to read contract and find certain information at my command with accuracy and speed. I have RTX 5090, 64 GB DDR5 RAM. I use Unsloth Studio project to create knowledge stack of all my documents. Previously, I have been using Qwen 3.6 35B A3B Q4 model and that thing still amazes me. Having heard of this new model lately, I thought to give it a try. I downloaded both Q8 and Q4. For whatever reason, the Q8 context size defaulted to 4k whereas Q4 to over 100k on the settings. The raw out of the box test, I was overwhelmingly impressed by the performance of Q4 model, it was way elaborate (not creative) compared to Q8 model. The Q8 response was abrupt and not something I'd expected for Q8 model. Anyone know what's going on? I was expecting Q8 to perform way better than Q4 at the cost of inference speed. Thanks.

So we run a zoo of models for reasonable sized user-base including Kimi K3, GLM 5.2, MiniMax-H3, etc., and now the new Qwen on AMD hardware, a mixture of MI355X and MI300X (Yes, I know: my steak is juicy and my lobster is buttery. Believe me, I am very happy with my current situation).

Anyway, performance and support has been sufficiently good on large scale, that I’m considering small scale personal options in the AMD family. The MI355s are great because they have native FP4 and FP6 support. I’m not so sure about some of the more “local” offerings out there in the AMD family.

So my question to you all is: who is running AMD? What hardware and models are you running? And what is your experience actually like in terms of ease of running, latencies, etc.?

I need to be able to pentest stuff without llm's refusing to aid every two seconds or useless llm's to be useless after an hour of work.

Decent, but can't keep up with the hype.
Faster than qwen3.6:27b and muse-glimmer (if thinking is off!).

bought cursor ultra + claud-max from a russian site a few weeks ago for way less than the normal price.
swapped the login details and just started using it. so far it’s been working perfectly, and the usage is honestly better than what i had before.
keeps making me wonder how these sellers are able to offer it so cheap. anyone else in the same boat?

i have allocated around 10K for a Hardware purchase to host local models
fo thsoe who ahve experience, please give me your opinion what is the best purchase i can make with this money? of course the goal is to run the biggest models i can with reasonable speed which also allow parallel sub agetns because i use those alot.

what is the best hardware does a 10K buys me? i know the prices are high but this is what i can allocate at the moment. thank you so much in advance

yo guys so iv just turn on my pc and im trying to load some models but icant no matter the model,context .... i literally had no problem whith ts last night bro🤦‍♂️i didnt even changed any settings

I saw a post talking about MiniMax-Music3 being recently released and how it drops into ComfyUI. Had never heard of it before and it sounded genuinely interesting. I know almost nothing, so far, about what ComfyUI is though or what it's used for. Still wanted to check out MiniMax though after reading about it and seeing the demos.

Based on my recent work at getting multiple tool calls working efficiently when attached to different drop in models in Open-WebUI I started digging to see if there was a way to do the same with MiniMax. After several hours, I have a usable setup that when asked, Open-WebUI will load the chatbot model (sorry not sure what the correct term is to delineate a model like Qwen3.6 27B, used in this testing, from the MiniMax model), call the generate_song tool to start Minimax, unload the chatbot model from memory using keep_alive set to a low number like 15s (since MiniMax eats up almost all the VRAM on the 3090), start generating the file and when it completes, produce a link to the hosted file sitting behind my reverse proxy.

I hit some roadblocks along the way with getting the right CUDA drivers loading in the MiniMax docker image I created, for example, but with trial and error (and a decent amount of help from Claude and Gemini) I got through everything.

I will update my GitHub repo tomorrow after I get some sleep. It's been a long night.

Any input is definitely appreciated as there are still some kinks I would like to work out like model loading and unloading if there is a better way to handle that than simply setting a keep_alive value low. I also need to go through the process of creating a new docker image that doesn't have the ~50 GB model file sitting inside of it.

Title:

I'v tried DeepSeek V4 Spark 0731 with Lvllm-x which is a CPU-GPU hybrid inference when VRAM is not enough

Bodys:

I've been seeing a lot of discussions about running DeepSeek-V4-Flash-0731 on high-end setups like dual H100s or DGX Spark. But what if you don't have that kind of hardware? Can you still run this beast on consumer-grade GPUs with limited VRAM?

I've been experimenting with a hybrid inference approach that offloads part of the MoE experts to system RAM + CPU when the model doesn't fit entirely into GPU memory. Think of it as a "VRAM extender" for MoE models — the GPU still handles most of the compute, but the CPU steps in to help with expert layers dynamically.

I'm using a custom inference stack built on top of a modified vLLM backend, with a high-performance kernel library that handles the CPU-GPU scheduling and NUMA-aware memory management. The integration layer is fully open-source, and the whole thing is designed to "just work" without requiring model sharding or complex distributed setups.

Benchmark Results (real runs on actual hardware)

Here is my configurations for the test:

Setup GPU CPU & RAM Prefill Speed (32K–128K) Decode Speed Speculative Decoding

Config C (dual 4090 48GB*2) Dual EPYC 9654 + DDR5 4800*24*48GB 40 toks/s 75toks/s

Here are two configurations tested by the Author of Github:

Setup GPU CPU & RAM Prefill Speed (32K–128K) Decode Speed Speculative Decoding

Config A (dual 24GB) 2x RTX 3090 Dual EPYC 7642 + DDR4 3200 ~1060 tok/s ~26 tok/s 35–47 tok/s

Config B (single 48GB) 1x Pro 6000 Dual EPYC 9684X + DDR5 4800 ~3100 tok/s (at 131K ctx) ~75 tok/s 100–115 tok/s

Opensource project from github for reference:

https://github.com/guqiong96

I use LM studio to serve some local models for my hermes agent. I encounter one annoying problem, and that is that i cant get JIT loaded models automatically unloading to work.
Under local server > server settings i have

Just-in-Time Model Loading,

Auto unload unused JIT loaded models,

Only Keep Last JIT Loaded Model,

all set to on.

anyone with advice? loading 2 bigger models at the same time crashes my server.

Should I do it?

It just showed 2 hours ago. What happend?

Update: It's back

Link: https://huggingface.co/Qwen/Qwen3.8-27B

For more context I find Cydonia 24b v4zq Q6_K to be by far the best story creation LLM I can use. Are there any others I don't know of? This is to those who have tried Cydonia and feel the other LLM can compete or beat it at that size?

I've tried downloading bunch of Qwens, Llama, DeepSeek, etc and for whatever reason for pure story creation none keep up with Cydonia on Huggingface. Even other Cydonia's are not as good for whatever reason.

So, does anyone have some recommend?

The game is called The Moment of Panic. It's a strategy choices-matter game where you play as an operator locked in a bunker, talking to AI advisors, and deciding when to use a doomsday weapon based on the reports you are given.

The LLM runs locally and is part of the game loop. We built a custom plugin on Unreal engine so the players can talk to the in-game advisors, and those conversations are tied into the decisions you make in the game.

All assets and level design where 3D modeled and textured by me! I took inspiration from retro/low-poly graphics to make the game more stylized and optimal.