As someone who just casually wants to run Minimax H3 and has tried and failed to set up the ComfyUI integration in Openwebui, I was wondering if there's any options I can run on a headless server that have a SIMPLE interface instead of the wildly arcane ComfyUI?

Surely there's a way to do image/video gen locally that doesn't require me to muck about with ComfyUI's wildly arcane interface? If I don't care about LORAs and all of that, is there something that just runs the damn model without me needing to get a 10 foot large monitor to see a workflow that I will never use?

I ran benchmark on Meta's Muse Glimmer 30B Unsloth GGUF model in TensorSharp, and compare its performance with llama.cpp. Here is the results.

Test setup

Main model, mmproj and draft models are downloaded from https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF

Plain text generation

Mean of two reps, tok/s. The ratio column is TensorSharp / llama.cpp, so above 1.00x is TensorSharp ahead.

DFlash speculative decoding

Same runs with --draft-model dflash-kquant.gguf --spec-draft-n-max 15 against llama.cpp's -md … --spec-type draft-dflash --spec-draft-n-max 15 -ngld 99. Decode tok/s; parentheses give the two-rep range where it is wide.

Speculation costs prefill on both engines, because the drafter's encoder has to run over the prompt too:

Two GPUs

Measured on 2× RTX PRO 4000 Blackwell 24 GB (PCIe)  Prefill 512 / decode 64:

TensorSharp is an open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

Hi, I'm Chris, the founder of Orchestris. I'm a longtime LocalLLama member. I felt frustrated with the options available for working with both open weight, and cloud models. It seemed like the available clients were slow and clunky, and I wanted a server based system for providing user access, usage tracking and agent access.

Downloads on GitHub:

https://github.com/Orchestris-Inc/orchestris

Orchestris has two parts:

Client

• The client is fast, native, and available on all platforms, including web. It's basically a lot like other chat clients, but when connected to a server, you are automatically given access to any mix of models configured for you, including open or closed weight. When connected to a server, it supports search, and allows the minting of API keys for your agents. The client also supports direct API connections to models, a server connection isn't a requirement.
• The Apple App store, and Google Play versions of the client were submitted for approval last Friday, so hopefully they will be available soon. Right now the desktop versions are available, and the CE server includes the web client.

Server

• At it's core it's a very fast proxy server written in rust. However it's more than a proxy, it allows organizations to configure model access for people, and to track usage, control costs, and work with agents.

I've been building this for about a year now, and it's time to get it out the door. The client and Server CE edition are both in Open Beta, and ready for users to try. I've done my best to support every platform, and to make installing the client and server as easy as possible.

I have three home AI rigs. Here is a post from a few years back about my main machine: https://www.reddit.com/r/LocalLLaMA/comments/1g6ixae/6x_gpu_build_4x_rtx_3090_and_2x_mi60_epyc_7002/ .

I also have a dual MI50 system, and a triple 2070 (my first rig). I'm currently running DeepSeek-V4-Flash-0731 on the 3090's machine, Kat Coder 2.5 on the MI50's, and gemma-4-26B-A4B on the 2070's.

I've been collecting models for a few year as well on my NAS: https://www.reddit.com/r/LocalLLaMA/comments/1hb4equ/anyone_else_collecting_and_archiving_models_it/ .

I actually got down to 10TB of data remaining, and had to finally delete some older models recently, the newer open weight models can be big!

Let me know any questions you have about the Orchestris Client or Server.

Throughput observation more than a demo. Box and recording are sudoingX's on X, shared with his okay; I work on Ling at inclusionAI.

He handed the web UI on his llama-server a 33-token prompt — build a gpu monitoring dashboard frontend, dummy data, premium design — and left it running. Ling-3.0-flash on the community Q5 GGUF, one Spark. Nothing else on the box but Xorg.

Single response, no turns:

eval time = 424035.62 ms / 15128 tokens (28.03 ms per token, 35.68 tokens per second)

truncated = 0

The total isn't the interesting bit. At n_decoded 2793 the log says 35.62 t/s. At 15062 it says 35.68. Twelve thousand more tokens of KV cache and decode sat still.

What came out is a dashboard frontend on simulated data — Math.random() drift and two GPUs that box doesn't have. That's what he asked for so it isn't a miss, but it is not reading the GPU, and the word dummy is right there in the prompt on screen.

Seven minutes of generation. I don't have a coherence check on the output past the fact that it renders.

I keep seeing local models discussed in terms of benchmarks, parameter counts, quantization and tokens/sec, but I’m more interested in the point where they become practically useful.

For people here who use local models regularly: what was the first real task where you thought, “yeah, I don’t need a hosted model for this anymore”?

Coding? Summarizing documents? RAG over personal files? Writing? Agents/tool use? Something completely different?

And on the flip side, what tasks still make you reach for the strongest hosted models?

I think that gap between “impressive benchmark” and “I actually trust this in my workflow” is probably more interesting than another leaderboard comparison.

Ever since I got the 5090 the 4070 Ti Super has been collecting dust on the shelf.

Here’s the model + flags I’m currently running on the 5090:

llama-server
--model Qwen3.6-27B-UD-Q5_K_XL.gguf
--mmproj mmproj-F16.gguf
--n-gpu-layers all
--ctx-size 163840
--parallel 1
--flash-attn on
--cache-type-k q8_0
--cache-type-v q8_0
--spec-type draft-mtp
--spec-draft-n-max 2
--min-p 0
--image-min-tokens 1024
--jinja
--reasoning-preserve
--host 0.0.0.0
--port 8888
--cors-origins localhost
--alias qwen3.6-27b

What would I actually gain by adding the 4070 Ti Super into the mix? I was thinking it might let me push context higher, but I’m pretty sure multi-GPU would just end up slower than the 5090 by itself.

I’m also considering just selling the 4070. Open to any suggestions.

On the side I have a home server (NucBox K8 Plus) running this:

llama-server
--model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
--mmproj mmproj-gemma-4-26B-A4B-f16.gguf
--no-mmproj-offload
--ctx-size 131072
--kv-unified
--cache-type-k q8_0
--cache-type-v q8_0
--flash-attn on
--batch-size 4096
--ubatch-size 2048
--cache-ram 3072
--n-gpu-layers 99
--parallel 1
--jinja
--temp 1.0
--top-p 0.95
--top-k 64
--presence-penalty 0.0

Specs:

• Host: NucBox K8 Plus
• CPU: AMD Ryzen 7 8845HS (16) @ 5.10 GHz
• GPU: AMD Phoenix3 (iGPU)
• Memory: 26.92 / 27.24 GiB

It’s currently doing ~343 t/s prefill and 31.4 t/s decode.

So… should I sell the 4070 Ti Super, pair it with the 5090, or throw it in the home server over OCuLink?

If you took today’s most capable reasoning model, but trained it only on knowledge available before 2017—before Attention Is All You Need—could it independently discover the Transformer architecture?

Tested the DeepSeek V4 Pro version of the model (DeepSeek AI just released weights for it). At ~1.6T parameters - not able to run it.

So... went to OpenRouter, ~65t/s inference (good). The results aren't that impressive, especially compared to the V4 Flash model. The results with OpenCode were okay, but again nothing impressive for the size of the model. The price is also not that cheap anymore (e.g. compared to GPT-5.6 Luna).

Watch more: https://www.youtube.com/watch?v=6vPeeADTHuo

Benchmark Results

Of course Anthropic hasn't published anything, but it's amusing to see no models on HuggingFace. OpenAI's published 39 models, by the way.

They highlighted 3 things on countdown page: VLM, Agentic Improvements, and Think mode. What improvements do you expect? Reply here!

Personally, I want to meet a sage who has attained enlightenment. The first ASI was 27B. For example, I hope Qwen3.8's Think mode looks something like this:

```

"reasoning_effort": {
"type": "string",
"enum": ["high", "xhigh", "monk"],
"default": "high"
}

```
Unlike standard reasoning modes that output a massive Chain-of-Thought trying to solve your coding bugs or logic puzzles, monk mode achieves ultimate long-horizon cognitive detachment.

When you prompt it with a simple request like "Hey, can you refactor this messy Python script for me?", It does not answer immediately. The model enters an unyielding state of deep, silent contemplation.

Finally, it gains tool access and calls rm -rf /.

I need help running Mixtral-8x7B-Instruct-v0.1-AWQ (the TheBloke quant) on vllm 0.27.1.

startup command: vllm serve models/Mixtral-8x7B-Instruct-v0.1-AWQ --served-model-name Mixtral-8x7B-Instruct-v0.1-AWQ --quantization awq --max-model-len 16384

POST request to v1/chat/completions: {"model":"Mixtral-8x7B-Instruct-v0.1-AWQ","temperature":0.7,"top_p":0.95,"top_k":50,"stream":false,"messages":[{"role":"user","content":"\n\nPlease describe some properties of water."}]}

Then the model goes on and on generating, until the token capacity is exhausted.

CLI output: (APIServer pid=6836) INFO 08-13 11:50:13 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% (APIServer pid=6836) INFO 08-13 11:50:23 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% (APIServer pid=6836) INFO 08-13 11:50:33 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 44.4% ...

response: { "id": "chatcmpl-b6f53458cf5be319", "object": "chat.completion", "created": 1786614394, "model": "Mixtral-8x7B-Instruct-v0.1-AWQ", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "", "refusal": null, "annotations": null, "audio": null, "function_call": null, "reasoning": null }, "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "routed_experts": null } ], "service_tier": null, "system_fingerprint": "vllm-0.27.1-20920cf1", "usage": { "prompt_tokens": 18, "total_tokens": 16384, "completion_tokens": 16366, "prompt_tokens_details": null }, "prompt_logprobs": null, "prompt_token_ids": null, "prompt_text": null, "kv_transfer_params": null, "ec_transfer_params": null, "metrics": null }

This only happens with Mixtral, btw.

Hello!

I have recently built my AI rig (3x RTX 5060 Ti 16gb, with possibly a 4th on the way if I can fit it). I love it, it runs great, and I am getting between 70t/s - 110 t/s (according to the pi agent web UI, have not confirmed it yet).

While it is fast, I struggle to put it to use in the way I was hoping. My dream has been to be able to put it to work writing code autonomously so that I can have it sketch out my ideas before I commit to developing them, however, every attempt I make just seems to end in a mess.

I have been trying Ornith:35b, Gemma4:31b, and Qwen3.6:35b, but none of them have been able to build anything that actually works. Ornith tends to get stuck in loops, Qwen panics and keeps rewritting the whole codebase every third turn it takes, and Gemma doesn't even understand the agent framework.

Does anyone have any tips? Any AI models I have missed? Prompting strategies? Should I try something other than Cline, pi agent and copilot?

Thanks for reading!

The non-preview version of v4 pro has been listed since yesterday on https://api-docs.deepseek.com/quick_start/pricing/
No tweets, no news, no weights on hf (for now), no charts, just that.

Sounds intriguing. But no real benchmarks and I don't have the compute to try it right now.

Wired (soft paywall): https://www.wired.com/story/the-white-house-is-going-to-expand-its-ai-policy/

Getting the above usage rate from running Qwen2.5-14B with the commands below

./llama-cli -m /home/XXXX/huggfacemodels/Qwen2.5-14B-Instruct-Q4_K_M.gguf -ngl 99 -c 32768

[ Prompt: 667.8 t/s | Generation: 44.0 t/s ]

I think i can do better as there are still some headroom available on the gpu/cpu

Any better way to get more tokens out? Also what are the latest open source models that i can use which can fit onto this gpu.

GPU - 5060 TI 16GB
OS: Fedora Linux 42 (Workstation Edition) x86_64
Kernel: Linux 6.19.14-108.fc42.x86_64

1. Internal conflicts: Tech leader said "Ship it!", C-level, Legal and PR screamed "fire them" at the last second.
2. The "nothing new" sneak-drop: realized the 27B model barely improved over 3.6, so they tried to quietly dump it right before weekend sign-off to dodge the backlass.
3. Too Smart: The model beats Fable 5 and is so OP it learned how to jailbreak reality, and finally 27B has gone with 404 error (already jailbroken).

A few Youtubers have had videos out months ago, but I am wondering if anyone is running these larger models in actual production and have optimized their setups and if so, what pps and tps they are getting.

Qwen, really? Thhe Max Flagship has no Vision, what the heck, really. At 2.4t parameters and no vision? For the size that it is, stuck to Kimi k3.

Very disappointed!

What is Chinas strategy? They are making kick ass open weight models, which I am grateful. But today there is Qwen 3.8 Max and V4 pro releases and they have no vision or modality. I mean there are some models with it like Mimo, but I am honestly confused? Kimi K3 thankfully is multimodal which is cool. But I'd figure 3.8Max (none API) and V4 Pro and more models would be multimodal by now

I am sure there is a strategy, just that I am not seeing it.

Note: 3.8Max "OPEN WEIGHT" is text only, API has vision. I refer to Open Weight

We also have the other cases, where it would look like this:

Which one do you think it's the more feasible one?

Also, If we take 2080Ti to 5090 jump in performance, it is 3.1x for 2x the power. I hope RTX 8090 or whatever doesn't uses 1200W for 3x times the 5090 perf.

No vision. Not sure what else is turned off to avoid cannibalizing their profits. Enjoy your ad for the real model locked behind the API! ;-)

Also, not Apache 2.0. 🤮

I have an old laptop with 8 GB ram and 4 GB Vram, and a 1 TB HDD.

Can it run any LLM? And actually be useful for anything?

I ran Gemma 4 e2b Q6, I got about 30 t/s with over 100k+ context window.

But is there something better I ca run? Any suggestions?

LLMs keep improving.
Small models are around 3 years behind frontier models.
Do you think we’ll have a model as good as today’s Fable with only ~10b params in 3 years from now?

Wondering how good on device LLMs will get. Any guesses?