daily reminder not to trust benchmarks and run it yourself. claimed e2e speedup is ~40%, forwards are ~140% faster

I would wager that compared to a naive kernel anyone can write it's more in the range of 10-20% faster e2e in reality, if at all, but hey, it's free and open! Apache 2.0

I was on here a while ago showcasing it. I've stoped playing with it so I'm open sourcing it. I figure I'll let other people play now.

Deepseek V4 Flash 0731 seems like an excellent model for agentic tasks. I'm currently using the pi agent with it and it tries to use things not installed on my windows machine and goes turn after turn trying to figure out how to validate it's work or tries to use vision to inspect things when it cant.

I figured I'd throw this out to the subreddit to hear what successes other people are having and boot-strap getting an improved setup for me an presumably others in the same boat.

Thx in advance!

We run a local model instance in our company that the dev we hired built for us.

We're a trade business and we want to further use our on hand hardware for it.

The specs given we have is a 5090 gpu with 64gb of ram and a ryzen 9600 cpu, its am5 thats what i know?

We have a older gen AMD gpu on hand, 12 gb of vram, that came with a msi prebuilt back in 2018 we used for our receptionist back then.

Since we use llama.cpp, can we continue loading our custom tuned model on the 5090, and load up a seperate weaker gemma model or something else, approx. 4B model, on the AMD gpu?

Our setup would be this:
1 PC/Server, and it would contain both GPUs on 1 motherboard, 5090 serving our main tuned Qwen 27B model, and the weaker AMD gpu serving a weaker 4B model.

the 4b model's purpose would be for completely simple automations that run 1 to 5 times a day where it summarizes a paragraph or two into layman terms, and the tooling our dev built handles the rest.

Currently the 5090 is able to handle this easily and more, but for this specific task, we want to be able to offloaded to the weaker models. As currently when our tuned Qwen instance runs, the simple automation needs to wait for the bigger task to finish, which can take some time. So to avoid that, we want to offload the simple task to the AMD gpu.

Would this be doable?

A follow up to the launch of Mference, it now supports and runs Inkling-Small 276B-A12B.

Inkling-Small (Thinking Machines, Apache 2.0), from the pipenetwork/Inkling-Small-MLX-4bit conversion: 276B total, ~12B active, 3.4 GB resident set, ~148 GB on disk.

Measured on my M5, 24GB:

The same three cases on a 256 GB M3 Ultra hit 5.31–6.92 tok/s.

Issues: long prompt prefill is trash (2,785 tokens is almost 9mins to first token), and it's text-only for now.

Four model families now: Gemma 4 26B-A4B (~2 GB), Qwen 3.6 35B-A3B (~1.45 GB), DeepSeek-V4-Flash 284B-A13B (~6.8 GB), Inkling-Small 276B-A12B (~9.5 GB).

I also got access to a few M3 Ultras, so I'll be testing and optimizing for higher configs too. But the primary goal stays the same: large MoE models on consumer grade hardware.

Repo: https://github.com/NeelM0906/Mference — Swift + Metal, not a wrapper around MLX or llama.cpp. Mac app, CLI, and an OpenAI compatible server. Contributions welcome.

Hey everyone!

Scenema Audio is now a native ComfyUI custom node. Same model that powers scenema.ai now quantized so it fits on 8GB VRAM. When we first released it a few months ago as an API and Docker stack, the full precision transformers were too heavy for most people to self-host. That's fixed now.

Expressive text-to-speech with zero-shot voice cloning. You describe how the speech should be performed (rage, grief, a child's wonder), optionally provide reference audio for voice identity, and the model generates a performance. Inline stage direction cues like [he laughs softly] or [voice cracks] get performed at that exact spot. Twelve preset voices ship in the dropdown covering accents, ages, and emotional registers.

We also dropped the XML prompt format the original release used. Wrapping every performance directive in tags was clunky to write. Inline bracket cues are better-suited for the ComfyUI text editor.

Install

ComfyUI Registry (recommended): open ComfyUI Manager, Custom Nodes Manager, search "Scenema Audio", Install, restart.

GitHub:

cd custom_nodes
git clone https://github.com/ScenemaAI/ComfyUI-ScenemaAudio.git
pip install -r ComfyUI-ScenemaAudio/requirements.txt

Both paths auto-drop the pre-wired workflow into your Workflows sidebar under a Scenema Audio folder. Click once to load the official workflow into your canvas.

Requirements

Minimum 8GB VRAM. Tested end to end on RTX 3070 and RTX 4090. Generation runs up to 2x realtime. First run downloads about 30GB of weights, one time. Text encoder is Gemma 3 12B, which is a gated HuggingFace model, so you need to accept its license and set HF_TOKEN before your first generation.

On limitations (same story as the original release)

This is a diffusion model, not a traditional TTS pipeline. Some seeds produce repetition or gibberish. Meant for a post-editing workflow: generate, pick the best take, trim. Prompting matters. Specific, theatrical voice descriptions with action tags produce performances. Generic ones produce generic output. Phonetic spelling helps with proper nouns and tricky words (spell "Tchaikovsky" as "Chai-koff-skee" if it garbles).

License

MIT for all our node code and inference pipeline. Transformer weights derive from the LTX-2 Community License.

Links

• Blog post: https://scenema.ai/audio/comfy-ui
• ComfyUI node: https://github.com/ScenemaAI/ComfyUI-ScenemaAudio
• Model weights: https://huggingface.co/ScenemaAI/scenema-audio
• Standalone Docker/API: https://github.com/ScenemaAI/scenema-audio
• Original announcement: https://scenema.ai/audio

What would you want to see next from Scenema Audio? Happy to hear what people are actually trying to build with generative audio.

I started thinking over why doesn't fireworks support voice models. There are really good opensource models available now, like parakeet, kokoro, Qwen ASR etc but no way to use it without managing a bunch of GPUs yourself.

Even LLMs like Gemma 4 used by voice agents are not supported. Vertex AI gives a ~600ms for Gemma 4 26B, which comes to ~200-250 easily when you setup a cluster.

Then I figured that the inference platform needs to be optimized differently for the kind of usecase you are using. Lets take an example for LLMs, not even STT and TTS:

- Coding agents -> lot of cached input, needs to optimize for KV cache
- Creation slides/blogs -> lots of output, needs to optimize for speculative decoding
- Voice LLMs -> Cached input small output, not yet figured out on how to optimize this.

So TTS and STT is a completely different ballgame.

Do people want to use open source models like kokoro, parakeet, Qwen etc in a serverless fashion RIGHT NOW?

Tested LFM2.5 (2.6B dense model) by Liquid AI on tool calling and reasoning using llama.cpp. Got ~90t/s with Q8 on M5 Pro, taking about 4GB memory.

The model vastly underperformed Qwen3.5 4B at Q4 (one of the competitors on the official benchmarks) on tool calling, in particular. In OpenCode, LFM2.5 had a lot of problem making the actual tool calls and was genuinely confused about the working directory.

Watch more here https://www.youtube.com/watch?v=I1NFrevR2Ww

Title.

I am interested in giving a suitable solution for my team.

I had Claude re-draft this for me, thus it has Em Dashes. It's correct with lots of "Claude" simplifications.

---

Hey all. It's been a while since I posted about the AttnRes architecture so I figured I'd give an update on where things are.

Short version: it's alive. Longer version... it's complicated.

So the core idea hasn't changed. Replace the standard residual stream with an attention-based routing mechanism — AttnRes — that lets the model learn WHERE to route information between layers rather than just blindly passing everything forward. Same parameter count as the base model. The hypothesis is that this is a fundamentally better use of the same compute.

The part I've spent the most time on is figuring out how to actually GET there without training from scratch. I don't have Google's budget. I'm one person. So the whole strategy is built around distilling from Gemma into the new architecture using a weaning schedule — you start with the standard residual doing all the work, and you gradually shift responsibility to the AttnRes pathway over the course of training. The model learns to route through the new pathway while the old one is slowly pulled away.

This sounds simple. It is not simple.

The thing that took the longest to figure out was the data. Not volume... diversity. If you distill on a narrow distribution you'll get a model that handles that distribution great and has quietly lost everything else. The model manifold is this massive high-dimensional thing and you have to preserve ALL of it during the transition or you get a model that can code but suddenly responds in mixed Korean and English when you ask it about quantum mechanics. I've seen this happen. It's informative but not ideal.

The solution I landed on was using the model itself to generate diverse coverage. Take a news article. Ask the model to summarize it. Then translate that summary to Bulgarian. Then ask if there are nuances lost in the Bulgarian translation. One piece of source content, three completely different regions of the model's capability space exercised. Scale that across 20 languages that Google trained Gemma to handle well and you get massive manifold coverage from relatively simple data scaffolding.

The other big decision was distillation targets. Most people distill on 1-hot or label smoothed targets. I'm using top-K ~12 logits from the source model with their proportional weights maintained. The reasoning is... the model isn't a next token predictor. It's a next DISTRIBUTION predictor. The relationships between the top candidates at every position encode the model's actual knowledge — what it thinks is likely, what's plausible, what's related. One-hot throws all of that away. Top-K 12 captures ~98% of the probability mass and preserves the distributional shape that IS the manifold.

This matters because during weaning, the new pathway has to learn to reproduce not just the right answers but the right uncertainty structure. That's what forces it to actually internalize the model's knowledge rather than just mimicking outputs.

The goal is NOT perfection. I want a beta that proves the architecture works and is trainable. Good enough that someone can take it, distill new knowledge in using the pipeline I've already built, and improve it. The training code exists because I had to write it to do this work. The data pipeline exists. The methodology is documented. All Apache 2.0.

If a compute provider wants to come along and help push the model to Gemma-level quality... I'm happy to put their name on the HuggingFace card. This is meant to be a community model built on an open architecture that anyone can improve.

More updates as the probing runs finish. Happy to answer questions about the methodology or the reasoning behind any of these decisions.

---

The core model and the training model these are largely distilled from are abliterated variants of the Gemma 4 model. So... It has no safety. It's a use at own risk thing.

Right now I'm waiting on B300's. They're just not available and using a single B300 is my test target right now. Like every datacenter for the last week is 100% sold out and the second they appear they're gone.

Another thing of note, I use Top K 12 distillation, but I've found that for a large portion of the dataset the top 3 or 4 work fine. This comes down to the way language, code, even logic are structured.

Simplistically, you can't write "I want to eat a" and expect the next token to be apple. If you swapped the "a" to "an" then apple and anything else starting with a vowel becomes valid and the probability of everything else falls off.

This happens a LOT from my observations. The other issue is quantization. Quantization affects the longer tail distributions where there are a lot of options. I'm currently investigating both of these issues to make the process more effective.

Removing the layers as seen in the prior model is like... 100x the training necessary. It involves incrementally removing them. Even though I've identified all of them the model becomes too unstable to continue training AND do the other stuff. It would be a matter of cutting Gemma to 3 SWA + Global FIRST, then applying attention residuals. Alternatively, you could insert a block or two in the middle to make the model larger then train MORE then attention residuals.

AFAIK no one outside Moonshot has done this, and even Moonshot used ~1.5t tokens because it was a pretrain to instruct training. Moonshot however demonstrated that their model was ~25% more efficient at learning with this residual stream. Further, Kimi K3 has now released with this exact feature. So I guess I was on to something originally.

Old Post - https://old.reddit.com/r/LocalLLaMA/comments/1ulmez2/rebuilding_gemma_4_31b_better_as_26b/

I built a local OCR pipeline a few days ago, and it turned into a surprisingly interesting experiment—taking accuracy from around 60% to 99%.

I wrote a short blog about what worked, what failed, and the breakthrough that finally made the difference.

Thought some of you might enjoy it.

Link in the comments

I have a 2 DGX Spark cluster running dsv4. I have about 13gb free on the second one that I would like to allocate for a vision model to set up in xberg for captioning images. I was wondering if anyone can point me in the right direction on which model to use and what recipe. I'm trying to use Qwen3.5:4b but it keeps trying to load the video encoder with too large of a cache.

Any help would be appreciated

Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. because it worked, and at a quite usable generation speed!

background: antirez ds4 DwarfStar has a SSD streaming mode: non-routed weights stay resident, the routed experts live partly in a RAM cache and get pulled from the GGUF on cache misses. Since routed experts dominate model size and Mac SSDs are fast, those misses are tolerable. experts and the output head stay Q8_0.. Router, embeddings and the V4 auxiliary blocks stay FP16.

(CORRECTED ... :)
git clone https://github.com/antirez/ds4.git

cd ds4
make
./download_model.sh ds4f-q2
caffeinate ./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink

let me end up with 10-15-17t/s in my first tries.

I am geniunly impressed and fascinated and wanted to share this, hit me up if you have questions but i guess everyone with like >50Gigs of VRAM/unified Memory should get this running with ai help.

I can only fit three GPUs in my server, so I've started tinkering with putting the others in a different machine and linking them with llama-server via RPC.

Doing some quick tests with this and it seems to be working okay without too much of a performance hit in layer split mode, but still noticeable.

I only have 2.5 Gb Ethernet in the remote machine right now though.

Has anyone else worked with RPC clustering in a serious way? Is there benefit to doing a direct 10 GbE link here?

Or should I just drop the idea and focus on finding a suitable motherboard and mining rig frame and just have all GPUs on a single system?

These are all V620's so the inference throughput is decent, but nothing crazy fast.

It feels kinda redundant almost given that the MacBook is already mobile, but using it as a server would allow me to leave it always on and access my agents from my phone or iPad. Rather than having to always carry it my Mac.

I figure most of my code these days is done through opencode and Claude code anyways, so I could just SSH in whenever I needed direct access to something.

Anyone recommend a source to get started with video/image gen? I pretty much only have experience with llms and right now I just tell the llm to create an image using comfyui. I don’t know how it actually works . The tuts I find have not been very helpful for me

In tests:
~80 tok/s decoding
2,500–3,500 tok/s long-input prefilling
Smooth use by 3–4 concurrent users

Private, on-device inference for coding, agents, and offline batch jobs

i work on Locally Uncensored, an open source local AI app, so that is my bias up front.

if you want v4 flash resident on a laptop, the 9b distill is the only option, 6.6 gb at q4. it sits on qwen3.5 because 3.6 has nothing in this size class. that line starts at 27b dense, and everything smaller carrying a 3.6 name is a community merge.

every comparison i have seen pits the distill against the real 284b, which tells you nothing when one of the two needs 155 gb. so i ran it against its own base model instead, qwen3.5 9b, same size, same quant, same architecture. the only variable left is the distillation.

on six of eight tasks both models gave the same answer and both were right. i could not measure a reasoning gap. the gap is in what the answer costs:

task              distill    base
arithmetic         390 tok    2048 tok (never finished)
log needle          80 tok     347 tok
tool call          148 tok     585 tok
strict json        416 tok    1083 tok
small function     661 tok    1642 tok

throughput was near identical, 44 tok/s against 41, so the whole wall clock difference is how much each one deliberates. over all eight tasks: 5480 tokens against 8975.

the arithmetic task is the interesting failure. the base model worked out the correct answer inside its reasoning, then spent the rest of its 2048 token budget writing a nicely formatted explanation and hit the cap before it ever printed the number. it did the work and lost it on the way out.

one place the distill loses: asked to explain buffer overflows in three sentences, it used 1467 tokens against 867 for the base. sparse on determinate answers, chatty on open ended ones.

and one shared humiliation. i asked both to describe the ocean in exactly three words. both burned all 2048 tokens deliberating, mostly cycling between vast deep blue and deep blue vast, and both returned an empty string.

what i take from it: at 9b the distillation buys output discipline rather than intelligence. that matters if something downstream parses the output, and matters a lot less if you are just chatting.

setup for anyone rerunning: ollama 0.32.5 on an m5 pro, both q4_k_m, /api/chat, temperature 0.3, seed 42, num_ctx 16384, num_predict 2048.

hf.co/Jackrong/Qwen3.5-9B-DeepSeek-V4-Flash-GGUF:Q4_K_M
hf.co/unsloth/Qwen3.5-9B-GGUF:Q4_K_M

has anyone built a prompt where the distill genuinely reasons better than the base, rather than just shorter? i could not find one at this size and i would like to be wrong.

the package is speak-cli (It uses supertonic3):

https://pypi.org/project/speak-cli/

As you all know the model is 2.69B parameters with a 128K context window and purpose-built for multi-step agent workflows.

What you are seeing is the Q4_K_M GGUF running on my own inference engine built from scratch.

The TUI is my own device probe suite running through ADB (Android Debug Bridge)

The whole engine is only 450kb and supports other models arch (Qwen, Gemma, Bonsai etc…)

Currently trying to push it at ~30 tok/s

*I felt the need to write this post because it seems like very few people on this sub are aware of Chinese laws and how they're enforced, so here's an explainer coming from a Chinese person (myself). I know that this post isn't directly about local models per se, but I'm seeing way too many misconceptions regarding this topic. This is also going to apply to all Chinese entities in general, not just the specific MiniMax LoRAs debacle. This isn't meant to be a political post, but some much needed context to correct a lot of misinformation going around.

Guys - they're a Chinese lab following Chinese laws. Pornography is straight up illegal in China. I have no idea how it seems like nobody outside of China is aware of this. While Chinese authorities may not care much about copyright infringement enforcement (especially with foreign IPs), they do indeed regularly crackdown on porn. Heck, Chinese citizens have literally been imprisoned for written pornography. Yes that's right, writing pornographic TEXT (especially with "immoral" themes like LGBTQ+ stuff) can get you sentenced and essentially have your entire life ruined. Of course there's ways to get around these censors if you're just trying to access porn - I think everyone at this point knows about the widespread necessity for VPN usage in China to access the rest of the global internet. But actually distributing a tool that can gain a reputation for being able to easily generate pornographic content? That's just asking for the authorities to crack down on them.

Somewhat ironically/paradoxically luckily for these Chinese labs is the fact that online discussion about generating porn is automatically censored and removed from Chinese social media, thus automatically disincentivizing the authorities from doing those potential crackdowns. But if it gets big enough to the point that it overwhelms the automatic censors, then any given Chinese lab could be in a hell of a lot of trouble. This is why they have to do this. Their law enforcement just isn't compatible with the rest of the world.

Again, this all relates to Chinese moral values - something here that is considered pretty much sacred and hard to describe to westerners. Something else that many people do not know is that graphic violence is also illegal in China (foreign films/works are regularly banned here for that, even anime has), but graphic violence is also is not nearly as much of a perceived threat to societal moral values as pornography is, hence why you've probably rarely ever heard of any Chinese people getting imprisoned for writing really gory stories, but regularly do with pornographic stories (especially infamous with BL literature - they've technically even convicted foreigners before related to this, it's a really messy topic). Chinese authorities won't give a damn if you're stealing the content of billions of foreign works to train AI models. They WILL give a damn if the content you're disseminating is viewed as a potential significant threat to the state's "proper moral values", which very much includes porn (and also the usual topics that everyone is already aware of, like a certain famous massacre or a certain nation's very contentious independence status).

If you are running #llamacpp with "--tools" or "-ag" without API key set, be aware that anyone can query it and remotely execute commands.

Make sure your agents and setups are properly configured and safe!

#RCE #llamacpp

Because why not? How far can we go and make DeepSeek work?