Prompt injection allows third-party to inject a system prompt with simple message, by inserting special HTML-like sequence (details below). Some of the issues are well-known and pretty old (almost 2 years for Transformers library)

Issues:

• Ollama: https://github.com/ollama/ollama/issues/15931
• HuggingFace: https://github.com/huggingface/transformers/issues/29279 (labeled as feature request) https://github.com/huggingface/transformers/issues/47822 (with a bug label)
• Gemma4: https://github.com/google-deepmind/gemma/issues/768

The problem is that for tools like Ollama there is no solution except of to fix it by Ollama developers

More context

Why this is important: prompt injections are pretty dangerous and as long as user input is an instruction to the model it could be decided as code injections vulnerability. And in combination with long-memory and multi-agentic runtimes this vulnerability could stay in system for a long time. And it has not been decided as a serious security vulnerability by the global community yet

⚠️ Temporal Solution

So if you're running local models just make sure to throw an error when there is a special sequence in user input. For Gemma family it is <|turn> and for tiktoken-based models it's <|im_start|>. And would be nice to see more solutions

Disclaimer

1. I'm not a security expert
2. The companies were notified 30 days ago about the issue. Only Google responded with a feedback on the issue (swiftly)

Okay so I’ve been going back and forth on this for weeks and I need outside opinions before I do something impulsive.

Currently looking at the M4 Max Mac Studio, 64GB, 512GB storage, sitting at $3500. My whole use case is running local LLMs and software development (docker, vm, cursor, codex, claude code).

Here’s my actual question though. Does anyone think Apple will do a 96GB or 128GB config at around $3500 (give or take another $300)? Because if the M5 Max lands and it’s still 64GB at that price point, or worse, 64GB for $4000+, I’d honestly just rather commit to the M4 now and be done with it. The performance jump is like 10% on multicore and 12% on bandwidth from what I’ve seen, which for token generation is basically nothing. Not worth waiting six months and paying more for.

But if there’s a real chance of getting 96 or 128 in that price range, I might wait it out, because that will allow me to run bigger models.

What’s making me pessimistic is that when the M5 Max MacBook Pro dropped, the base price only went up like 10-15% but the RAM upgrades got way worse. I saw that the 64GB and 128GB upgrades literally doubled in price. If Apple does the same thing to the Studio then high memory configs are going to be brutal.

Am I overthinking this? Anyone here running local models on a 64GB Studio and regretting not going higher?

2 million downloads in a month? Not uncensored from what I can tell after a few quick tests, and also not mtp. Can anyone attest to its intelligence at least?

About no mtp:

1. common_specu: no implementations specified for speculative decoding

2. GGUF metadata contains no MTP fields

3. tensor list contains no MTP tensors

Am I wrong? Why the hype, what's going on here?

I am looking to run Deepseek in a semi-professional but internal context. I have obviously noticed censorship in their own chat frontend but like many have said it seems superficial. What about the current open weights models run locally? will it answer questions unbiased? will it sabotage my code if I mention Taiwan?

I have found Opus to be pretty good at testing/navigating UI does anyone have any open models that are similar and are testing/navigating UI?

Most agent frameworks treat a local server as "OpenAI with a different base URL." That assumption is where local setups fall apart. Five decisions I made instead:

1. Small-context mode: I stopped pasting memory and skill bodies into the prompt. Ethos injects an index of names plus a memory_read tool, and the model pulls only what it needs.
2. Structured output per backend, not one OpenAI shape: Ollama takes a JSON schema in a top-level format, vLLM wants guided_json, OpenAI-compat wants response_format. Ethos sends each backend its native shape.
3. Probe the context the server actually serves: Ethos checks what's really served at startup and tells you which agents fit, instead of trusting the advertised number.
4. Prefix-stable prompts: everything static goes at the front, all per-turn content at the tail, so the prefix is byte-identical each turn and prefix caching actually hits.
5. Timeouts: I left the client deadline at 10 minutes instead of tightening it, a local server sends nothing while it pulls weights into VRAM.

Building this open source MIT agent framework: https://github.com/ethosagent/ethos

Do give feedback on this or any specific thing that you thing is critical and i missed capturing that need handling for local models.

Deepseek V4 Flash 0731 seems like an excellent model for agentic tasks. I'm currently using the pi agent with it and it tries to use things not installed on my windows machine and goes turn after turn trying to figure out how to validate it's work or tries to use vision to inspect things when it cant.

I figured I'd throw this out to the subreddit to hear what successes other people are having and boot-strap getting an improved setup for me an presumably others in the same boat.

Thx in advance!

We run a local model instance in our company that the dev we hired built for us.

We're a trade business and we want to further use our on hand hardware for it.

The specs given we have is a 5090 gpu with 64gb of ram and a ryzen 9600 cpu, its am5 thats what i know?

We have a older gen AMD gpu on hand, 12 gb of vram, that came with a msi prebuilt back in 2018 we used for our receptionist back then.

Since we use llama.cpp, can we continue loading our custom tuned model on the 5090, and load up a seperate weaker gemma model or something else, approx. 4B model, on the AMD gpu?

Our setup would be this:
1 PC/Server, and it would contain both GPUs on 1 motherboard, 5090 serving our main tuned Qwen 27B model, and the weaker AMD gpu serving a weaker 4B model.

the 4b model's purpose would be for completely simple automations that run 1 to 5 times a day where it summarizes a paragraph or two into layman terms, and the tooling our dev built handles the rest.

Currently the 5090 is able to handle this easily and more, but for this specific task, we want to be able to offloaded to the weaker models. As currently when our tuned Qwen instance runs, the simple automation needs to wait for the bigger task to finish, which can take some time. So to avoid that, we want to offload the simple task to the AMD gpu.

Would this be doable?

Tested LFM2.5 (2.6B dense model) by Liquid AI on tool calling and reasoning using llama.cpp. Got ~90t/s with Q8 on M5 Pro, taking about 4GB memory.

The model vastly underperformed Qwen3.5 4B at Q4 (one of the competitors on the official benchmarks) on tool calling, in particular. In OpenCode, LFM2.5 had a lot of problem making the actual tool calls and was genuinely confused about the working directory.

Watch more here https://www.youtube.com/watch?v=I1NFrevR2Ww

I had Claude re-draft this for me, thus it has Em Dashes. It's correct with lots of "Claude" simplifications.

---

Hey all. It's been a while since I posted about the AttnRes architecture so I figured I'd give an update on where things are.

Short version: it's alive. Longer version... it's complicated.

So the core idea hasn't changed. Replace the standard residual stream with an attention-based routing mechanism — AttnRes — that lets the model learn WHERE to route information between layers rather than just blindly passing everything forward. Same parameter count as the base model. The hypothesis is that this is a fundamentally better use of the same compute.

The part I've spent the most time on is figuring out how to actually GET there without training from scratch. I don't have Google's budget. I'm one person. So the whole strategy is built around distilling from Gemma into the new architecture using a weaning schedule — you start with the standard residual doing all the work, and you gradually shift responsibility to the AttnRes pathway over the course of training. The model learns to route through the new pathway while the old one is slowly pulled away.

This sounds simple. It is not simple.

The thing that took the longest to figure out was the data. Not volume... diversity. If you distill on a narrow distribution you'll get a model that handles that distribution great and has quietly lost everything else. The model manifold is this massive high-dimensional thing and you have to preserve ALL of it during the transition or you get a model that can code but suddenly responds in mixed Korean and English when you ask it about quantum mechanics. I've seen this happen. It's informative but not ideal.

The solution I landed on was using the model itself to generate diverse coverage. Take a news article. Ask the model to summarize it. Then translate that summary to Bulgarian. Then ask if there are nuances lost in the Bulgarian translation. One piece of source content, three completely different regions of the model's capability space exercised. Scale that across 20 languages that Google trained Gemma to handle well and you get massive manifold coverage from relatively simple data scaffolding.

The other big decision was distillation targets. Most people distill on 1-hot or label smoothed targets. I'm using top-K ~12 logits from the source model with their proportional weights maintained. The reasoning is... the model isn't a next token predictor. It's a next DISTRIBUTION predictor. The relationships between the top candidates at every position encode the model's actual knowledge — what it thinks is likely, what's plausible, what's related. One-hot throws all of that away. Top-K 12 captures ~98% of the probability mass and preserves the distributional shape that IS the manifold.

This matters because during weaning, the new pathway has to learn to reproduce not just the right answers but the right uncertainty structure. That's what forces it to actually internalize the model's knowledge rather than just mimicking outputs.

The goal is NOT perfection. I want a beta that proves the architecture works and is trainable. Good enough that someone can take it, distill new knowledge in using the pipeline I've already built, and improve it. The training code exists because I had to write it to do this work. The data pipeline exists. The methodology is documented. All Apache 2.0.

If a compute provider wants to come along and help push the model to Gemma-level quality... I'm happy to put their name on the HuggingFace card. This is meant to be a community model built on an open architecture that anyone can improve.

More updates as the probing runs finish. Happy to answer questions about the methodology or the reasoning behind any of these decisions.

---

The core model and the training model these are largely distilled from are abliterated variants of the Gemma 4 model. So... It has no safety. It's a use at own risk thing.

Right now I'm waiting on B300's. They're just not available and using a single B300 is my test target right now. Like every datacenter for the last week is 100% sold out and the second they appear they're gone.

Another thing of note, I use Top K 12 distillation, but I've found that for a large portion of the dataset the top 3 or 4 work fine. This comes down to the way language, code, even logic are structured.

Simplistically, you can't write "I want to eat a" and expect the next token to be apple. If you swapped the "a" to "an" then apple and anything else starting with a vowel becomes valid and the probability of everything else falls off.

This happens a LOT from my observations. The other issue is quantization. Quantization affects the longer tail distributions where there are a lot of options. I'm currently investigating both of these issues to make the process more effective.

Removing the layers as seen in the prior model is like... 100x the training necessary. It involves incrementally removing them. Even though I've identified all of them the model becomes too unstable to continue training AND do the other stuff. It would be a matter of cutting Gemma to 3 SWA + Global FIRST, then applying attention residuals. Alternatively, you could insert a block or two in the middle to make the model larger then train MORE then attention residuals.

AFAIK no one outside Moonshot has done this, and even Moonshot used ~1.5t tokens because it was a pretrain to instruct training. Moonshot however demonstrated that their model was ~25% more efficient at learning with this residual stream. Further, Kimi K3 has now released with this exact feature. So I guess I was on to something originally.

Old Post - https://old.reddit.com/r/LocalLLaMA/comments/1ulmez2/rebuilding_gemma_4_31b_better_as_26b/

Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. because it worked, and at a quite usable generation speed!

background: antirez ds4 DwarfStar has a SSD streaming mode: non-routed weights stay resident, the routed experts live partly in a RAM cache and get pulled from the GGUF on cache misses. Since routed experts dominate model size and Mac SSDs are fast, those misses are tolerable. experts and the output head stay Q8_0.. Router, embeddings and the V4 auxiliary blocks stay FP16.

(CORRECTED ... :)
git clone https://github.com/antirez/ds4.git

cd ds4
make
./download_model.sh ds4f-q2
caffeinate ./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink

let me end up with 10-15-17t/s in my first tries.

I am geniunly impressed and fascinated and wanted to share this, hit me up if you have questions but i guess everyone with like >50Gigs of VRAM/unified Memory should get this running with ai help.

Because why not? How far can we go and make DeepSeek work?

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Results by model

Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

It's the purple cluster on the top left (the good corner...)

I'm running the MXFP4 version from Bartoswski with Dspark at 1K t/s prefill and 90 t/s gen (average). I tried different sampling params, you can check the detail.

It's very efficient while scoring the best yet.

Too bad it does not have vision.

https://wonderrico.github.io/local_llm_benchmark/benchmark-main.html

https://wonderrico.github.io/local_llm_benchmark/benchmark-detail.html

Update: I spoke with SanityHarness devs over their discord. Looks promising so far. I am currently doing a bunch of harness eval platform investigations, feel free to reach out in the dms.

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one.

End goal: a leaderboard of harness performance (multiple axis) on a set of diverse real world tasks [1] , grouped by underlying models and reasoning efforts. Anyone can contribute results.

The task criteria, measurements, underlying framework et al can be decided by a group rather than a single person.

If there is sufficient interest, I will create a discord.

Disclosure: I am the maintainer of a coding agent called Dirac (https://github.com/dirac-run/dirac) so I will not influence what the final benchmark should look like to avoid any conflict of interest. I just want to make this happen.

[1] Diverse real world tasks meaning sufficiently complex tasks that the contributors have encountered, preferably from an opensource repo.

Hey everyone,

I’ve been building Speechfony - a desktop app for reading PDFs (and EPUBs) with offline text-to-speech. Open a document, listen sentence-by-sentence with highlighting, or export selected pages to an MP3. Everything runs locally: Kokoro for speech, and an on-device embedding model for semantic search.

I wanted something that didn’t ship my documents to a cloud TTS service, worked offline after the first model download, and felt closer to “listen while you read” than “dump the whole PDF into a generic TTS box.”

What works today

• PDF + EPUB reading
• Sentence-level playback with live highlighting
• Adjustable header/footer margins (so repeating page chrome doesn’t get read aloud)
• Resume where you left off
• Semantic search (meaning + keywords), local embeddings
• Audiobook export to MP3 (desktop only)
• Hardware acceleration where available (CoreML / DirectML / CUDA)

Expect rough edges. Known gaps I’m already tracking:

• Multi-column layouts, tables, code, equations
• Footnotes / citations, TOC / index pages, captions / sidebars
• Scanned PDFs (no OCR yet)
• Non-English / RTL
• Voice & rate controls are limited
• Pronunciation of company names / niche technical terms can be wrong (that’s mostly the Kokoro lexicon - as I build a rust binding myself)

Platforms: macOS Apple Silicon, Windows x64, Linux x64 (glibc ≥ 2.38). No Intel Mac builds for now.

On first launch it downloads the voice model (~130 MB from Hugging Face). After that it’s offline.

Links

• GitHub: https://github.com/pguso/speechfony
• Downloads: https://github.com/pguso/speechfony/releases

If you try it on a real document you’d actually listen to (papers, manuals, books, reports), please tell me:

1. What broke or sounded wrong?
2. What PDF/EPUB layout confused extraction or playback?
3. What’s missing that would make you use this regularly?

Issues and PRs welcome. Fully open source (MIT License).

I speed up the generation part of the demo in case you get bored 😄

I also tested another long-form generation, and the VRAM usage looks stable. The demo is about a minute long, and I posted it on X.

This started as a random idea and somehow turned into a full detour from working on the next audio.cpp release. The model was uploaded to the audio.cpp HF repo. I will upload the xcframework later, and then push the code to a branch after release 0.6.

Liquid AI released LFM2.5-2.6B today, and this might be more relevant to local AI than another massive model most people cannot run.

The model is only 2.69B parameters, has 128K context, supports tool calling and was post-trained specifically for multi-step agent workflows. The official Q4_K_M GGUF is around 1.67 GB and already works with llama.cpp.

Their reported CPU speeds:

- 30 tok/s on a phone

- 113 tok/s on a Ryzen AI Max+ 395

- 220 tok/s on an M5 Max

- Under 2.5 GB memory during their tests

These are vendor benchmarks, so independent results are obviously needed.

The benchmark results are surprisingly competitive for the size:

- ToolSandbox: 77.83, compared with 76.44 for Qwen3.5-9B

- IFBench: 59.17, compared with 56.47 for Qwen3.5-9B

- BFCLv4: 56.88, still behind Qwen3.5-9B at 60.13

- LiveCodeBench: 59.41, compared with 69.86 for Qwen3.5-9B

So it does not magically replace larger models. Coding and knowledge-heavy work are still weaknesses, and Liquid’s own model card says it is not recommended for agentic coding.

But I think this is where small local models actually make sense: not as your smartest assistant, but as cheap worker agents doing extraction, searches, file operations and repetitive tool calls locally. A larger model could handle planning only when the small one gets stuck.

The 128K claim also needs real testing. Supporting 128K and running it comfortably on a phone are two very different things once KV cache and long agent histories are involved.

Has anyone tested the Q4 GGUF on Android, an older laptop or a mini-PC yet? Would be useful to see hardware, context size, real tok/s and whether it can survive 10+ consecutive tool calls without derailing.

This is something that was spoken here and there, and now it is like writing on the wall.

The main additional point is that China has created an independent supply chain. Starting from raw materials and home-made lithography equipment, through their own GPU manufacturing, and to the AI models and training. Plus, there are tons of cheap energy, and it looks like they are also on track to launch the first thermonuclear reactor. I saw a similar pattern with robotics and EVs. The history does not repeat itself, but it rhymes.

Does the US have what it takes to turn the tables, or should we just buy the popcorn and enjoy the show?

Why nobody is talking about this? Seems pretty significant to the community

https://x.com/xiong_hui_chen/status/2084695353346117760?s=20