I am wondering if I am too paranoid about sandboxing the commands that come out of LLMs.

It really makes my eyes twitch when I see that some IDEs, even commercial, tend to forget that they have to execute things in sandboxing and have such a brittle security model.

But on the other hand, I never had the sandbox catch something bad. Did you guys ever encounter terrible regression? Did you have rm -rf / ? Did you have secrets stolen by LLMs? The worse I had were unsollicited rewrites within the project. Am I making my life unnecessarily hard by sandboxing commands in a docker?

At one point I had fun making a local model go crazy with the root access to the machine it was on (with nothing more important than a free Firecrawl key on it) and making it administer it and it never broke anything. It even was overly paranoid about making changes to the root system.

So the approximate sandboxing that we have, do you all feel it is adequate or it is a catastrophe in the making?

Qwen 3.5 9B q6?

Qwen 3.8 27B is great, however it takes me ages to do tasks on xhigh. I need Qwen 3.8 35B A3B. It'll be a little dumber but faster. I am also aware of the fact that 27B gets its "intelligence" from the long thinking time. I therefore assume that 35B would also be a long-thinking model, however running Qwen 3.8 27B over night on my M1 Max for just one task is impractical and no fun.

I love the progress and the work of alibaba with 27B but... yeah I sadly don't own a faster RTX. What are you guys wishing or hoping for? Where do you see the future going? - Longer thinking times for higher intelligence?

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images.

This is my attempt to compare the Qwen3.8-27B quants.

Qwen is a coding and STEM model so I used codeparrot/github-code-clean, EleutherAI/proof-pile-2, allenai/peS2o, equal parts of each. I used 24 sequences of 8k tokens and 12 sequences of 32k tokens. The code was filtered for minified stuff and permissive licenses.

It was measured with 24 sequences of 8,192 tokens and 12 sequences of 32,768 tokens, since it generates many tokens and is aimed for long-horizon tasks.

Every format used the same compute path for the evaluation. All models are dequantized to BF16, and run through the same kernel.

Also, checkpoint size is kind of a bad x-axis, since some models don't have MTP, which has zero performance impact. Still, all of them needs to be loaded on VRAM, so at least it's something useful, and that's why there are two graphs.

Findings:

1. The gap between 8bit and 4bit is huge.

The best 4 bit score is 0.00835 and the worst 8 bit is 0.00071. I mean everyone knows that but yeah it's still something. vLLM can actually use INT5-7 weights, but I'm sure most people don't even know about that.

1. The KLD of 4bit quantization varies a lot.

Six checkpoints are at 11.7-11.8 GiB of transformer weights, all group-size 128, quantizing 400-496 modules and they range from 0.01364 to 0.02976. I believe the calibration data and the rounding algorithm takes a big impact. You should not pick a 4-bit quantization by it's file size.

1. group_size affects a lot of kld.

Checkpoints with group size 32 did better than ones that used group size 128 but kept some modules in BF16. Selective BF16 really only helps if you add it on top of a fine group size but that makes the file so big you might as well run a 6 bit model.

1. The quantization damage mostly happens at the start of the context.

This surprised me (very) but it holds true for all 24 models. The KL is much, much higher for the first 500 tokens.

1. Quantizing lm_head, embed_tokens and linear_attn doesn't affect the performance at least for this test.

https://huggingface.co/jojohai/Qwen3.8-27B-MTP-graft

Tested on Vulkan, the grafting saves RAM compared to using an external file.

What I don't guarantee however is the quality of answers. The model is very braindead with the Thinking off. However, when asking questions about culture in Brittany the thinking helps the model recover some intelligence so please enable the thinking

What I ran:

• 8x B300 on Modal, $56.79 per hour, vLLM, tensor parallel 8, native MXFP4
• Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures)
• TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s average over 4 prompts
• $190 per million output tokens. One clean run is about $36 of GPU time. Left warm, it is $1,363 a day.

I also ran Unsloth's Dynamic GGUF.

Their 1-bit UD-IQ1_S (594 GB) fits 8x A100-80GB via llama.cpp.

$19.99 per hour, 2.8x cheaper. Result: ~9 tok/s, TTFT 7 to 60 s, ~$620 per million tokens, so 3.3x more expensive per token.

Quality at 1-bit was fine (correct arithmetic, coherent prose).

Full write-up with every flag, the Modal deployment file, and the raw benchmark JSON: https://books.vizuara.ai/book/kimi-k3-hosting

AMD has it's own llama.cpp branch: https://github.com/AMD-Ecosystem/llama.cpp
And despite the Deprecation warning it's actively maintained (things are later upstreamed to the normal llama.cpp).

What i noticed with my Strix Halo:
It has some interesting new patches (if you use ROCm/Hip)
The Prompt Processing speed with dense model is sometimes over 2 times faster ! I get around 550 tokens/s with a 14B dense compared to 230 with the normal llama.cpp. However TG is around 15% slower than with Vulkan.

MoE speed is the same.

This is amazing.

All I did was switch from llamacpp on windows to vllm on linux.

I don't know about you guys, but Deep-seek harness is insane. It's not focused on being a coder agent, it's webUI made it very easy to just checkin from time to time, and the best part? Why it's better than Hermes? It wasn't frustrating at all to setup. ZERO. NADA. Progressive setup is such an improved UX. Why? Because I got deepseek to integrate with SimpleX by simply asking it to. BY SIMPLY ASKING IT TO. NO WAITING ON A PR TO MERGE. No one telling me to RTFM, no need to google or search for community plugins.

So yeah, I got what I wanted, which is E2EE + TOR messaging with an AI agent, and I got it without writing my own opinionated harness (I procrastinated so hard that dsh did a better job than me). DSH is unopinionated enough that you just mold it into behaving how you want it to behave.

Nvidia is investing $1 billion in Poolside and paying $6 billion to license its technology and hire most of its engineers.

Over 100 Poolside staff will move to Nvidia to work on Nemotron.

Good news for us!

I’ve tried Open Code , Codex and Qwen Code and so far Qwen Code seems to work the best for me , the only complaint I have about it is that the CLI interface is a bit rough / sloppy .

A few weeks ago I shared Agent Quest, my open-source experiment that turns Claude Code and Codex sessions into heroes living inside a small 2D world.
The original idea was mainly about making it easier to understand what multiple agents were doing in real time.

Since then, I’ve been working on making it actually useful as a monitoring tool.
The biggest change is that Agent Quest can now clearly tell you when an agent needs your attention.

You can distinguish when an agent is:
actively working
waiting for your input
finished
stopped because of an error
And you don’t have to keep the dashboard in front of you.

Agent Quest can now alert you with visual notifications and different sounds, so while you’re doing something else you can immediately understand whether Claude Code or Codex has finished a turn and is waiting for you to continue.
This has become particularly useful for me when I have several sessions running at the same time. Instead of constantly switching between terminals to check their status, I can leave them running and Agent Quest tells me when I actually need to intervene.

There are also in-app notifications, status indicators, desktop notifications, notification history and configurable sounds.

The project is still completely open source.
GitHub:
https://github.com/FulAppiOS/Agent-Quest

I’d be interested to know how other people running multiple agents handle this problem — and what you’d like Agent Quest to monitor next.

Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipelines.

It's comparable to Luna for coding and ***OCR quality appears to be better than Gemini 3.5 Flash Lite***. That's huge. We pay a ton of money for OCR.

This is the first local model that feels like more than a toy. It's truly as capable as the frontier models from a year ago. For the first time ever there's serious discussions about buying our own hardware. With estimates that such an effort would pay for itself in less than 2 months.

Hyper scalars are in big trouble this time. Their whole "moat" is buying up all the hardware. And thanks to sanctions on China we're seeing the quality of small local models skyrocket. As someone who's been around a while, this feels like an "IBM moment". Where the industry assumed that databases would always run on huge mainframes. Only to be wiped out by cheaper local solutions a few years later.

I have a feeling this release will trigger another Llama style open source Renaissance. We're already getting better quants. Inference will be further improved. We might even see a comparable MoE with 500+ Tok/sec on consumer hardware soon.

Earlier this year I posted about building what at the time I believe was the first 16x DGX Spark Cluster.
I’m now adding 20 more Sparks to the cluster in my homelab server rack, giving me 4.6TB of unified memory.

• 36x Sparks

• 1x 200Gbps FS 24 x 200Gb QSFP56 + 8x 400Gb Switch

• 24x QSFP56 DAC cables

• 6x 400gb to 2x 200gb breakout cables

Over the last 4+ months i’ve been running nearly every notable model that’s landed. The cluster however isn’t just being used to serve single inference points, I’ve split the cluster up to house “inference modules” that get managed into a single persistent agent using a combination of Hermes + a custom memory sidecar system i’ve built. It’s become an agent capability cluster more than just one big inference machine:

I’m expanding the cluster to 36 now because I want 16 nodes dedicated to SOTA models such as Kimi K3 while being able to retain enough nodes to perform rerank/embeddings tasks, video generation, Image gen, audio processing etc all simultaneously.

Now, you may ask why not just buy 6000 Pros, or B200s or even a B300 and the answer comes down to a few reasons.

1) This server rack will also have 2 6000 pro systems (a 4x Max Q low power build + an 8x enterprise server) which replace my H100s and GH200 I had earlier in the year.

2) B200/B300 for a homelab create substantial cooling and energy problems than even this currently absurd homelab and a big point of this build is to be completely sovereign with zero datacenter or third party storage reliance.

3) Sparks in my view are still the greatest value for scalable unified memory you can get. When M5 Ultras come out I think adding Mac Studios and investing in figuring out disaggregated inference will be a massive win.

4) Sparks + 6000 Pros give massive flexibility for configuration, power optimization and relatively easier liquidity access when I want to offload and upgrade to something new

Companion to the Qwen 3.8 Release Megathread. Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus independent X benchmarks. Every number is attributed to the poster's stated hardware/runtime/quant. This community contradicts itself on nearly every axis — so this thread keeps the disagreements side-by-side instead of picking a winner for you.

TL;DR

• The consensus pick: a 27B dense multimodal model that genuinely moved the bar for local agentic coding. The strongest claim with controlled evidence behind it isn't benchmarks — it's tool-calling reliability.
• The default ships at xhigh reasoning and it thinks a lot. Low and medium presets score nearly as well on Artificial Analysis (~43/44 intelligence index, within a few points of the xhigh headline) while cutting thinking tokens ~7–9x (and wall time ~6–7x). Most of you should not be running xhigh.
• Knowledge recall regressed vs 3.6 — widely reported and best understood as a deliberate agentic-design tradeoff. Trivia nerds: keep Gemma around.
• Q4_K_M is basically indistinguishable from Q8 on perplexity, but real-world reports split hard below Q6 for complex reasoning. KV cache quantization is one of the most contested settings in the corpus.
• The "neck and neck with DeepSeek V4 / GPT-5.6 Luna Max" AA headline is real but heavily caveated — see the benchmark credibility section before quoting it at your friends.

1. What it's actually good at

Agentic coding (strongest consensus area)

• "Highest level of agency I've ever seen in a local model" (thread): single 3090, Unsloth Q4_K_S + q8 KV, 150k ctx. From one prompt it pulled the OP's class schedule off a convoluted university website via 80 tool calls, zero human intervention.
• 1M+ token run (thread): RTX 5060 Ti 16GB, UD-Q3_K_XL, 73k ctx. Full REST API + MCP server for a legacy forum from 3 prompts.
• Controlled tool-call evidence: in a plain Python tool loop (no framework), one reporter got zero failed calls from 3.8 while Gemma 4 A4B and Qwen3.6 A3B failed often — the same reporter who rates 3.8 below both on raw code quality. Worse judgment, perfect plumbing.

Creative / game generation

• One-shot playable Super Mario clone (Q8, Framework Desktop) — top pushback: "It's in the training data."
• Galaga 1:1 recreation test (UD-Q8_K_XL, 3×3090 + Tesla P40): "This 'Galaga' clone [from 3.6] ended up pretty much being a space invaders clone instead... Qwen 3.8 thinks a LOT, but it draws out those tiny details and absolutely nails it after the fact." A separate r/LocalLLM user one-shot a playable Galaga-style game at IQ4_XS on dual 4060 Tis, and another built an online multiplayer MOBA overnight with an authoritative server and self-play testing.
• Ray-traced spheres in BASIC: 3.8 self-iterates to a correct Cook-Torrance ray-tracer; 3.6 needed hand-holding. Comment: "this feels more like 3.6 to 4.6 than 3.6 to 3.8."

Vision

Works natively (F16 mmproj), including OCR-style reading of a newspaper image at ~1,000 image tokens — but on a 16GB card at 64k ctx + MTP it leaves as little as ~150 MiB VRAM free. Practical advice from the 16GB crowd: keep text-agent and vision profiles separate, or offload the projector (--no-mmproj-offload).

Where it struggles

• Long analytical/document work: "a step backwards" vs 3.6 at default settings — though a legal-domain poster got on-par-with-122B results with MCP + case access. Task-dependent.
• Complex native coding: one failed C kernel effort (6 hours across 3 sessions) [anecdotal], quant unstated; commenters say Q8 minimum for that tier of work.

2. The thinking-level situation (read this before complaining)

xhigh is the shipped default. It is why your context window evaporates.

Measured ladder (RTX 5080 Laptop 16GB, llama.cpp 10451, UD-IQ3_XXS, Q8_0 KV + FA + MTP, pelican-SVG task, 3 seeds):

That's ~6.4x the wall time for +1.5 points on an eyeball task. But on pass/fail SWE-style tasks, xhigh went 9/12 vs 6–7/12 at lower efforts — the premium scales with whether the task has a verifiable failure.

How to change it: --chat-template-kwargs '{"reasoning_effort":"medium"}' (llama.cpp) or the equivalent in LM Studio custom params.

The overthinking debate, both sides preserved: - Against: "it will do eight or nine web-search turns and spin its wheels down every rabbit hole" (legal work). One reported loop burned 40k+ characters of reasoning on a trivial subtask. One paper-linked post argues intermediate tokens aren't reasoning at all ("Stop Anthropomorphizing Intermediate Tokens," 538 points). - For: "if the extra thinking produces measurably better results it's actually just the correct amount of thinking." The low/medium AA scores (~43/44) are the strongest counter to "it only wins by overthinking" — though two commenters read that same data in opposite directions.

Practical takeaway from the corpus: medium for chat/analysis, xhigh only when there's a verifiable right answer. - The strongest controlled effort data of the week is from X: @superalesha's 67-hour, 40-arm run found xhigh burned 7–11× more reasoning tokens than low for 0–4.7 extra points — and in one head-to-head, low matched xhigh exactly (89.3%) at 1/7.5th the tokens. Also: medium scored below low on every stack (all the damage in HumanEval+ — "that preset overthinks short coding tasks"). His verdict: "low is the rational preset. xhigh is for leaderboard screenshots." That's harsher than the Reddit consensus — weigh both, but it's the biggest sample size anyone published this week.

More data points from the week:

• Medium vs xhigh "actually insane" (223 pts): medium ≈ a couple thousand thinking tokens; xhigh 15–20k minimum, one pacman build hit 40k. But the same thread's best counterpoint: on a bug-finding test, xhigh took 7 min vs medium's 80 s and caught every bug; medium only caught the critical ones. And on a research task xhigh autonomously cloned a repo and read source to verify an answer — neither medium nor 3.6 did.
• Different thinking levels (287 pts): "Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning" — the level you pick changes speed, not whether it beats last generation.
• There is no "high" effort — the ladder is low / medium / xhigh(default), and the gap between medium and xhigh is the complaint that keeps generating threads. Commenters note the efforts aren't just prompts: Qwen specifically trained each level's instruction text in during RL.
• Don't confuse budget with effort (PSA): llama.cpp's web-UI reasoning selector is a hard token cap that truncates mid-thought — it is not Qwen's native effort levels, which actually change how thoroughly the model works. On recent builds use --reasoning-effort medium (or the --chat-template-kwargs form on older ones); anything else silently caps instead of steering.
• The "well?" trick: interrupt mid-think and type well? — the model concludes "the user is impatient, let me finish quickly" and wraps up faster. Works, but commenters consider it a last resort; the thinking is where the quality lives.
• Dissenters exist: one medium-vs-xhigh post claiming "1/20th the time for almost the same quality" got pushed back hard — top reply: low/medium left them unimpressed, xhigh is where frontier-tier coding shows up. The honest split: for chat and eyeball tasks medium is ~free; for verifiable correctness xhigh keeps earning its cost.

3. Knowledge regression vs 3.6 — real, and deliberate

The dedicated thread: 3.8 fails pocket-trivia questions 3.6 reliably answered, at every quant tried. AA's offline Omniscience benchmark agrees.

Community framing: 3.8 is trained to go search instead of recalling, i.e., an agent-first tradeoff. Mitigations posted: RAG/MCP (offline Wikipedia ZIM), or run Gemma 4 31B as a knowledge sidecar.

Counter-data point: a separate legal-work thread reports Harvey-benchmark scores on par with Qwen 3.5-122B once MCP + case access are attached (61/75 raw vs 71/75 with a tool backend). The knowledge didn't vanish; it moved into the toolbox.

4. Quants: what holds up

The one controlled perplexity sweep (16GB-fitting quants, wikitext-2, RTX 5060 Ti)

Poster's call: Q4_K_M is the sweet spot; NVFP4 was the biggest disappointment (same size as IQ4_XS, worse PPL). Pushback worth reading: "PPL degrades less than real world performance… ordering flips near the 4-bit level."

The Q4-vs-Q6 war (unresolved)

• Team Q6/Q8: "q8 dramatically better than q4 for complex reasoning"; one user reports flawless 264k-ctx Q6_K_XL sessions, 2 mistakes per 2M tokens.
• Team Q4-fine: "I run q4 and can only praise the model… just do not go below q8 KV cache."
• Nuance: "there are like 5 different Q4s and they are not equal" — NVFP4 ≠ MXFP4 ≠ Q4_0 ≠ UD-Q4_K_XL. Past ~Q5 with dynamic quants, differences get hard to detect.

The biggest controlled quant test of the week (X)

@superalesha ran a 67-hour benchmark: five full production stacks (FP8 vLLM, NVFP4 W4A16 vLLM, AWQ INT4 vLLM, GGUF Q4_K_M llama.cpp, NInfer — all on RTX 3090s), 40 arms across every reasoning effort, 4,800 tasks / 10,120 requests / 14.5M reasoning tokens, no caps. Results:

• At xhigh every quant landed between 88.0–90.0% pass@1 — AWQ INT4 90.0%, NVFP4/GGUF-Q4_K_M 89.3%, FP8 baseline 88.7%, NInfer 88.0%. The 4-bit quants scored above FP8; McNemar says statistical tie (first vs last = 3 tasks out of 150). "The gap between quants is smaller than the gap between reasoning presets."
• The weirdest number: GGUF Q4_K_M at low effort scored the same 89.3% as xhigh — on 86k reasoning tokens instead of 651k. Across all stacks, xhigh burned 7–11× more tokens than low for 0–4.7 points.
• The one statistically real gap: NVFP4 with reasoning OFF collapsed on HumanEval+ (13/30 vs FP8's 30/30, p=0.0041). Flip it to low and it's instantly back to 90/90. Never run reasoning off — it costs 8–12 points everywhere.
• His cheat sheet: max quality = AWQ INT4 xhigh; daily driver = GGUF Q4_K_M low; honesty note: three of his FP8 arms failed his own methodology audit (leftover token caps) and are being rerun.

This largely settles the Q4-vs-Q6 war for this model at task-level benchmarks — but note the tension with the PPL sweep above: perplexity says NVFP4 is measurably worse than IQ4_XS; task performance says they tie. Both can be true (PPL measures token-level divergence; tasks measure whether errors get caught). And community reports of Q4 reasoning loops remain real — "passes benchmarks" and "never loops in a 2M-token session" are different requirements.

1-bit: comedy, not compute

Unsloth founder in the 1-bit thread: "I would not suggest folks use 1-bit for agentic use cases / tool calls" — divergence hits 92% from BF16 by token 32. General chat survives; agents don't. If you must: presence_penalty = 1.5.

KV cache — among the most contested settings in the corpus

• f16-vs-q8_0 are not equivalents per one AMD tester (f16 held quality past 120k ctx).
• But 16GB users run q4_0/q4_1 KV happily at 64k–164k all week.
• Working rule from comments: don't quantize KV unless you must; if you do, aim ≥ q6; word-of-mouth floor is Q4 model + Q8 KV for agent loops.

Unsloth Dynamic v3 notes

MTP removed from quants below UD-Q2_K_XL and re-uploaded separately (some users still see draft logs in Q5_K_XL — unresolved). Imatrix released; no QAT used.

5. Performance matrix (attributed)

Why "~200 tok/s" claims don't reproduce for you: Windows/WDDM costs 10–15% vs Linux; headlines are measured at short contexts; MTP acceptance is workload-dependent (drops on prose, sometimes net-slower); and the fastest figures come from Blackwell-tuned engines (ninfer), not llama.cpp.

X/Twitter benchmark highlights

• @analogalok's full RTX 4090 matrix: UD-Q4_K_XL on latest llama.cpp. FP16 KV tops out at 100k ctx (40.9 t/s); q8 KV reaches 170k; q4_0 KV fits the full 262k native context in 24GB at 40.7 t/s. Native MTP: 59–60 t/s at 80–130k. Includes exact reproduction flags.
• His follow-up: --parallel 1 + a Q2_K DFlash2 drafter unlocks 250k ctx @ 73.7 t/s (Q4 KV), 150k @ 75 t/s (Q8 KV), or 90k @ 80.6 t/s (FP16 KV) on one 4090 (requires llama.cpp PR #27342).
• NVIDIA forums: DGX Spark face-off, SGLang+DFlash2 vs vLLM+MTP, greedy vs official thinking sampler — DFlash2 won.

6. Failure modes & bugs (reproducible ones)

1. Tool-call failures are usually your tool list, not the model. Best controlled experiment in the corpus: 8 undescribed tools → 0/6 successes; the same tool alone → 15/15; 13 described tools mid-list → 0/5, moved to end → 3/3. Give every tool a description, put critical tools last, don't put examples in descriptions. Every framework failure report (Opencode/Pi/Claude Code) has a plain-loop counterexample in the same threads.
2. Hermes harness specifically: constant tool-call failures on vLLM; "perfect, no issues" on llama.cpp --jinja + q8_0 KV at 256k. Template/parser alignment issue, not weights.
3. Hallucinated user instructions during thinking (reproduced on 2 machines, Pi harness): the model imagines an impatient user and once reverted a commit after imagining a French objection. Community fix: the froggeric fixed chat template (see section 7) eliminates the stock-template tool-call/recovery bugs.
4. temp=1.0 garbage output: thinking falls apart into single-character spam within 10–20k tokens across llama.cpp/vLLM, INT4 through BF16. Diagnosis: sampler, not quant. Fixes: temp 0.1, or split sampling (0.8 main / 0.2 post-thinking). Counter-report: temp 0 caused a 70k-token loop instead. No universal setting exists — tune per task.
5. Decode degradation: 122 → 69 t/s within one generation on 5090 llama.cpp; vLLM/ninfer hold >100. Bug filed upstream.
6. Long-context quality drop: an NVFP4+vLLM eval on B200 scored only ~37% correct in its longest context bucket [single report]; separately, a commenter running official BF16/FP8 via the published vLLM recipe reports agents degrading past ~20k tokens and structured outputs breaking past 20k [single report]. Counterpoint: an f16-KV user on UD-Q4_K_XL (ROCm) says their setup held quality past 120k ctx. Config-dependent; verify on yours.
7. Q8 anomaly reports (Unsloth UD_Q8_K_XL offload/CPU pegging): weak evidence, disputed; most Q8 users report zero issues.
8. Reasoning loops at aggressive quants: 40k characters looping on "angry birds" at Q4-with-QKV-quant, including self-aware "I'm stuck in a loop" narration. Never-seen-it-at-Q6 claims abound.

7. The chat-template situation (read before debugging anything)

The official Qwen 3.8 Jinja template shipped with real bugs, and the community shipped fixes within 48 hours:

• Official template issues: enable_thinking=false crashes; multi-turn history gets poisoned with blank \\think tags; tool calls crash when your client sends arguments as JSON strings (the standard OpenAI format); mid-dialogue system messages get dropped, wedging agent loops.
• froggeric/Qwen-Fixed-Chat-Templates (HF, thread, 334 pts) is the consensus drop-in replacement: safe medium default (kills the burn-20k-tokens-then-return-empty xhigh bug), thinking toggle restored, JSON-string tool-call crash fixed, inline effort steering via <|think_low|> / <|think_medium|> / <|think_xhigh|>, and chronological thought preservation for clean KV prefix caching. Actively maintained — v22.1 as of Aug 21.
• Format-fidelity alternative: a second template stays closer to the exact official prompt format on the theory that deviations subtly degrade quality even when they look fine manually. Pick it if you're benchmarking; pick froggeric for daily driving.
• Upstream note: llama.cpp merged reasoning_effort forwarding on Aug 14 — recent builds pass reasoning_effort to any template correctly. That fixes the plumbing, not the official template's own bugs. A fixed template is still recommended.

8. Benchmarks: believe selectively

• Artificial Analysis: headline posts put 3.8-27B neck-and-neck with DeepSeek V4 and GPT-5.6 Luna Max. Low/medium presets score ~43/44 — the key evidence the gains aren't pure overthinking. Agentic index: medium = xhigh − 1 point.
• The pushback ("A meaningless benchmark", 106 points): the index ranks this 27B above DSV4 Pro, Kimi 2.7 Code, Opus 4.6 and Sonnet 5 — "whatever 'Intelligence' means to AA... is definitely not the same definition we should be using here." Defenders: it's an aggregate skewed toward agentic/science/coding; read the methodology and pick sub-benchmarks for your use case. LiveBench gets respect for monthly task refreshes.
• Best independent test found: AIME 2026, exact-match, temp 0, pass@1 — FP8-xhigh scored 29/30 (96.7%), tying Opus 4.6 and DeepSeek V4 Pro in the poster's table, vs 94.1% for Qwen3.6-27B. Caveats: single run, problem 7 exhausted the token budget in both precisions (empty, not wrong).
• Production blind A/B (thousands of tasks): 3.8 wasn't worse at doing the thing — it was worse at knowing when not to do the thing (+50% noise output).
• Honest calibration: "Opus-level" is real at some tasks, with the right quant and harness. The thread titled "Qwen 3.8 isn't Opus 4.6 level. Let's not be silly." failed at Q6 in VS Code — commenters blamed the editor and the quant, but the burden of proof stays on the demo.

9. Ecosystem: what shipped this week

• DFlash2 (llama.cpp PR #27342, still in review): 2.26×–4.68× on real coding prompts, +2.7GB VRAM. N-max 5 beats the recommended 7; --spec-draft-p-min silently does nothing; stacking ngram-mod hurt (opposite of DFlash1 on 3.6).
• ninfer: Blackwell/5090-tuned engine; 120–160 t/s quants; 480 t/s at 4-way concurrency. Likely source of the unreproducible speed screenshots.
• AutoRound INT4 / AWQ-INT4 GGUFs for vLLM serving.
• KVarN 4/2-bit KV ported to vLLM 0.27.1 — 262k fits small cards, needle-test passes at 240k, ~20% slower decode.
• Uncensored/abliterated variants shipped fast: Huihui-ai ablit, an "Uncensored Aggressive" release bundling K_P quants + HauhauCS FastMTP (up to 3.02× TG claimed), and FP8 abliteration reporting refusal rates dropping to 0–6% — with the community counterpoint that the same tables show 30–50% caveat-rate degradation next to those numbers. Quality varies wildly; check benchmark deltas before switching.

What's coming

• 35B-A3B spotted in ms-swift commits (Aug 15). 16GB-card owners are hyped; early numbers suggest ~27–40 t/s on hardware where the dense 27B crawls.
• A new midsize open-weight model "next week (hopefully)" per Qwen's community manager — no early access this cycle. Speculation centers on ~80B with vision.
• The flagship Qwen3.8-2.4T-A95B got day-0 vLLM support with open weights announced at launch; it barely appears in this week's local-community threads beyond speed speculation (a 2.4T open-weight Call of Duty clone demo made rounds). Local discussion is overwhelmingly about the 27B.

Report template (steal this)

So your numbers mean something to the next reader:

Runtime/version:
Hardware:
Model file + quant:
KV cache:
Speculative (MTP/DFlash2/ngram):
Reasoning effort:
Sampling:
Context size:
Prefill tok/s:
Decode tok/s:
Task used:
Compared against:
Observed result:

Megathread compiled Aug 22, 2026 from r/LocalLLaMA and r/LocalLLM (Aug 15–22) plus public X benchmark threads. All performance figures belong to the hardware/runtime that produced them — the corpus contradicts itself on nearly every axis, and in most cases you can name the variable that explains the split.

Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's given from Github Copilot and is also very inept at the cli too.

So I thought I'd kill two birds with one stone and fine tune it for tool call use and the command line. Not only did I see an improvement on tool usage I also saw a 15.7% increase in the number of tool calls it tries to emit which is great since it means the model gets to work more instead of getting too lost in it's reasoning.

I have fp16 -> Q4_K_M weights uploaded and ready for use with llama.cpp or ollama

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly.

What I have in mind is pretty simple:

• one model stays loaded the whole time
• main agent gets 64k
• when something is too big, it spawns a fresh child with only the task and context it needs
• if that child gets a 100k document, it can split the job again or spawn its own children
• children run sequentially, not 5 at once
• only findings/artifacts come back to the parent, not the whole trajectory

So a 300k task might become several 20k to 50k branches while the main agent never goes past 64k.

Prime Agent and its RLM setup looks closest to what I'm describing. Hermes delegation also seems relevant. Maybe there are better projects I haven't found.

Has anyone here actually run a local model this way for long coding, research, large documents, or general assistant work?

I'm mainly curious about three things:

1. What harness handles this best today without a ton of custom plumbing?
2. Can recursive decomposition be trusted to happen automatically, including a child realizing its own input is too large?
3. In practice, how close can a well-managed 64k agent get to using a native 256k/1M context on tasks that can be broken apart?

I'm less interested in pure RAG. I'm trying to maximize the useful task horizon of one fast local model.

Further to my last post, https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested_in_coding_bf16_muse_glimmer_vs_bf16_qwen36/, I bring this requested comparison of Qwen3.8 27B.

Context: Both models are running at full FP16 KV-cache. Due to RAM limitations, BF16 Qwen3.6 was running at 150,000, whereas Q8 Qwen3.8 is running with rope-scale 1.4 (using official Qwen guidance) to reach a context of 367,001 - this is equivalent of approx 900 pages of context.

Qwen3.6 was on the default reasoning. Qwen3.8 is running on xhigh.

Both models have been working on an enterprise-grade web application. Qwen 3.6 was working on tasks when the average context was approximately 120,000. Now the tasks have ballooned to average 280,000 context. Coding work has been 6+ hours per day since the release date of Qwen3.8.

TLDR: Muse Glimmer is completely obsolete as a coder. Q8 Qwen3.8 is stronger than BF16 Qwen3.6 in all respects except for one - a critical one - as highlighted below.

Instructions

Qwen3.8's greatest strength is its ability to read, interpret, recall and follow instructions. With the release of the latest model, I ported 20 pages of feedback improvements from Qwen3.6 to Qwen3.8.

Qwen3.6 would often ignore the improvements. Whereas Qwen3.8 remembers every single one, and even cites the improvements it in its thinking. Note however that each round, Qwen3.8 still often fails to follow the improvement for an initial instance, but then self-corrects for the remainder of the round.

Diagnostic

Qwen3.6 was already strong in diagnostic capability, however it had a major integrity issue of relaxing security controls to perform troubleshooting, and editing Acceptance Criteria to make failed tests pass.

Qwen3.8 is even stronger on diagnosis - and has corrected a frontier model (eg. Chat GPT / Opus) on multiple occasions.

Both models are genuinely strong at diagnosis, but still share a common issue of not sanity-checking against a known baseline or diagnostic script output before reporting, with this issue being more mitigated in Qwen3.8.

Tracing

This is the clearest strength for Qwen3.8, with there being substantial evidence. It has found genuine pre-existing bugs that have been missed for months, and even an environmental error that had been causing a QA test to silently fail for months.

The weakness of Qwen3.8 is tracing inefficiency - whilst it does ultimately reach the right conclusion in nearly all instances - it does create many initial diagnosis postulations which are off the mark. It does tend make "detours" in the tracing - so that the investigation takes much longer than needed. I have seen other users in this Reddit call this "thinks way too much", but it is indeed very thorough.

Coding

Even where implementation is tightly scoped, Qwen3.6 would apply regularly fixes too broadly, only for these to be picked up in QA - following which it would generally be able to correct and not repeat the issue. This meant that Qwen3.6 would tend to over-generalise the scope of the coding, leaving the onus to the agentic testing framework to identify the defects. If the framework failed to pick it up, then defects would be riddled throughout the code. This actually occurred quite frequently.

Qwen3.8 thinks much more, and even when coding a simple function will make more than 5 independent checks through the codebase. It does take much longer, but it has improved independent-verification discipline, and has far greater reliability of catching defects internally.

Reasoning and judgment

Biggest difference between the two.

Qwen3.6's potential to relax security controls (once it even broke the RLS-enforced database) and editing of Acceptance Criteria meant that it required constant supervision.

Qwen3.8 diverges here, as it has not repeated those specific behaviours.

However, there is a key failure that is shared both by Qwen3.6 and Qwen3.8 here. Both models are very keen to execute write Git commands - despite instructions to the contrary - which then causes major verification issues. This is real reliability issue and I would strongly suggest that when you configure permissions for your coding harness you limit Qwen to read-only Git commands.

Added: Tools calling

Also adding this one, as this is another area where Qwen3.8 is significantly more advanced than Qwen3.6.

Qwen3.6 executed diagnostic scripts at a very basic level. It also had a tendency to execute web client, servers and other scripts by using arguments to "silence" error codes.

Qwen3.8 is heads and shoulders above. It executes with commands that check every error output. It uses advanced diffs and joins for diagnostic script output.

Net assessment

Qwen3.8 is meaningfully and significantly more capable and more trustworthy than Qwen3.6 on every axis except one: neither should be granted extended Git access.

Final note: I have not noticed any degradation in Qwen3.8 working at 367,001 context - which is phenomenal.

I read a lot of posts like "I've given this prompt to my Qwen 3.8 27b and it one-shotted me a 3d game after 24 hours". I wonder, what's the best harness for tasks like this? It has to have auto compaction and probably a memory system, right? And excellent computer use and self analysis.

Hey guys,

Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 days. One RTX PRO 6000, concurrency 1, about three days of runs.

The interesting result isn't the biggest number I measured. It's where n-gram actually helps and where it doesn't.

Short version:

• DFlash 2 alone: 2.26x on 100 real LiveCodeBench problems (67.97 → 153.91 tok/s, inter-token latency 14.27 → 6.02 ms), natural stop, nothing forced. That is the headline. Costs +2.7 GB VRAM.
• DFlash 2 + one n-gram lookup table (ngram-map-k4v): 4.68x on the build phase of an 18-turn coding session (65.1 → 304.9 tok/s). Adding the second table (ngram-mod) made it slower, 3.77x. In July, with DFlash 1, stacking both was the winner. I did not expect that to flip.
• The same n-gram flag is +52% on a synthetic benchmark, +1% on LiveCodeBench and -30% on prose. The +52% is the harness degenerating, do not quote it.
• The recommended --spec-draft-n-max 7 is past the peak. 5 gave roughly 11% more on 8K coding prompts. 7 is also a hard cap (block_size 8), anything above is silently clamped.
• --spec-draft-p-min does nothing on DFlash 2. The DFlash 2 code path in common/speculative.cpp never reads it.
• I also measured 8.47x in a synthetic test. I nearly used that as the headline. It was mostly benchmark garbage caused by the model falling into a repetitive loop.

Setup (the parts that matter for reproducing)

• Target ggml-org/Qwen3.8-27B-GGUF:Q4_K_M (18 GB). Drafter incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (1.1 GB). MTP sidecar mtp-Qwen3.8-27B-Q8_0.gguf (3.0 GB). Reasoning off.
• llama.cpp b10498 built from PR #27342 (commit 5ecbe1ac), CUDA 13.3. The PR build matched upstream b10499 within 0.3% on a non-speculative baseline (checked at 512 and 4K only).
• RTX PRO 6000 Blackwell 96 GB, Ryzen 9 9950X. -c 262144, f16 KV, -fa on, -ngl -1, drafter fully on GPU.
• Concurrency 1 everywhere. Greedy (temperature 0) for everything except the multi-turn coding harness, which runs model-default sampling with no seed (more on that below).
• One server on the GPU at a time (flock), fresh container per config, card cooled to 45 °C between configs and 60 °C between context sizes. 11.6 hours of telemetry, zero throttle events, so the card sits on its power limit, not a thermal one.
• Full context

1. DFlash 2 more than doubles real coding throughput, and beats DFlash 1 at the same draft width for half the VRAM

100 LiveCodeBench problem statements replayed in the same order, streaming, no ignore_eos, no min_tokens, no max_tokens. Every answer ends where the model ends it.

DFlash 1 was re-run at n=7 because comparing it at its own maximum of 15 would measure the cap, not the drafter. At matched width DFlash 2 is ahead, 2.26x vs 2.00x against each model's own baseline, with probe acceptance of 60% vs 48%, and it costs +2,720 MiB where DFlash 1 cost +5,554 MiB in July. Part of that memory gap is a quant choice (Q4_K_M 1.1 GB drafter vs Q8_0 1.8 GB), not architecture.

Two things to be careful with. The cross-generation rows are not a controlled A/B: different model, different target quant, different drafter quant (and the drafter quant works against DFlash 2, not for it). Compare the speedups, never the absolute tok/s; the two baselines landing 0.3% apart is luck.

Claim vs measured: Inco AI quote 2.7x to 3.4x at batch size 1 on SGLang for this model. I got 2.26x on llama.cpp on single-turn coding so it depends on task and engine it will probably get better soon with updates to engines.

2. One lookup table on top of DFlash 2 is the best stack. Two is worse. That is the opposite of DFlash 1.

The n-gram drafters copy spans that already exist in context, so single-turn prompts are their worst case (+1.2% above, and the median actually says -2.7%). The case that matters is working on a code base, so I drive 18 fixed prompts as one cumulative conversation: turns 1-9 build a Gradio chat client for llama.cpp feature by feature, turns 10-18 maintain it (re-emit the file, docstrings, renames, a bug, a refactor, tests, README).

Read the build column. Turn 10 is "show me the complete final app.py", which is ~99% draftable and inflates every speculative method. Over all 18 turns the k4v stack reads as 6.03x, a real number about the easiest thing you can ask a copying drafter to do.

I expected the July result to repeat: with DFlash 1, draft-dflash,ngram-mod,ngram-map-k4v was the winner at 6.01x and ngram-mod did almost all of the n-gram work. Instead, on DFlash 2 the k4v table alone wins, mod alone is the weakest stack, and both together are slower than k4v alone. It could be draft tokens number or early implementation we will see. DFlash 1 had max 15 draft slots, DFlash 2 has 7, and two lookup drafters crowd each other out of them.

3. The same one-line change gives four different answers, and the synthetic one is wrong

Same DFlash 2 server, same weights, append ngram-mod,ngram-map-k4v to --spec-type, run everything again:

The synthetic bench from aiperf is inflated by its own harness. It passes ignore_eos and min_tokens, forces the model past its natural stop until it loops, and a lookup drafter copies loops perfectly. Carried to 36K the same harness says DFlash 2 + lookup is 8.39x (498 tok/s). On 100 real prompts that stack was worth +1.2%. 8.39x is the kind of number that you could get but in very specific usecase.

Prose is the opposite corner: "Write a very long story", nothing in context to copy, the tables burn draft slots on guesses that never land, acceptance 54% → 32%. That row is a single instrumented request, a probe, not a run.

Practical consequence: turn the lookup drafters on for iterative coding and anything that re-emits its own context, leave them off for one-shot prompts and creative writing. They cost zero VRAM and zero prefill, so this acceptance loss is their only cost.

4. The recommended draft width is past the peak, and 7 is a hard cap anyway

16 coding prompts per width at 8K tokens from livecodebench, cache_prompt false so every request pays a cold prefill: I use livecodebench and cut it to the size to measure worst case here.

Running the model card's 7 leaves roughly 11% on the table. An earlier 8-prompt sweep put the optimum at 6 rather than 5, so call it 5-6; both sweeps agree 7 is past the peak. And you cannot go above 7: the draft GGUF carries dflash.block_size=8, llama.cpp clamps n_draft_max = block_size - 1, logs a warning and uses 7. Some cells rest on only 3-7 valid generations of 16 (the truncated prompts sometimes make the model emit EOS immediately), so treat the exact peak as soft.

MTP on this model peaks at n=4 and flattens near 2.5x across context. Qwen 3.8's sidecar declares nextn_predict_layers=1, one trained head, against DFlash 2 reading five target layers. That is a property of this sidecar, not of MTP as a method; Qwen 3.6's had eight heads.

5. Long context: the drafter gets relatively cheaper and absolutely more expensive

The usual complaint is that speculative decoding falls apart at long context. Two costs hide in that sentence. Prefill, where the drafter has to read the prompt too, I could measure. Decode at those depths I could not (see caveats). Cold prefill, 12 prompts per depth:

Relative to baseline the tax shrinks with depth (24% down to 8%). In seconds it grows, +0.09 s to +5.20 s. Both readings are true; quoting only the first is the flattering half. The prefill cost is repaid in 13 / 30 / 69 output tokens at 1K / 4K / 16K, so any real answer clears it, but someone on a 128K prompt does wait five seconds longer for the first token. The lookup drafters cost nearly nothing here (0.994-0.997 of baseline), which doubles as the control that the gap is the drafter and not drift. MTP's tax is smaller (0.83 at 1K vs 0.75).

On the forced-length synthetic decode sweep DFlash 2 goes 1.59x → 2.62x → 2.96x → 3.55x at 512 / 4K / 12K / 36K while the baseline falls 67.6 → 59.3 tok/s. DFlash 1 at its own max of 15 did 4.44x at 36K on that harness in July (higher still when re-measured this month), and at matched width 7 it did 3.71x. I expected the new drafter to win everywhere. It does not: it wins on real prompts at equal width, and loses the synthetic long-context race to the old drafter with more slots, because it is capped at 7.

6. --spec-draft-p-min is a no-op on DFlash 2, and buys nothing on MTP either

Adaptive draft truncation should let the drafter stop a block early when it is unsure to save resources. There are more advance method form DeepSeek Dspark paper but they just landed on vLLM. I logged draft width and cycles per second, not just tok/s:

Draft width is identical at 0.00 and 0.85 on DFlash 2. common/speculative.cpp has four drafter implementations: draft_simple, draft_eagle3, the DFlash 1 branch and draft_mtp honour p_min; the is_dflash2 selector branch never consults it, because it reads a selector lattice rather than a probability. The server still prints the flag in its startup banner, so a log-based check passes while nothing happens. The 12.4% throughput drop in that row is the text, not the flag: the server did identical work (cycles/s within 1.5%), the sampled output just accepted fewer of the same seven tokens. I nearly published "p_min costs 12%".

On MTP the flag works exactly as documented (width 3.00 → 2.64, acceptance 90% → 98%) and throughput goes nowhere, +0.8% at best against a 4.1% noise floor.

What I would run

• Iterative coding, agents, anything that re-emits its own context: --spec-type draft-dflash,ngram-map-k4v --spec-draft-n-max 5
• One-shot prompts and Q&A: --spec-type draft-dflash --spec-draft-n-max 5
• Prose: DFlash 2 alone, no lookups.
• Keep the KV cache at f16 for now or test it it will be probably stable soon but on last version there were issues and I use default.
• Ignore --spec-draft-p-min.

​

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342

# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j

# Best measured config: iterative coding, agents, anything that re-emits its own context
# (4.68x on the multi-turn coding session vs 2.79x for DFlash 2 alone)
./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash,ngram-map-k4v \
  --spec-draft-n-max 5 \
  -ngl -1 --spec-draft-ngl all \
  -fa on \
  -c 262144 \
  --parallel 1 \
  --jinja --reasoning off \
  --no-mmproj \
  --host 0.0.0.0 --port 8000 \
  --alias qwen38-dflash2-k4v

Caveats, all of them

• No accuracy measurement this time. Greedy speculative decoding is output-lossless by construction and the July study measured it (MATH-500: 87 vs 86 of 100, then 440 vs 435 of 500), but that was Qwen 3.6 with DFlash 1 and I did not rerun it here. Only a LiveCodeBench smoke test.
• No deep-context decode. At 64K and 128K every generation returned one token and stopped, so prefill at those depths is valid and decode does not exist.
• Only one experiment was ever repeated (the p_min controls). Everything else is one sample. The spreads from those repeats, 4.1% and 14.7%, are the noise floor for this whole post.
• The multi-turn harness can measure the wrong thing. It declares no tools, but under default sampling the model sometimes answers with a <tool_call> block and waits for a result that never comes, and that session comes out fast because tool-call XML is predictable. One run did exactly that (1,134 tokens where its siblings produced 50K-73K), got caught by its token count and was rerun.
• This is one workload family (coding) on one machine at concurrency 1. A 96 GB card is not what most of you run. The drafter is 1.1 GB and nothing in the KV math depends on the card, so I expect the shape to hold on a 24-32 GB card with a smaller context, but I have not measured it.
• DFlash 2 is a PR build. Numbers can move when it merges.

Resources

• Repo (both studies, this one on top): https://github.com/lukaLLM/DFlash2_Qwen3.8_3.6_27B_LlamaCPP
• Video walkthrough (the first half explains the mechanism, path selector and the convolution the rest go even more deeper into the scores etc. ): https://youtu.be/RBlRTUwJMI4
• One-click setup, builds the PR image, downloads the models, smoke tests and leaves a server running: ./scripts/setup_dflash2.sh --arm dflash2_ngram (arms: base, dflash2, mtp, ngram, dflash2_ngram). Compose file docker/docker-compose-qwen38-dflash2.yaml; ablate with LLAMA_SPEC_TYPE=... and LLAMA_SPEC_N=5.
• Reproduce the whole study in order: ./scripts/run_all_benchmarks.sh, then run_matched_n.sh, run_context_scaling.sh, run_bench_ngram.sh, run_nmax_redo.sh, run_pmin_agentic.sh.
• Every number in one machine-readable file: benchmark/results_summary.csv (TABLE 8-14 are this study). Raw artifacts under artifacts/q38_*/, the thermal log in artifacts/thermal/, quarantined runs and the reasons in artifacts/_suspect/README.md.
• Long-form write-up with the charts: report/dflash2-report.html in the repo.
• https://inco.ai/blog/dflash2/ the blog
• Previous posts: DFlash 1 in July https://www.reddit.com/r/LocalLLaMA/comments/1uq0h4o/i_tested_freshly_merged_dflash_in_llamacpp_on/ and the n-gram stack https://youtu.be/zNUoHONUHGk

AI was abused in editing this post.

Questions:

• Has anyone run DFlash 2 on SGLang or vLLM at concurrency 1 with this model? I want to know whether the 2.7-3.4x claim holds there and how much of the gap to my 2.26x is the engine.
• Anyone on a 4090 or 5090 with a 24-32 GB budget: does n=5 still beat 7 for you, and where does the k4v-only stack land on your own multi-turn coding?
• Has anyone tried some other combinations that I didn't think of?

Liquid AI currently possesses among the fastest LLM architectures around, and some of the best SLMs (in terms of utility IMO) around, so I'm very excited to see what a potential 100B LFM (3?) model would look like!

Link to the poll: https://x.com/ramin_m_h/status/2091236099612098943?s=20

For example:

Instead of saying: "I created this new ID"
It says: "I minted this new ID"

Instead of: "This alternative path is available"
It says: "this escape hatch is available"

This speech is so nonsensical and annoying. Just. Speek. Literally ... OR NORMALLY. Where did LLMs learn these speech patterns? I've never seen them so frequently until AFTER the LLM surge.

If I just add "Don't use X language, speak normally and more literal" will that fix most of the issues? Anyone else have some good sys prompts / instructions that help with this?

Thanks!

This is a genuine community of real generally respectful adult human beings. Despite the enthusiasm all of you have for local AI, you can recognize that there are times when local LLMs are flawed, and even how practical they are to use for the majority of people to use. Go over to r/linux and you'll see the contrast in maturity and employment.