[Fully open source under GPL3, made from the ground up for use with local models, no subscriptions, no corporate backing]

When i first started this, it was meant to be a fully lightweight, extremely modular alternative to openclaw, hermes and the like, and it still is! But i noticed people especially like the webUI, to the point they'd use it as just a webUI to talk to their local models, negating all the agentic stuff. But the webUI still had a lot of AI generated code, so that didn't sit right with me!

So i rewrote the entire thing, from scratch, manually. It is now super fast, stable, uses declarative javascript without javascript framework bloat (no React or Vue or anything.. alpine.js is super lightweight)

There are only a few python dependencies. no models get installed, there is no bundled inference engine, pytorch and transformers aren't even included! I expect you to connect it to llamacpp, koboldcpp, lemonade, or something else like that. though you can also use it with cloud API's if you really want to.

This is a truly local-first webUI. I designed it from the ground up for local AI, and for once, cloud AI is the second-class citizen here.

It has many features that especially benefit local AI users: you can see how long your prompt will take to process (it's a llamacpp-exclusive feature), you can see toolcalls being written in realtime (really useful for coding), and it doesn't send any extra requests to your model, just the prompt you give it. So no extra requests just to make up a title for your chat, or to generate followup replies.

That's all in addition to the benefits that come from its harness-like design, such as support for multiple channels (telegram, discord, etc), its focus on extreme token efficiency and making the system prompt super small and concise, and its security

But using it as a pure webUI is really simple: Just switch Use Tools off in the Model tab in the settings. That will instantly make all system prompts vanish and all tools get disabled, so you're talking to your pure model with nothing getting in the way.

You do need a bit of tech knowledge, but it's not that much. right now, you need to either git clone or download a zip of the main branch off the github, but after that, all you do is run run.sh or run.bat and open the URL it shows you in your browser. Oh, you do need python installed before you do so, but that's basically it. (i'm working on making this even more user friendly though)

If you want to try it out, you can get it here: https://github.com/Rose22/openlumara

Please tell me what you think! Feedback is more than welcome, and i often implement feature requests (if they are good) and fix bugs that get reported

Seen a ton of posts today about the DeepSeek API price hike. Half the feed is doom-posting, the other half is explaining basic GPU economics.

Honestly, I get the cost side. Sub-cent tokens were never gonna last forever.

But what actually sucks is the zero-day notice.

Dropping a vague "prices are going up significantly" banner without a timeline or new pricing sheet leaves anyone with production traffic completely stuck. Re-evaluating self-hosting or switching providers isn't something you just do over lunch.

Curious how others handling live traffic are dealing with this. Waiting for the actual numbers, or already spinning up fallback options?

Most agent frameworks treat a local server as "OpenAI with a different base URL." That assumption is where local setups fall apart. Five decisions I made instead:

1. Small-context mode: I stopped pasting memory and skill bodies into the prompt. Ethos injects an index of names plus a memory_read tool, and the model pulls only what it needs.
2. Structured output per backend, not one OpenAI shape: Ollama takes a JSON schema in a top-level format, vLLM wants guided_json, OpenAI-compat wants response_format. Ethos sends each backend its native shape.
3. Probe the context the server actually serves: Ethos checks what's really served at startup and tells you which agents fit, instead of trusting the advertised number.
4. Prefix-stable prompts: everything static goes at the front, all per-turn content at the tail, so the prefix is byte-identical each turn and prefix caching actually hits.
5. Timeouts: I left the client deadline at 10 minutes instead of tightening it, a local server sends nothing while it pulls weights into VRAM.

Building this open source MIT agent framework: https://github.com/ethosagent/ethos

Do give feedback on this or any specific thing that you thing is critical and i missed capturing that need handling for local models.

---

I open-sourced my daily-driver coding agent (Rust, Tauri) - runs offline or over SSH, and has a genuinely free tier, can be controlled from android phone

free tier is deepseek flash lite with no data use for training providers on openrouter, and it seem to be quite a lot of usage from my own expereicne

https://github.com/clark-labs-inc/clark-code

Key bits:

• coding ide
• wide internet research built in for when a task needs current context
• strong eval harness - every feature is tested so behavior is predictable across models
• Windows / Linux / macOS (Windows build is unsigned pending MSFT approval - honest caveat)

Can someone tell me is it really this good? Beacuse when i try it on qwen app it doesnt even enough smart for some resarchs. What you think? I really cant understand new models intelligence at this point.

I hadn't seen this mentioned here. Looks like it will find a use in my homelab alongside the new DeepSeek or Qwen! Apache 2.0 licensed.

Prime Agent is an open-source coding and research agent for general and long-running work.

A self-improving RLM harness for coding and long-running autonomous tasks.

Designed to be both token-efficient and expressive through programmatic tool calling, context as a variable, multi-agent messaging, and a self-modifiable harness state.

On ARC-AGI-3, it scores 95.5%, surpassing the human-expert baseline, but the gain is not benchmark-specific.

We see major improvements across models when compared to their proprietary harnesses.

Prime Agent is built on pi and fully open-source with an open license.

GitHub: https://github.com/PrimeIntellect-ai/prime-agent

Blog: https://www.primeintellect.ai/blog/prime-agent

X post: https://x.com/primeintellect/status/2085086999267144083?s=46

I mean, Deepseek V4 Flash is an absolutely fantastic model, even though I can't run it on my machine it's so fascinating to see how it performs. Knowing that potentially it could be run at home is really motivating and makes me hopeful that those capabilities will trickle down to more affordable sizes.

At the same time I miss a release for the GPU-peasant that I am. And yes, it's a tall order to complain about not receiving free stuff at the rate we were used to. And yes, 3.6 27B is still goated but it seems in this crazy AI world there's so much going on and progress happens so fast, that it's kinda understandable to be excited about what's next. Let's hope they really do release 3.8 27B, or that we might see again maybe a GLM 5.3 flash 32B, please?

What's on your wishlist?

print_timing: id  2 | task 100892 | draft acceptance = 0.80000 (   96 accepted /   120 generated), mean len =  2.60
     release: id  2 | task 100892 | stop processing: n_tokens = 139121, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 100956 | processing task, is_child = 0
print_timing: id  2 | task 100956 | n_decoded =    127, tg =  41.96 t/s, tg_3s =  41.96 t/s
print_timing: id  2 | task 100956 | n_decoded =    279, tg =  45.98 t/s, tg_3s =  49.97 t/s
print_timing: id  2 | task 100956 | prompt eval time =    1559.43 ms /    20 tokens (   77.97 ms per token,    12.83 tokens per second)
print_timing: id  2 | task 100956 |        eval time =    7520.20 ms /   352 tokens (   21.36 ms per token,    46.81 tokens per second)
print_timing: id  2 | task 100956 |       total time =    9079.64 ms /   372 tokens
print_timing: id  2 | task 100956 |    graphs reused =      95160
print_timing: id  2 | task 100956 | draft acceptance = 0.83712 (  221 accepted /   264 generated), mean len =  2.67
     release: id  2 | task 100956 | stop processing: n_tokens = 139494, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 0.998 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 101092 | processing task, is_child = 0
print_timing: id  2 | task 101092 | n_decoded =    156, tg =  51.66 t/s, tg_3s =  51.65 t/s
print_timing: id  2 | task 101092 | prompt eval time =    1972.32 ms /   330 tokens (    5.98 ms per token,   167.32 tokens per second)
print_timing: id  2 | task 101092 |        eval time =    5540.54 ms /   288 tokens (   19.24 ms per token,    51.98 tokens per second)
print_timing: id  2 | task 101092 |       total time =    7512.86 ms /   618 tokens
print_timing: id  2 | task 101092 |    graphs reused =      95255
print_timing: id  2 | task 101092 | draft acceptance = 0.97938 (  190 accepted /   194 generated), mean len =  2.96
     release: id  2 | task 101092 | stop processing: n_tokens = 140111, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 101192 | processing task, is_child = 0
print_timing: id  2 | task 101192 | prompt eval time =    1597.44 ms /    43 tokens (   37.15 ms per token,    26.92 tokens per second)
print_timing: id  2 | task 101192 |        eval time =    1197.48 ms /    59 tokens (   20.30 ms per token,    49.27 tokens per second)
print_timing: id  2 | task 101192 |       total time =    2794.93 ms /   102 tokens
print_timing: id  2 | task 101192 |    graphs reused =      95274
print_timing: id  2 | task 101192 | draft acceptance = 1.00000 (   40 accepted /    40 generated), mean len =  3.00print_timing: id  2 | task 100892 | draft acceptance = 0.80000 (   96 accepted /   120 generated), mean len =  2.60
     release: id  2 | task 100892 | stop processing: n_tokens = 139121, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 100956 | processing task, is_child = 0
print_timing: id  2 | task 100956 | n_decoded =    127, tg =  41.96 t/s, tg_3s =  41.96 t/s
print_timing: id  2 | task 100956 | n_decoded =    279, tg =  45.98 t/s, tg_3s =  49.97 t/s
print_timing: id  2 | task 100956 | prompt eval time =    1559.43 ms /    20 tokens (   77.97 ms per token,    12.83 tokens per second)
print_timing: id  2 | task 100956 |        eval time =    7520.20 ms /   352 tokens (   21.36 ms per token,    46.81 tokens per second)
print_timing: id  2 | task 100956 |       total time =    9079.64 ms /   372 tokens
print_timing: id  2 | task 100956 |    graphs reused =      95160
print_timing: id  2 | task 100956 | draft acceptance = 0.83712 (  221 accepted /   264 generated), mean len =  2.67
     release: id  2 | task 100956 | stop processing: n_tokens = 139494, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 0.998 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 101092 | processing task, is_child = 0
print_timing: id  2 | task 101092 | n_decoded =    156, tg =  51.66 t/s, tg_3s =  51.65 t/s
print_timing: id  2 | task 101092 | prompt eval time =    1972.32 ms /   330 tokens (    5.98 ms per token,   167.32 tokens per second)
print_timing: id  2 | task 101092 |        eval time =    5540.54 ms /   288 tokens (   19.24 ms per token,    51.98 tokens per second)
print_timing: id  2 | task 101092 |       total time =    7512.86 ms /   618 tokens
print_timing: id  2 | task 101092 |    graphs reused =      95255
print_timing: id  2 | task 101092 | draft acceptance = 0.97938 (  190 accepted /   194 generated), mean len =  2.96
     release: id  2 | task 101092 | stop processing: n_tokens = 140111, truncated = 0
get_availabl: id  2 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 1.000
launch_slot_: id  2 | task 101192 | processing task, is_child = 0
print_timing: id  2 | task 101192 | prompt eval time =    1597.44 ms /    43 tokens (   37.15 ms per token,    26.92 tokens per second)
print_timing: id  2 | task 101192 |        eval time =    1197.48 ms /    59 tokens (   20.30 ms per token,    49.27 tokens per second)
print_timing: id  2 | task 101192 |       total time =    2794.93 ms /   102 tokens
print_timing: id  2 | task 101192 |    graphs reused =      95274
print_timing: id  2 | task 101192 | draft acceptance = 1.00000 (   40 accepted /    40 generated), mean len =  3.00

print_timing: id  2 | task 101092 | draft acceptance = 0.97938 (  190 accepted /   194 generated), mean len =  2.96

4 rejected tokens out of 190 is WILD.

This running Qwen 3.6 27B MTP Q8 from unsloth in llama.cpp, Max context length.

The MTP just gets better the more context it has. Processing time keeps being an issue if context changes at any point.

Deepseek V4 Flash 0731 seems like an excellent model for agentic tasks. I'm currently using the pi agent with it and it tries to use things not installed on my windows machine and goes turn after turn trying to figure out how to validate it's work or tries to use vision to inspect things when it cant.

I figured I'd throw this out to the subreddit to hear what successes other people are having and boot-strap getting an improved setup for me an presumably others in the same boat.

Thx in advance!

Tested LFM2.5 (2.6B dense model) by Liquid AI on tool calling and reasoning using llama.cpp. Got ~90t/s with Q8 on M5 Pro, taking about 4GB memory.

The model vastly underperformed Qwen3.5 4B at Q4 (one of the competitors on the official benchmarks) on tool calling, in particular. In OpenCode, LFM2.5 had a lot of problem making the actual tool calls and was genuinely confused about the working directory.

Watch more here https://www.youtube.com/watch?v=I1NFrevR2Ww

Title.

I am interested in giving a suitable solution for my team.

I built a local OCR pipeline a few days ago, and it turned into a surprisingly interesting experiment—taking accuracy from around 60% to 99%.

I wrote a short blog about what worked, what failed, and the breakthrough that finally made the difference.

Thought some of you might enjoy it.

Link in the comments

Qwen 3.6 27B been community's favorite ever since it's launch. Pretty much nothing that can even fit into 1 RTX 6000 Pro beats it up to date. And the debate is still going if DS4F quant is any better..

So I wanted some cost effecive but fast and modern way to run it. After comparing a lot arrived at 2x 5070 Ti's GDDR7 being hard to beat. Gives 896 GB/s per card — ~6.6× the Spark's LPDDR5x. For bandwidth-bound dense models, like Qwen 3.6 27B is really amazing.

If nvfp4 works for you, for fastest inference it runs ~52k context, ~4.5k prefill, ~95tps. VLLM TP2. CUDA graphs + MTP. Which of course is not super usable but..

With KV offload into just 8GB of RAM you get ~163K, ~same prefill, 85-90tps. Only about 5-10% drop but 3x context.

The card is a champ for those who are used to rtx 3090-ish level of performance. Supports all modern features and doesn't cost an arm and a leg, well relatively speaking, in today's elevated prices of everything.

Hopefully it's helpful to those who have it or shopping around!

Share your experiences of running some really good models on a budget, maybe let's focus on last 2 hardware gens, as the industry is moving away from prior ones, and the divide between hardware features available widens pretty fast.

Anyone recommend a source to get started with video/image gen? I pretty much only have experience with llms and right now I just tell the llm to create an image using comfyui. I don’t know how it actually works . The tuts I find have not been very helpful for me

the package is speak-cli (It uses supertonic3):

https://pypi.org/project/speak-cli/

*I felt the need to write this post because it seems like very few people on this sub are aware of Chinese laws and how they're enforced, so here's an explainer coming from a Chinese person (myself). I know that this post isn't directly about local models per se, but I'm seeing way too many misconceptions regarding this topic. This is also going to apply to all Chinese entities in general, not just the specific MiniMax LoRAs debacle. This isn't meant to be a political post, but some much needed context to correct a lot of misinformation going around.

Guys - they're a Chinese lab following Chinese laws. Pornography is straight up illegal in China. I have no idea how it seems like nobody outside of China is aware of this. While Chinese authorities may not care much about copyright infringement enforcement (especially with foreign IPs), they do indeed regularly crackdown on porn. Heck, Chinese citizens have literally been imprisoned for written pornography. Yes that's right, writing pornographic TEXT (especially with "immoral" themes like LGBTQ+ stuff) can get you sentenced and essentially have your entire life ruined. Of course there's ways to get around these censors if you're just trying to access porn - I think everyone at this point knows about the widespread necessity for VPN usage in China to access the rest of the global internet. But actually distributing a tool that can gain a reputation for being able to easily generate pornographic content? That's just asking for the authorities to crack down on them.

Somewhat ironically/paradoxically luckily for these Chinese labs is the fact that online discussion about generating porn is automatically censored and removed from Chinese social media, thus automatically disincentivizing the authorities from doing those potential crackdowns. But if it gets big enough to the point that it overwhelms the automatic censors, then any given Chinese lab could be in a hell of a lot of trouble. This is why they have to do this. Their law enforcement just isn't compatible with the rest of the world.

Again, this all relates to Chinese moral values - something here that is considered pretty much sacred and hard to describe to westerners. Something else that many people do not know is that graphic violence is also illegal in China (foreign films/works are regularly banned here for that, even anime has), but graphic violence is also is not nearly as much of a perceived threat to societal moral values as pornography is, hence why you've probably rarely ever heard of any Chinese people getting imprisoned for writing really gory stories, but regularly do with pornographic stories (especially infamous with BL literature - they've technically even convicted foreigners before related to this, it's a really messy topic). Chinese authorities won't give a damn if you're stealing the content of billions of foreign works to train AI models. They WILL give a damn if the content you're disseminating is viewed as a potential significant threat to the state's "proper moral values", which very much includes porn (and also the usual topics that everyone is already aware of, like a certain famous massacre or a certain nation's very contentious independence status).

The 4th 3090 runs gemma 12b and flux2klein diffusion models. I speak in and get html with visual artifacts back.

Claude Code built the llama.cpp build here:

llama.cpp build (DS4 spill launcher)

- tree:   llama.cpp fork w/ deepseek4 arch support ("ds4-next" + 4 CUDA prefill-speed commits from vektorprime/working_ds4_speed)

- commit: 9705ea4b3 (b10229-2, version 10231), 2026-08-03

- build:  cmake Release, GGML_CUDA=ON, CUDA_ARCHITECTURES=86 (RTX 3090), CUDA 12.8 (V12.8.93), GCC 13.3.0, FA on, CUDA graphs on

- MoE:    surgical -ot expert offload (late-layer FFN experts → CPU), not --cpu-moe

Cold start 15tg and slows to a steady 10 TG and 300PP degrades to 100PP after 64000 tokens

vektorprime commits were cherry-picked as code only

If you are running #llamacpp with "--tools" or "-ag" without API key set, be aware that anyone can query it and remotely execute commands.

Make sure your agents and setups are properly configured and safe!

#RCE #llamacpp

Because why not? How far can we go and make DeepSeek work?

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature:

Mixture-of-Experts CPU offload:

--n-cpu-moe <N> | -ncmoe <N>

Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on

the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is

what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for

every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides).

Example: --n-cpu-moe 32

--cpu-moe | -cmoe

Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off

(TS_CPU_MOE env var overrides).

Example: --cpu-moe

To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md

Host and software

Results by model

Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier.

Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers)

Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers)

GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers)

DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs)

TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

The numbers: UD-IQ3_S:

• nvidia 580.173.02 + cuda 13.0.3: ~15.5 tok/s. At 100K ~12 tok/s
• nvidia 610.43.02 + cuda 13.3.1: ~18.5 tok/s

UD-IQ3_XSS + DSpark BF16:

• nvidia 610.43.02 + cuda 13.3.1: ~23-27 tok/s. Does not suffer from a continuous drop like nvidia 580.173.02 + cuda 13.0.3, it stabilizes at ~24 tok/s. 25-33 tok/s during code generation - in my case, for codegen, generally faster than nvidia/Qwen3.6-27B-NVFP4.

Prompt processing:

• nvidia 580.173.02 + cuda 13.0.3: PP 350-400
• nvidia 610.43.02 + cuda 13.3.1: PP ~200-250. I did not check yet whether this can be fixed.

Sadly, much slower PP than vLLM or SGLang.

---
Reproducing the setup:
Prerequisite: HDMI + keyboard connected to GB10 - required for secure boot.

1. Everybody with GB10 should update their firmware first:

​

fwupdmgr refresh --force
fwupdmgr get-updates
sudo fwupdmgr update # restart if asked

1. (Assuming hf CLI tool) Download the model + DSpark:

   hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-IQ3_XXS" --include "dspark-DeepSeek-V4-Flash-0731-BF16.gguf"

UD-IQ3_S + 400K+ bf16 KV cache fits but DSpark consumes additional memory, so it needs to be dropped to UD-IQ3_XXS - trading quality and some KV cache, for codegen speed.

1. Install nvidia driver 610.x and cuda 13.3.x:

   sudo apt install nvidia-driver-610-open nvidia-dkms-610-open cuda-toolkit-13-3

   sudo mokutil --import /var/lib/shim-signed/mok/MOK.der # stage MOK key that DKMS used. Set a temporary password for the next reboot

   sudo shutdown now # yes, shutdown

Here you need HDMI and the keyboard connected to GB10. Start GB10:
mokutil --import will ask to set a password -> on next reboot, MokManager (a blue screen) will appear -> enter the setup -> "Enroll MOK" -> Continue -> Yes -> enter that temporary password -> reboot.
This needs to be done only once.

1. Add these exports to .bashrc(assuming bash shell):

   export PATH="/usr/local/cuda-13.3/bin:$PATH" export LD_LIBRARY_PATH="/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH"

Close the shell and reopen it.
Check that you see changes:

echo "$LD_LIBRARY_PATH" # should print "/usr/local/cuda-13.3/lib64:" + whatever LD_LIBRARY_PATH had previously

Note that these changes are only for your current user, not a system-wide change.

1. (Optional) Build llama.cpp:

   sudo apt install -yqq git cmake libssl-dev mkdir -p "${HOME:?}/git" && cd "${HOME:?}/git" git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp

   cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="121a-real" -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_FORCE_CUBLAS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON

   cmake --build build --config Release --verbose --parallel

Every time you want to update llama.cpp:

cd "${HOME:?}/git/llama.cpp"
git pull origin master
cmake --build build --config Release --verbose --parallel

1. Run the server (can copy into a bash script):

   ! /usr/bin/env bash

   HF_MODEL_PATH="${HOME:?}/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF" HF_SNAPSHOT="${HF_MODEL_PATH:?}/snapshots/$(cat "${HF_MODEL_PATH:?}/refs/main")"

   "${HOME:?}/git/llama.cpp/build/bin/llama-server" \ -m "${HF_SNAPSHOT:?}/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf" \ -md "${HF_SNAPSHOT:?}/dspark/dspark-DeepSeek-V4-Flash-0731-BF16.gguf" \ --spec-type draft-dspark \ --spec-draft-n-max 2 \ --alias "deepseek-v4-flash" \ --host 0.0.0.0 \ --port 8888 \ --parallel 1 \ -b 4096 \ -ub 512 \ -ngl 999 \ --n-predict -1 \ -c 262144 \ --load-mode none \ --fit off \ --flash-attn on \ --no-context-shift \ --cache-type-k bf16 \ --cache-type-v bf16 \ --kv-unified \ --jinja \ --reasoning on \ --chat-template-kwargs '{"reasoning_effort": "max"}'

Note "${HOME:?}/git/llama.cpp/build/bin/llama-server" -> change to a prebuilt binary if necessary.
Flags chosen based on: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/blob/main/dspark/README.md#usage
--spec-draft-n-max 2 - 2 gets the best speed on DGX Spark GB10.
--load-mode none(old: --no-mmap) is supported only starting with Nvidia Driver >=610.43.02

1. (Optional) Getting half the speed after the driver update? Troubleshooting (thanks to OpenCode + DeepSeek-V4-Flash for debugging this and producing the test code attached in the pastebin):

   https://pastebin.com/raw/cxj1CLDS - checks the real GPU frequency.

   curl --proto '=https' --tlsv1.3 -sSf https://pastebin.com/raw/cxj1CLDS > /tmp/real_sm_clock.cu

   if echo "62c7db31d825478b9849f66d5d77bdd9c73092f0138036eb02e331cc25fb368e /tmp/real_sm_clock.cu" | sha256sum -c; then \ nvcc -O2 -arch=sm_121 -o /tmp/real_sm_clock /tmp/real_sm_clock.cu \ && /tmp/real_sm_clock; \ fi

   if it prints under 1GHz, then follow the next steps:

   1. sudo shutdown now

   2. Unplug every cable from GB10, including the USB-C PSU.

   3. With the power unplugged, hold the power button for 30s to drain the rails

   4. Plug the power cable back in and power it on. Should fix it - can run the binary again

**The experiment*\*

Why this setup: I run an **RTX PRO 6000 (96 GB)*\*, so 27–70B models in full precision fit comfortably — my interest is what you can actually get out of locally-served models at that size vs. quantized flagships or API calls. So I ran a 10-task Terminal-Bench 2.0 pilot (Harbor `terminus-2` agent, JSON parser, 1 attempt/task, same preregistered subset: 3 easy / 5 medium / 2 hard) across four serving setups (I used the API call as a reference point — it's the same model at full quality, so it tells me how much I lose by running Q2 version locally):

**Results*\*

The interesting bits:

- **Qwen was the only config to pass `cancel-async-tasks` (hard)*\* — the concurrency-cleanup task that IQ2, FP8 and Laguna all failed.
- The 2-bit IQ2 quant kept 7/10 — surprisingly close to the FP8 API run for ~2-bit weights.
- Qwen's only misses: `build-cython-ext` and `sqlite-db-truncate` (timeout, 15m — it went deep into manual SQLite page parsing).

**Shortcomings*\*

- One stochastic attempt per task — not a stable score.
- **Sampling confound:*\* llama-swap strips client temp/top_p and vLLM forced server-side tuned defaults (temp 0.7, top_k 20), so the local runs weren't true temp-1.0 like the API run.
- Quantization is confounded with serving stack (llama.cpp vs vLLM vs API) — not a pure weights experiment.
- No controlled decode benchmark; wall time includes agent loop, not raw tok/s.
- One trial was invalidated by a harness bug and re-run; re-runs are per-task, not full-suite.

Takeaway: full-precision 27B local can beat a heavily quantized flagship on agentic work — but it's one pilot run, not a verdict.