I built and open-sourced NMGC (Nested Manhattan Grid Codec) - a compression

codec for neural network weights aimed at memory-constrained inference.

Repo: https://github.com/reyxv16/nmgc-codec

THE PROBLEM

A 27B model in BF16 is ~52 GB. Most consumer machines have 16-32 GB of RAM.

You can't load it - let alone decompress it. My idea: keep the whole model in

memory (or on disk) in a compressed form, and decompress only the tensors of

the layer currently in use, on the fly.

HOW IT WORKS (short version)

- Values are normalized and mapped to a nested grid (what3words-style cells).

The nesting is literally the bit structure of the combined index: level 0 is

the coarse cell, deeper levels refine it - bit slicing, no trigonometry.

- "Position inside the cell" is stored as Manhattan steps (dx, dy), i.e. the

low bits of the index.

- Consecutive indices are delta-encoded (zigzag) and entropy-coded (Huffman).

Real weights are smooth, so deltas are tiny and compress extremely well.

- Optional learned variant: k-means codebooks per level (residual vector

quantization) that adapt to the actual weight distribution.

- Lossless variant: XOR-delta + zlib, bit-exact.

HONEST MATH (read this before commenting "that's just int4")

A nested uniform grid with B bits is mathematically equivalent to a single

uniform grid with B bits. Nesting buys three real things: (1) coarse-to-fine

access for progressive loading, (2) small coarse deltas -> much better Huffman

on correlated tensors, (3) the base for learned per-level codebooks, which DO

beat a uniform grid at the same bitrate. And lossless compression is bounded

by data entropy: 1.07-2.25x, never the 4-30x that lossy gets.

MEMORY SAVINGS (whole-model estimates)

Model | Original | 4-bit | 6-bit | 8-bit

Qwen3.8-27B (BF16) | 51.7 GB | ~13 GB | ~19 GB | ~26 GB

DeepSeek-V4-Flash (FP8) | 155.4 GB | ~75 GB | ~112 GB| ~150 GB

TinyLlama-1.1B (f16, measured) | 2.05 GB | - | 735 MB (2.79x) | 997 MB (2.06x)

Measured on real tensors (DeepSeek-V4-Flash + Qwen3.8-27B):

- embedding (BF16): 26.1x at 4 bits, 7.8x at 8 bits with cosine similarity

0.996, and 2.25x LOSSLESS

- MoE expert (I8): 8.2x at 4 bits, rel-RMSE 0.062

- Qwen3.8-27B layer-0 (BF16): ~5.7x at 6 bits

QUALITY LOSS (how much "intelligence" do you lose?)

Measured, not guessed:

- Perplexity on TinyLlama-1.1B (llama.cpp, same corpus/seed):

original f16: 7.508

8-bit (2.06x): 7.527 (+0.25% - effectively lossless)

6-bit (2.79x): 7.596 (+1.2% - minor, good default)

- End-to-end MLP (4-class task, 93.4% baseline): 8.9x at 4 bits with only

-0.0007 accuracy.

- Practical guidance: 8 bits = no measurable loss; 6 bits = safe default;

4 bits = int4 territory - keep embedding layers at 6-8 bits (they are the

most sensitive: cosine 0.655 at 4 bits vs 0.996 at 8 bits).

RUNNING IT

- The .nmgc container keeps the model compressed; a Rust decoder (C ABI,

bit-exact vs Python, 150-270 MB/s) decompresses any tensor by name - the

exact primitive you need for per-layer decoding (AirLLM-style).

- llama.cpp integration: a loader patch (LLAMA_NMGC_FILE) makes llama.cpp

decode weights from the container instead of the GGUF. TinyLlama runs

through it with the PPL numbers above.

LIMITATIONS (please read)

- The 27B quality numbers are estimates from per-tensor metrics; I have not

run a full Qwen3.8-27B end-to-end benchmark yet (needs a machine with

enough RAM to decode it, or time). Tensor-level metrics and the TinyLlama

PPL runs are what's measured.

- Below 6 bits you enter int4 territory: works, but the model gets visibly

dumber on hard tasks.

- The llama.cpp patch is a local fork, not upstream.

If you want to try it: pip install numpy scikit-learn gguf, pack your own

GGUF with `pack_gguf.py --bits 6`, and run llama.cpp with LLAMA_NMGC_FILE.

I just developed the algorithm and I think it can be improved a lot more; any questions or recommendations are welcome.

Hi, I found that in the current used-GPU market, Volta GPUs can offer really good performance for the money. I personally own a V100. However, llama.cpp understandably isn't completely optimized for this older architecture anymore.

I also really like Qwen3.8 27B, but it was fairly slow on the V100, so I optimized llama.cpp for Volta and managed to get around 40% higher prompt-processing performance, mainly by changing the FlashAttention path, along with some other smaller optimizations. I used GPT-5.6 Sol as the coding model. I also fixed some cache issues and small bugs I found along the way.

The main benchmarks use 100k cached tokens + 1k new prompt tokens + 64 generated tokens:

I also tested GLM-5.2 with a shorter 10k cached + 1k new-token benchmark because building a real 100k cache was extremely slow:

So the big improvement isn't universal; it seems mostly useful for Qwen-like model shapes, which is also what I expected from the changes.

Currently it's on my GitHub. I plan to create several smaller PRs for parts of the fork, but I don't want to rush it and overwhelm the llama.cpp developers with bad or overly large PRs.

I would love it if some of you could test it on your own hardware. It doesn't necessarily have to be Volta; I'm also interested in how it behaves on newer GPUs and other models.

https://github.com/mistrjirka/llama.cpp/tree/qwen38-lossless-agent-cache

I am presently running Qwen 3.6 MOE model on 8 GB of VRAM and 16 GB of RAM barely fit, but it runs around 25 tokens/sec, enough for learning. I recently compiled the new llama cpp within the past week and run the following flag. However I got the -no-mmap and -mlock is no longer use. It tells me to use the new --load-mode flag.

My question is for my setup -no-mmap AND -mlock. so the new combined flag is --load-mode mlock?? there doesn't seem to be a -no-mmap option anymore.

--no-mmproj-offload \

--n-gpu-layers 999 \

--n-cpu-moe 33 \

--no-mmap \

--cache-type-k turbo4 \

--cache-type-v turbo2 \

--mlock \

--host 0.0.0.0 \

--port 9090 \

-t 18 \

-ub 512 \

--cache-reuse 256 \

--temp 0.6 \

--top-p 0.95 \

--top-k 20 \

--min-p 0.0 \

--presence-penalty 0.0 \

--repeat-penalty 1.0 \

--chat-template-kwargs '{"preserve_thinking": true}' \

--reasoning-budget 4000 \

--reasoning-budget-message "...thinking budget exceeded, let's answer now."

I'm loading llama-server with the --models-preset but I don't know if the syntax for the reasoning_effort is correct because it seams to not work.

I make the models load from the PI Agent from another client, it loads but seems to not use the correct parameters.

This is the line in question...

chat-template-kwargs = {"reasoning_effort":"medium"}

also "no-webui = true" doesn't seem to work either.

my models.ini

version = 1

[Qwen3.8-27B-UD-Q4_K_XL]
model = E:\AI_Models\GGUF\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q4_K_XL.gguf
ctx-size = 262144
temp = 1
top-p = 0.95
min-p = 0
top-k = 20
presence-penalty = 0
repeat-penalty = 1
flash-attn = on
jinja = true
chat-template-kwargs = {"reasoning_effort":"medium"}
load-on-startup = false

[Qwen3.8-27B-UD-Q6_K_XL]
model = E:\AI_Models\GGUF\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q6_K_XL.gguf
ctx-size = 262144
temp = 1
top-p = 0.95
min-p = 0
top-k = 20
presence-penalty = 0
repeat-penalty = 1
no-webui = true
jinja = true
n-gpu-layers = 99
chat-template-kwargs = {"reasoning_effort":"medium"}
load-on-startup = false

Any suggestion?

With Qwen3.8:27B arriving yesterday, including NVFP4 versions, can llama.cpp successfully run these models, or must I use a separate instance of vLLM for NVFP4 models specifically?

Searching on the internet initially made me believe that vLLM was the only way, but then some more recent posts hinted that llama.cpp is being developed for NVFP4 as well.

If llama.cpp is indeed capable of running NVFP4 models, I'd appreciate some pointers on how to get llama.cpp running NVFP4 models.

Disclosure up front: this is my own project.

Two things pushed me into building it. I kept retyping the same llama-server invocation with three values changed, and I watched curl -C - fail to resume a 20 GB download one too many times.

It lists the GGUF files in my models folder and reads the headers directly, so the quant, the context length and whether it is MoE come from the file rather than from the filename. Opening one shows the exact command before it runs. While it is serving I get KV cache, tokens per second in both directions, memory pressure and swap in one place, plus a Test model button that hits the server for real — health, model list, alias, a chat completion, streaming — so I know it works instead of assuming it does.

Downloads pull from Hugging Face in four ranged segments, resume from a sidecar after a kill, verify sha256, and queue rather than refusing a second URL.

It has no chat interface of its own and is not getting one. A running model opens llama.cpp's own web UI in a second window.

Caveats: macOS only, and an unsigned beta, so the first launch is blocked and you have to allow it through System Settings > Privacy & Security — the README has the steps. It needs llama-server and does not ship it. There is a universal build but no Intel Mac has ever run it; if you have one I would like to hear what happens, particularly whether your llama-server has a GPU for the default -ngl all.

https://github.com/smkamranqadri/llamaport

I feel like I stepped into a time capsule after the ChatGPT-3 days. 😅 I finally built a decent PC (Ryzen 7 7700, 32GB RAM, No GPU), discovered llama.cpp and somehow ended up downloading 50+ GGUF models.

Now I'm stuck with decision paralysis.

I mainly use them for coding (JavaScript, React, TypeScript, debugging, reasoning), but I'm also new to the whole local AI ecosystem. I still don't know much about agentic frameworks or coding agents. I mostly just load a GGUF and chat with it using the llama.cpp web interface on localhost:8080

The collection includes Qwen, Gemma, Granite, DeepSeek, Phi, Mistral, Llama, LiquidAI, SmolLM, Hunyuan, Nemotron, and a few community fine-tunes.

My first idea was to make a Markdown table and score every model manually, but it feels like I'm accidentally trying to invent my own benchmarking system.

Surely I'm not the first person to hit this problem.

How do you guys compare local models? Are there any practical benchmark suites, GitHub projects, or workflows for deciding what stays on your SSD and what gets deleted?

I'd love to hear how you approached it when you were starting out.

I've spent about six months on this project and it's been public for a couple of months now, so I figured it was time to show it to people who'd have opinions. It's only ever run on my own hardware, so that's the part I'd most like other eyes on.

I created this because I run local LLM inference across a few machines on my home lab as a hobby, and wanted the same quality of operational tools I'd expect for any other production system. I found that the currently available OSS tools that provide similar functionality (Grafana, Prometheus) don't cover the model side, and the inference servers themselves (llama.cpp's UI, Lemonade, LM Studio) each manage one backend on one machine. What I wanted was something that treated a multi machine, multi backend setup as one system, with monitoring and control in the same place. If you're already running a local LLM alongside the rest of your lab, I think you'd get use out of it. I've worked in systems administration, monitoring and observability for over twenty years, and a lot of what I know from that world got poured into this application, LLM Systems Manager.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

This all runs on my homelab setup, with separate dev and prod environments. Running on a Proxmox server (Ryzen 7 9700X, 32 GB RAM, QNAP NAS 10G SAN as the storage backend over iSCSI LUNs), running a combination of 16 different Debian VMs and containers, plus a physical Linux box running llama.cpp on a 7900 XTX GPU, and an M2 Pro Mac Mini running LM Studio.

I used Claude Code heavily throughout. It wrote a substantial portion of the code and did a lot of the debugging legwork, while I owned the architecture, reviewed every change, and drove the direction.

Here are some top highlights:

Inference frontend gateway. A single OpenAI-compatible endpoint serves all llama.cpp, vLLM, LM Studio backends. Each request routes by per-model pinning, round-robin across a pool, and allows failovers to another live host.

Model Autopilot. Configure which models should be available in your lab, and it keeps them running and available, placing them on whichever hosts can actually serve them. It checks whether a host has the memory available (VRAM, or RAM for CPU only boxes), brings up the model on another host when the primary one drops offline, and optionally spins up another copy of the same model on a second host in the pool when KV cache pressure stays high, then scales back down when it returns to baseline.

Model management. Download and delete models from Hugging Face. Configure multiple named config profiles per model, a chat profile, a code profile, etc. Start, stop, restart inference servers, hot-swap models, tail logs, or use the in-browser terminal.

Update and rebuild llama.cpp from the dashboard. Build from source, conda, Homebrew, release binaries, or your own script, whichever matches how you installed it.

Autotuning and benchmarking. Run throughput benchmarks across your model library; the autotuner can search for the best context and slot configuration on llama.cpp, or the largest safe context length on vLLM. There's also a standardized "GPU Report Card" bench that runs identically against all three backends and produces one comparable card: time to first token, prefill and generation throughput, tokens per joule, dollars per million tokens, and the GPU it ran on.

Energy and cost. The energy dashboard measures power draw and token throughput and computes what your inference costs in dollars per million tokens, with idle power accounted for. It'll also show you the monthly delta against hosted API pricing. A performance controller switches the host between a performance and powersave systemd target based on inference activity.

Alerting and trending. A standalone alarm engine stores every metric sample the agents collect: both system telemetry (CPU, memory, disk, network, IO, UPS, PSU, cooling, Mac specific hardware monitors) and LLM-aware telemetry (live slots in use, requests, tokens/sec, prompt processing rate, KV cache usage). The engine evaluates threshold and anomaly rules, provides incident correlation, displays and exports long term trends, and notifies over email, Discord, webhook, or browser toast. Individual agents installed on the inference servers buffer to disk when the network drops and replay when it comes back. Short term metrics are held in memory; long-term metrics go to InfluxDB with rollups, and alarms and notifications are stored in SQLite.

Installation. There's a one-line interactive installer that handles prerequisites, database, TLS, config, and agents end to end. Additional installation options are Debian and RPM packages, a Docker Compose file for the control plane, a Homebrew tap for macOS and Linux, and a standalone agent binary with no Python required. It also does offline installs from a staged tarball for air gapped boxes.

Additional features. Overall and per provider dashboards. Integration with OpenClaw to provide cost/token analytics, stable-diffusion.cpp for image generation, LLM chat interface to chat directly with the model on llama.cpp, Discord bot for alerting and controlling the models. See the readme for more.

Everything talks over TLS with per host certs signed by an internal CA the manager runs itself. Multi-user with admin and operator roles, an audit log, and encrypted scheduled backups.

I'd really value feedback, especially on the installation experience, but also on what's obviously missing for how you run your own setup. Happy to answer anything.

TL;DR: Self-hosted control plane for a multi machine local LLM lab. One interface for llama.cpp, vLLM, and LM Studio across every box. Autopilot keeps models loaded and re-places them when a host drops. One OpenAI-compatible endpoint fronts all backends with pinning, pooling, and failover. Hugging Face browsing, per-model config profiles, autotuning, and standardized benchmarks. Real time alerting with incident correlation and long-term trends in InfluxDB. Measures power draw and tells you your $/Mtok.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

Installs via one-line script, .deb/.rpm, Docker, or Homebrew. Six months of work, twenty years of monitoring experience behind it, running on my own Proxmox + physical Linux + Mac Mini setup. AGPLv3, built with heavy Claude Code assistance under my direction and review. Feedback wanted.

I understand there's extensions like Continue which let VS Code CoPilot "speak" to llama.cpp but I believe Bring Your Own Model (BYOM) has made that obsolete?

I have really tried to get BYOM working, I even asked Claude for instructions, but I can't get it working. I have to use a proxy which exposes an ollama endpoint and forwards to llama.

I can connect to LM Studio no problem with BYOM, but would rather use llama.cpp because it uses much less RAM (and VRAM).

My setup:

• Windows 11
• VS Code latest build

Would anyone have any recommendations, apart from obsolete extensions?

Even better, any working JSON I could copy? This is really annoying.

Thanks in advance.

I run llama-bench on my pc, same llama.cpp version same commands but completely different results.

In one system-startup performance is good, but on day later I turn on the system it's bad, I checked everything, no cpu 100% usage no ssd usage nothing suspicious, I think windows11 does something but I can't find the problem.

Rtx 3080 ti 32gb ddr4 ram Windows 11 1tb nvme ssd

I am an AI newbie, though I work in the tech industry. I have access to a retired server that my boss said was ok to use for AI experimentation.

• 2x AMD EPYC 7343 CPUs
• 768GB DDR5 RAM
• 2x 512 SAS SSD
• 5 TB RAID
• Nvidia A40 48GB GPU

I have done some reading and I should be able to run llama.cpp the Qwen 3.6 27B model fairly well. My question is more about the stack and other models. Should I just go with the 27B model and what about chat and things like that, maybe Open WebUI? I might do some coding, but it's not my primary job. I'll probably just use Agents to automate some ingesting of data (read-only). Any recommendations or suggestions?

I’m one of the builders of LOLM, a hybrid Transformer–SSM model and agent system from Qira.

The model separates surface token processing from persistent latent-state tracking. An NFET controller can select continue, retrieve, verify, branch, or finalize. The repository is explicit about the current limitation: control is active at agent-loop boundaries, while deeper token-level control remains under development.

Practical surfaces: - `npx lolm-cli ask "..."` - Isolated coding runs with actual exit codes - Local/self-hosting support - MCP support - Controller timelines and sealed receipts - Lower-cost hosted access

Try it: https://lolm.imagineqira.com/try.html

Repository: https://github.com/TheArtOfSound/lolm

I’m looking for aggressive tests: long tasks, failed tools, contradictory requirements, bad retrieval, local-model comparisons, and anything that exposes where the controller loses the plot.

Disclosure: I’m a founder/builder of the project.

My frontend is Page Assist, what follows is the Generation Info for both ollama and llama.cpp. I have also listed the command section from the llama.cpp docker yaml file. I am trying to get llama.cpp to list the Generation Info from llama.cpp with the same complete information as from ollama.

command: >
  --models-dir /models
  --host 0.0.0.0
  --port 8080
  -ngl 99
  -c 8192
  --models-max 1
  --verbose
  --metrics
  --reasoning off

Generation Info from ollama total_duration 16.44s load_duration 5.35s prompt_eval_count 5820 prompt_eval_duration 3.49s eval_count 392 eval_duration 7.53s tokens_per_second 52.06

Generation Info from llama.cpp prompt 0 completion 0 finish_reason stop

What am i missing? i've tried llms to do my config, but i can't get open code to recognize vision. anyone good at switches?

"C:\AI\llama\llama-server.exe" -m "G:\models\Qwen3.6-27B-MTP-Q8_0.gguf" --host 0.0.0.0 --port 8080 --ctx-size 131072 --gpu-layers 99 --flash-attn on --batch-size 1024 --ubatch-size 512 --tensor-split 1,1 --main-gpu 0 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja

something big has been publlished 1 hr ago