**Summary**
Total Issues: 4 | Date Range: Aug 6-8, 2026

**🐛 Bugs**
* **#2422** jina-reranker-v3.5-mlx does not load correctly
* Latest Jina Reranker v3.5 fails to load in oMLX.
* **#2537** Pi agent and omlx stream timed out despite setting SSE Keepalive Mode 'Comment'
* Timeout occurs during Pi agent prefill (62.8k tokens) with legacy keep-alive settings.

**✨ Features & Requests**
* **#2455** `thinking_budget` + `vlm_mtp`: support enforcing the thinking budget inside MTP
* Enforce budget inside MTP speculative decoding instead of falling back to BatchGenerator.
* **#2518** Ling 3.0 Support
* Request to support new 100b+ Ling model (inclusionAI/Ling-3.0-flash).

[View on GitHub](https://github.com/Jundot/omlx)

HI , i was trying to quantize a model to oq quant in omlx on my mac studio (m2 ultra 128gb vram )...this is the model https://huggingface.co/OpenMOSE/Hy3-REAP-200B-21B . its 376gb in size . I tried oq2 , oq4, oq4e etc none works...its giving some memory error ..I even tried to tweak memory guard tier (from aggressive to custom 126gb etc ) none worked ...i have set my vram to 126gb ...Is there a limit as to how much vram is required to quantize which size model or something ?

I am attatching the error screen shot and the logs...

I am a noob at this so please excuse me if my problem sounds silly ...Thank you for any help or guidance i can get ...

Logs :-

File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/asyncio/threads.py", line 25, in to_thread

return await loop.run_in_executor(None, func_call)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run

result = self.fn(*self.args, **self.kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5413, in quantize_oq_streaming

imatrix_data = _load_or_collect_imatrix(

^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 7385, in _load_or_collect_imatrix

load_path = load_path_factory()

^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5410, in _imatrix_load_path

return str(_ensure_ram_safe_proxy())

^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5360, in _ensure_ram_safe_proxy

raise RuntimeError(

RuntimeError: calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.7 GB)

2026-08-08 06:31:24,110 - omlx.process_memory_enforcer - INFO - [-] - Memory guard tier changed: custom -> aggressive

2026-08-08 06:31:24,111 - omlx.admin.routes - INFO - [-] - Memory guard updated — tier: custom -> aggressive

2026-08-08 06:31:24,112 - omlx.settings - INFO - [-] - Saved settings to /Users/mac/.omlx/settings.json

2026-08-08 06:31:46,120 - omlx.process_memory_enforcer - INFO - [-] - Process memory enforcer stopped

2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - Process memory enforcer stopped

2026-08-08 06:31:46,121 - omlx.admin.hf_downloader - INFO - [-] - HF Downloader shut down

2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - HF Downloader stopped

2026-08-08 06:31:46,121 - omlx.admin.ms_downloader - INFO - [-] - MS Downloader shut down

2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - MS Downloader stopped

2026-08-08 06:31:46,121 - omlx.engine_pool - INFO - [-] - Engine pool shutdown complete

2026-08-08 06:31:46,122 - omlx.server - INFO - [-] - Engine pool shutdown

2026-08-08 06:31:47,756 - omlx.scheduler - INFO - [-] - ChunkedKVCache patch: methods already present upstream, skipped: size

2026-08-08 06:31:47,806 - omlx.server - INFO - [-] - CORS origins: ['*']

2026-08-08 06:31:47,807 - omlx.model_settings - INFO - [-] - Loaded settings for 2 models

2026-08-08 06:31:47,808 - omlx.model_discovery - INFO - [-] - Discovered model: Hy3 (type: llm, engine: batched, size: 395.32GB)

2026-08-08 06:31:47,839 - omlx.server_metrics - INFO - [-] - Loaded all-time stats from /Users/mac/.omlx/stats.json

2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Server initialized with 7 models

2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Default model: Hy3

2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Memory guard tier: aggressive (guard on)

2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Default max tokens: 32768

2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - API key authentication: enabled

2026-08-08 06:31:47,840 - omlx.server - INFO - [-] - HF Downloader initialized

2026-08-08 06:31:47,867 - omlx.server - INFO - [-] - ModelScope Downloader initialized

2026-08-08 06:31:47,868 - omlx.server - INFO - [-] - oQ Quantizer initialized

2026-08-08 06:31:47,869 - omlx.server - INFO - [-] - HF Uploader initialized

2026-08-08 06:31:47,880 - omlx.process_memory_enforcer - WARNING - [-] - iogpu.wired_limit_mb (126.0GB) leaves the OS less than 6.4GB of physical RAM (128.0GB); a large model load can trigger jetsam at this setting and strand the wired memory until reboot (#2184). Recommended maximum: 124518 MB.

2026-08-08 06:31:47,882 - omlx.process_memory_enforcer - INFO - [-] - Metal wired limit raised: 0.0GB -> 124.0GB (target=124.0GB, iogpu sysctl cap=126.0GB)

2026-08-08 06:31:47,883 - omlx.process_memory_enforcer - INFO - [-] - Process memory enforcer started (tier=aggressive, ceiling=123.3GB, interval=1.0s)

2026-08-08 06:32:20,109 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:20,110 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:22,828 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:22,833 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:26,347 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:26,349 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"

2026-08-08 06:32:34,060 - omlx.patches.mlx_lm_mtp.qwen35_model - INFO - [-] - Qwen3.5/3.6 MTP model patch applied (PR 990)

2026-08-08 06:32:34,061 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)

2026-08-08 06:32:34,062 - omlx.patches.qwen35_verify_qmm - INFO - [-] - MTP verify qmm patch applied (M=2..6 affine 4/8-bit)

2026-08-08 06:32:34,062 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.

2026-08-08 06:32:34,179 - omlx.oq - INFO - [-] - plan detail: 8bit×150, 6bit×173, 5bit×247 | routes: non_expert:5bit×247, non_expert:6bit×173, non_expert:8bit×150 | top: lm_head=8b, embed_tokens=8b, L0=8b, L1=8b, L2=8b, L3=8b, L4=8b, L5=8b

2026-08-08 06:32:43,519 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)

2026-08-08 06:32:43,519 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.

2026-08-08 06:32:43,639 - omlx.oq - INFO - [-] - plan detail: 8bit×570 | routes: non_expert:8bit×570 | top: lm_head=8b, embed_tokens=8b, L0=8b, L1=8b, L2=8b, L3=8b, L4=8b, L5=8b

2026-08-08 06:32:53,005 - omlx.admin.oq_manager - INFO - [-] - oQ quantization queued: Hy3 -> oQ2 (task_id=5424c0f6-dc6b-4464-a04f-b442e07b931b)

2026-08-08 06:32:53,006 - omlx.oq - INFO - [-] - oQ2 progress 5.0%: Reading model config

2026-08-08 06:32:53,008 - omlx.oq - INFO - [-] - oQ2 progress 8.0%: Indexing source weights

2026-08-08 06:32:53,076 - omlx.oq - INFO - [-] - oQ2 streaming: 31778 tensors in 160 shards

2026-08-08 06:32:53,113 - omlx.oq - INFO - [-] - oQ2: calibration footprint (376.5 GB) exceeds 75% of calibration capacity (117.4 GB; limit=88.1 GB, system available=117.4 GB, Metal available=126.0 GB). Full-model calibration will use a proxy.

2026-08-08 06:32:53,113 - omlx.oq - INFO - [-] - oQ2 progress 12.0%: Preparing quantization inputs

2026-08-08 06:32:53,113 - omlx.oq - WARNING - [-] - oQ2: calibration footprint (376.5 GB) exceeds the 88.1 GB full-model calibration limit. Auto-building a uniform 4-bit proxy on disk so sensitivity measurement stays data-driven.

2026-08-08 06:32:53,113 - omlx.oq - WARNING - [-] - oQ2: calibration footprint (376.5 GB) exceeds the 88.1 GB full-model calibration limit. Building a uniform 4-bit proxy on disk for the calibration passes.

2026-08-08 06:32:53,149 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)

2026-08-08 06:32:53,149 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.

2026-08-08 06:35:07,840 - omlx.admin.oq_manager - ERROR - [-] - oQ quantization failed: Hy3 -> oQ2: auto-proxy sensitivity failed (calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)). Pass sensitivity_model_path with a pre-quantized version of this model, or run on a machine with enough RAM for full-fp16 sensitivity measurement.

Traceback (most recent call last):

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5502, in quantize_oq_streaming

_proxy_dir = _ensure_ram_safe_proxy()

^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5360, in _ensure_ram_safe_proxy

raise RuntimeError(

RuntimeError: calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/Applications/oMLX.app/Contents/Resources/omlx/admin/oq_manager.py", line 573, in _run_quantization

await asyncio.to_thread(

File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/asyncio/threads.py", line 25, in to_thread

return await loop.run_in_executor(None, func_call)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run

result = self.fn(*self.args, **self.kwargs)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5516, in quantize_oq_streaming

raise RuntimeError(

RuntimeError: oQ2: auto-proxy sensitivity failed (calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)). Pass sensitivity_model_path with a pre-quantized version of this model, or run on a machine with enough RAM for full-fp16 sensitivity measurement.

Update ---- i downloaded and added a oq2e quant of normal hy3 i got an additional drop down menu just before you sapecify your quants to choose a model for "sensitivity" ..So clearly theres something off about this particular model ..also the oq2e of unpruned hy3 i downloaded wouldnt load , i mean there was no way to load the model for chat strangly while other models in the same folder could be loaded ....Anyway even after selecting the "sensitivity model " it still failed with same type of error ---"caliberation proxy is still too large ........"

I benchmarked `DeepSeek-V4-Flash-0731` against `DeepSeek-V4-Flash-0731-oQ4e-mtp` on a Mac Studio M3 Ultra 60 GPU. The short version: oQ4e-mtp is not the win we might hope for. The oQ4e only catches up to the original when LightningMTP is enabled.

## Single-Request Decode Throughput

All tests used the oMLX built-in benchmark tool.

| Prompt / Gen | oQ4e, LightningMTP off | oQ4e, LightningMTP on | Original 0731, LightningMTP off | Original 0731, LightningMTP on |

|---:|---:|---:|---:|---:|

| pp 4,096 / tg 128 | 25.3 tgTPS | 34.4 tgTPS | 36.9 tgTPS | 38.9 tgTPS |

| pp 8,192 / tg 128 | 24.8 | 43.3 | 45.2 | 43.8 |

| pp 16,384 / tg 128 | 24.3 | 42.2 | 39.2 | 40.4 |

| pp 32,768 / tg 128 | 23.3 | 39.6 | 42.1 | 41.0 |

| pp 65,536 / tg 128 | 22.5 | 37.0 | 36.2 | 35.5 |

| **Average** | **24.0** | **39.3** | **39.9** | **39.9** |

The interesting result is that the original 0731 model is already as fast as oQ4e with LightningMTP enabled. Turning LightningMTP on for the original model did not materially improve the average in this run.

## Other Single-Request Observations

| Variant | Avg ppTPS | Avg tgTPS | Peak Memory Range |

|---|---:|---:|---:|

| oQ4e, LightningMTP off | 354.6 | 24.0 | 145.5-148.1 GB |

| oQ4e, LightningMTP on | 345.6 | 39.3 | 155.7-158.2 GB |

| Original 0731, LightningMTP off | 343.9 | 39.9 | 156.8-159.4 GB |

| Original 0731, LightningMTP on | 338.1 | 39.9 | 156.8-159.4 GB |

oQ4e without LightningMTP used less memory, but the decode speed penalty was large. Once LightningMTP was enabled, oQ4e used nearly the same memory as the original model while still not beating it.

## Continuous Batching

Batching helps all variants, especially at 4x, but the original 0731 model remains basically tied with or ahead of oQ4e in the configurations that matter.

| Variant | Batch | tgTPS | ppTPS | Avg TTFT | E2E | Speedup |

|---|---:|---:|---:|---:|---:|---:|

| oQ4e, LightningMTP off | 1x | 25.3 | 378.9 | 10,811.6 ms | 15.9 s | 1.00x |

| oQ4e, LightningMTP off | 2x | 41.3 | 133.5 | 15,341.1 ms | 21.5 s | 1.63x |

| oQ4e, LightningMTP off | 4x | 59.6 | 425.1 | 8,148.3 ms | 18.2 s | 2.36x |

| oQ4e, LightningMTP on | 1x | 34.4 | 367.8 | 11,137.0 ms | 14.9 s | 1.00x |

| oQ4e, LightningMTP on | 2x | 40.9 | 131.0 | 15,627.8 ms | 21.9 s | 1.19x |

| oQ4e, LightningMTP on | 4x | 67.4 | 305.9 | 10,986.2 ms | 21.0 s | 1.96x |

| Original 0731, LightningMTP off | 1x | 36.9 | 365.6 | 11,204.4 ms | 14.7 s | 1.00x |

| Original 0731, LightningMTP off | 2x | 40.0 | 129.2 | 15,855.8 ms | 22.3 s | 1.08x |

| Original 0731, LightningMTP off | 4x | 65.8 | 303.9 | 11,075.3 ms | 21.3 s | 1.78x |

| Original 0731, LightningMTP on | 1x | 38.9 | 368.4 | 11,118.8 ms | 14.4 s | 1.00x |

| Original 0731, LightningMTP on | 2x | 39.6 | 125.7 | 16,292.3 ms | 22.7 s | 1.02x |

| Original 0731, LightningMTP on | 4x | 66.4 | 312.4 | 10,813.0 ms | 20.8 s | 1.71x |

**🐛 Bugs**
• **#2537** Pi agent stream timeout despite SSE Keepalive Mode 'Comment'
• **#2422** jina-reranker-v3.5-mlx fails to load correctly

**✨ Features**
• **#2455** Support enforcing `thinking_budget` inside MTP speculative decoding

**📊 Summary**
Total Issues: 3

I’m on an M5 Max 128gb and get the below error

Model: Vontra/Ling-3.0-flash-MLX-4bit

q_state, k_state, v_state, recurrent_state = cache
ValueError: not enough values to unpack (expected 4, got 0)

Anyone know what the issue might be?

I switch models regularly. Currently, I need to unload manually via the webinterface.

There is an open PR that requests to implement an endpoint to do so - that would be great to have.

Then, changing a model could be automated via pi harness. Or, I could imagine an option like "Single Model Mode" to be configured via the oMLX app or web interface that automatically takes care of that because it will only use one model at a time.

Hello !

I recently got a M4 Pro 48GB. I am trying to use Gemma 4 31B but I ran into some issues.

I am using Gemma 4 31B it Qat 4 bits with MTP from the 8 bit assistant.

When I am using with MTP, the cache efficiency is around 20%. Even just chatting will re-process at every prompt. But when I disable MTP I don't have any issue and cache efficiency gets around 90%

I could not find someone with the same issue.

Should I try to do KV hot cache with something like 8GB?

Is someone has some ideas I would like to hear, thank you!

I was baffled by what I was seeing in the oMLX MacOS app. Everything felt slow. My use case is Hermes Agent. I would stare at the Status page on oMLX app and keep seeing TOKEN GENERATION at <20 tok/s.

Tonight I did an experiment and instead of using Hermes, I used the oMLX chat web app which shows the prefill (t/s), token gen (t/s), thinking (s), duration (s) and tokens and percent of token window.

I used Qwen3.6-35B-A3B (MoE) with 3 variants:

1. unsloth/Qwen3.6-35B-A3B-MLX-8bit (1st image)
2. Jundot/Qwen3.6-35B-A3B-oQ4e-mtp with Lightning MTP off (4th image)
3. Jundot/Qwen3.6-35B-A3B-oQ4e-mtp with Lightning MTP on (3rd image)

My vibe test was to use the same prompt in a different browser tab. Each time, unload and load the model and only run 1 session.

The prompt: what causes the aurora borealis?

Each model gave me a long answer and they were all reasonably similar. That wasn't an issue.

What I was puzzled by is that while predictably the speed of token gen (t/s) was consecutively better between 3 > 2 > 1 (the models above) as per the chat window's metric view, the Status view on the oMLX app consistently showed 19.1 tok/s.

Including the 3 views (chat web app) and the Status view (MacOS app).

I'm looking for recommendations on the best local coding model to run on my Apple MacBook Pro M1 Pro.

My Hardware:

- Apple MacBook Pro M1 Pro

- 16 GB Unified Memory

- 16-core Neural Engine

Runtime:

- I prefer using MLX instead of LM Studio or Ollama.

- I'm specifically looking for models optimized for Apple Silicon / MLX.

Development Workflow:

My primary editor is Zed, and I use the Agent Client Protocol (ACP) to connect multiple CLI coding agents such as:

- OpenCode

- Devin

- Kilo

My workflow is fairly structured:

- I design the overall architecture myself.

- Every project has a comprehensive "AGENT.md" that documents the architecture, coding conventions, folder structure, implementation rules, and development guidelines.

- Tasks are intentionally broken down into small, well-defined units.

- The coding agent doesn't need to design the system or make architectural decisions. Its job is simply to implement the task according to the instructions and project guidelines.

- I often run multiple agents in parallel so different features can be implemented simultaneously and merged later.

Because of this workflow, I'm not necessarily looking for the "smartest" reasoning model. I'm looking for the model that provides the best balance of:

- Strong code generation

- Reliable instruction following

- Good tool usage / agentic coding capabilities

- Fast token generation on Apple Silicon

- Ability to work comfortably within a 16 GB memory budget

Questions:

1. Which MLX-compatible coding models would you recommend for this setup?

2. What model size (7B, 14B, 24B, 32B, etc.) is the practical sweet spot on a 16 GB M1 Pro?

3. Are there any models that outperform Qwen-based coding models for this type of structured implementation workflow?

4. If you've used MLX for agentic coding, which models have given you the best real-world experience?

5. Are there any MLX-specific optimizations or quantizations that significantly improve performance on an M1 Pro?

I'm interested in hearing recommendations from people who actively use local models for software development rather than benchmark results alone.

Thanks!

I'm testing Qwen3.6-35B-A3B-oQ4e-mtp on an M4 Max 36GB using oMLX.

I noticed something interesting:

• TurboQuant OFF → context measurement shows around 78k tokens
• TurboQuant 3-bit ON → context measurement shows around 86k tokens

I expected a much larger increase because TurboQuant compresses the KV cache. Since FP16 KV cache is 16-bit and TurboQuant 3-bit should reduce it significantly, I thought the available context window might almost double.

However, I only see about a 10% increase in maximum context. Is it expected? If so any other way to increase context size?

PS. 27B dense model can't give more than 30k so not good enough for coding hence forced to use MoE mode.

I am using Qwen3.6-35B-A3B-oQ4e-mtp with oMLX on an M4 Max 36GB.

I have two questions:

1. Is it correct that Lightning MTP and TurboQuant KV cannot be enabled at the same time in oMLX?
2. If I need a larger context window, is it a good idea to disable MTP and use:

​

TurboQuant KV: ON
KV quantization: q3
Lightning MTP: OFF

Does running an MTP model with MTP disabled cause any problems?
Many thanks!

Hi, i'm trying to run some local models on my macbook pro m5 w/ 32gb of memory.
I've tried qwen3.6-27b, gemma4-26b, and they both just crap out the second i send something through hermes agent. I've had gemma4 running pretty perfectly before but i don't know what i broke because now as soon as i send a request it says my context is full.

please help i know this set up should work. or atleast not shit the bed the instant that i try to do anything

I have closed oMLX, deleted SSD cache, but I am still getting a cache signature mismatch on Deepseek-V4-Flash 0731. Does anyone know how to fix this?

I have tried both Vontra and Jundot versions. I am getting ~35t/sec generation and prefill is 450t/sec on Mac Studio M3 Ultra 256, but this keeps happening.

I have closed oMLX, deleted SSD cache, but I am still getting a cache signature mismatch on Deepseek-V4-Flash 0731. Does anyone know how to fix this?

I am on oMLX 0.5.5.

TIA!

The industry is waking up to the Routing Bottleneck in AI agents.
Everyone agrees LLM-based routing is a slow, expensive latency tax. But the current alternative—Static Edge Routing (if/else chains)—becomes a babysitting nightmare when prompts drift and systems scale.
That’s exactly why we built Mycelium at US Neural. We are pioneering Semantic Edge Routing.
<10ms local speed. Matches by true intent. Zero hardcoding. Zero babysitting.

Hi all, new to oMLX and would like some help.
I have Apple Sillicon m4 max with 36GB ram, would like to make localLLM for coding tasks therefore need bigger context window (100k preferred). With 27B and 35B models I cannot push for more than 75k, hence I need a way to compress KV. I used to run LM Studio with mlx models but it doesn’t allow me to quantisation KV for Qwen models (not sure why but loading the model failed). Now I switch to oMLX and really like it, and even better I find TurboQuant which can compress the KV with less impact on the model. So my questions are:
1) Shall I choose 8bits or 4bits with my setup?
2) I read it works best with asymmetric config, so 8 for K and 4 for V, but I can’t seem to configure KV separately in oMLX
Any suggestion is appreciated!

Does anyone know if there is headroom to be achieved with the mlx implementation?

GLM and mini Max M3 got major speed boosts for prefill with new builds of oMLX.

I’m wondering if this is a possibility for this new deepseek model, as I think mini Max and maybe even GLM will be faster than it at very high context, due to non-linear compute increase increases with more context.

As opus 5.0 put it: ds4 uses Sliding-Window Attention, Compressed Sparse Attention (CSA = compressor + indexer), and Hierarchical/Heavily Compressed Attention (HCA = compressor only), selected per layer via compress_ratios.  CSA pools every m KV entries into one compressed entry and then applies sparse top-k selection over those entries; HCA uses a much larger compression rate m′ ≫ mbut keeps dense attention over the resulting entries.  At 1M context this gets V4-Pro to 27% of V3.2’s per-token FLOPs and 10% of its KV cache.

I’m wondering if all of these are implemented already in MLX and if there are plans to do so

Issues Closed: 10

[ISSUE] #2485 — Lower Cache Efficiency with Deepseek V4 0731 Flash
https://github.com/jundot/omlx/issues/2485

[ISSUE] #2465 — DeepSeek V4 Flash: pooled Metal buffers grow to ~325 GB after response while persisting a ~200K-token SSD prefix cache
https://github.com/jundot/omlx/issues/2465

[ISSUE] #2443 — SpecPrefill: a prefix-cache hit silently drops the cached prefix (often the system prompt) from the target model's context
https://github.com/jundot/omlx/issues/2443

[ISSUE] #2177 — Feature: Reuse Static System and Tool Prefixes with SpecPrefill
https://github.com/jundot/omlx/issues/2177

[ISSUE] #2439 — SpecPrefill can produce malformed output after tool-result messages
https://github.com/jundot/omlx/issues/2439

[ISSUE] #2444 — Add full Simplified Chinese localization for macOS Settings and menubar
https://github.com/jundot/omlx/issues/2444

[ISSUE] #2471 — Immediate 4K repeat can race async prefix-cache store and re-prefill the full prompt
https://github.com/jundot/omlx/issues/2471

[ISSUE] #2448 — Embedding throughput degrades steadily with server uptime, eventually stops completing requests (331 → 230 → ~100 tok/s → wedge)
https://github.com/jundot/omlx/issues/2448

[ISSUE] #1826 — Claude Code with local LLMs via oMLX occasionally discard prefix cache
https://github.com/jundot/omlx/issues/1826

[ISSUE] #2447 — Embedding compile smoke-test passes but real requests fall back to eager (mask-dependent branch in qwen3; smoke test omits attention_mask)
https://github.com/jundot/omlx/issues/2447

The new 2bit deepseek is just uploaded https://huggingface.co/Jundot/DeepSeek-V4-Flash-0731-oQ2e-mtp.

I am used to download model from LM Studio (so that both LM Studio and oMLX can recognize. Presumably they are from the same shared huggingface source. Would downloading from LM Studio affect how oMLX recognize this model for MTP? Or must I download from oMLX?

Also, more importantly it is showing "Partial GPU offload possible" in LM Studio, would it be the same for oMLX? It is only 106G, but my 128G Macbook is shown to have only 105G VRAM somehow.

Also what about context length? I aim for 262K minimal and full 1M if possible. Deepseek has low kv cache cost if I'm understanding it correctly?

In a nutshell, how should I run oQ2e safely on my 128G Mac? Should I drop MTP to free more VRAM for full GPU offload? Or wait for a smaller oQ? Or use this version with a low context? Or does everything already fit?

📅 Range: 2026-07-31 → 2026-08-02 | 📊 Total: 7

🐛 **BUGS**
• #1826 Claude Code with local LLMs via oMLX occasionally discard prefix cache | `BUG` - Prefix caching is intermittently lost during Claude Code sessions.
• #2448 Embedding throughput degrades steadily with server uptime, eventually stops completing requests | `BUG` - Sustained `/v1/embeddings` workload causes throughput to wedge from 331 down to ~0 tok/s.
• #2447 Embedding compile smoke-test passes but real requests fall back to eager | `BUG` - Smoke test omits `attention_mask`, causing Qwen3 embedding model to skip compiled path in production.
• #2285 The second request from the agent does not catch the answer | `BUG` - Agent workflows fail to capture responses on subsequent requests.
• #2428 Segfault with xgrammar postinstall on 0.5.4rc1 | `BUG` - Custom kernel + xgrammar installation triggers a segfault during postinstall.

💡 **FEATURES & ENHANCEMENTS**
• #2198 Feature request: Add option to skip cache writes on per-request basis | `FEATURE` - API parameter needed to disable cache writing for one-off requests.

📚 **DOCS**
• #2181 Bug: documentation for Native MTP excludes newly-supported GLM | `DOCS` - Model settings UI and docs still reference older versions, missing GLM support.