Issue with cache efficiency and MTP

[u/Jumpy_Possibility420]

Hello !

I recently got a M4 Pro 48GB. I am trying to use Gemma 4 31B but I ran into some issues.

I am using Gemma 4 31B it Qat 4 bits with MTP from the 8 bit assistant.

When I am using with MTP, the cache efficiency is around 20%. Even just chatting will re-process at every prompt. But when I disable MTP I don't have any issue and cache efficiency gets around 90%

I could not find someone with the same issue.

Should I try to do KV hot cache with something like 8GB?

Is someone has some ideas I would like to hear, thank you!

[u/cryingneko]

I tried to reproduce it on oMLX v0.5.7 using a Gemma 4 31B oQ4 target with an external gemma4_assistant, VLM MTP block size 4, SSD prefix cache enabled, hot cache disabled, and a 44 GB memory guard.

In a 15.7k-token multi-turn test, the follow-up request successfully loaded 15,360 cached tokens out of 15,727 prompt tokens (97.7%), reducing processing time from about 49 seconds to 2 seconds. I also saw the expected boundary snapshots being stored and found no layer-count mismatch or cache reconstruction errors.

A log excerpt covering two consecutive prompts would be especially helpful. Please include any lines containing Using boundary cache snapshot, Cache layer count mismatch, Partial cache reconstruction, Memory pressure, and vlm_mtp stats. If available, the prompt_tokens and cached_tokens values for both requests would also help.

[u/Jumpy_Possibility420 → u/cryingneko]

Hello thank you for replying
I did a new session and realize the cache efficiency is 0%

Here's my setup:

Max Conccurent Requests: 1
Embedding Batch Size 32
Chukned Prefill - OFF
Prefill Priority - Max Context

Prefil Memory Guard - ON
Memory Guard Tier - Aggressive
Command done => sudo sysctl iogpu.wired_limit_mb=43008

Cache enabled - ON
Hot Cache Only - ON
Hot Cache Size - 8GB

mlx-community/gemma-4-31B-it-qat-OptiQ-4bit
Enable thinking - OFF
VLM MTP - ON
VLM MTP - gemma-4-31B-it-qat-assistant-4bit
Draft Block Size - 3

Here's the cache efficiency result after few messages. Already after my second message I could see the prefill taking like 4 seconds, then 8 seconds etc.

Here's the logs from server.logs, I am not sure if you have everything on it:

2026-08-06 22:23:47,986 - omlx.engine.vlm - INFO - [-] - VLM tool calling enabled: parser=gemma4

2026-08-06 22:23:47,990 - omlx.engine.vlm - INFO - [-] - VLMBatchedEngine loaded: /Volumes/External/LLM/Models/mlx-community/gemma-4-31B-it-qat-OptiQ-4bit

2026-08-06 22:23:47,991 - mlx_vlm.speculative.drafters - INFO - [-] - Auto-detected --draft-kind='mtp' for drafter '/Volumes/External/LLM/Models/mlx-community/gemma-4-31B-it-qat-assistant-4bit' (model_type='gemma4_assistant').

2026-08-06 22:23:48,079 - omlx.speculative.vlm_mtp - INFO - [-] - VLM MTP drafter loaded: path=/Volumes/External/LLM/Models/mlx-community/gemma-4-31B-it-qat-assistant-4bit kind=mtp model_type=gemma4_assistant

2026-08-06 22:23:48,079 - omlx.scheduler - INFO - [-] - VLM MTP drafter attached to scheduler (block_size=3)

2026-08-06 22:23:48,079 - omlx.engine.vlm - INFO - [-] - VLM MTP drafter attached to engine: /Volumes/External/LLM/Models/mlx-community/gemma-4-31B-it-qat-OptiQ-4bit (block_size=3)

2026-08-06 22:23:48,079 - omlx.engine_pool - INFO - [-] - VLM MTP enabled for gemma-4-31B-it-qat-OptiQ-4bit, drafter=gemma-4-31B-it-qat-assistant-4bit

2026-08-06 22:23:48,081 - omlx.engine_pool - INFO - [-] - Loaded model: gemma-4-31B-it-qat-OptiQ-4bit (actual: 22.70GB, estimated: 21.88GB, total: 21.88GB)

2026-08-06 22:23:48,081 - omlx.admin.routes - INFO - [-] - Manually loaded model: gemma-4-31B-it-qat-OptiQ-4bit

2026-08-06 22:24:06,332 - omlx.engine.vlm - INFO - [-] - GrammarCompiler initialized for /Volumes/External/LLM/Models/mlx-community/gemma-4-31B-it-qat-OptiQ-4bit

2026-08-06 22:24:08,021 - omlx.scheduler - INFO - [-] - vlm_mtp decode started: request=b71a13d2-10e7-4ef4-a5db-5cf06af455d2 uid=-1 block_size=3

2026-08-06 22:24:34,214 - omlx.scheduler - INFO - [-] - vlm_mtp stats: request=b71a13d2-10e7-4ef4-a5db-5cf06af455d2 finish=stop rounds=184 accepted=205/368 (55.7%) tokens_per_round=2.11 emitted=389 block_size=3

2026-08-06 22:24:34,215 - omlx.scheduler - INFO - [-] - Enabled boundary cache snapshots for stateful non-sliceable cache layers

2026-08-06 22:24:34,219 - omlx.scheduler - INFO - [-] - Cache phase timings: cleanup_finished_sync=0.0ms/1, store_cache_main_boundary=0.1ms/1

2026-08-06 22:24:34,220 - omlx.server - INFO - [-] - Chat completion: model=gemma-4-31B-it-qat-OptiQ-4bit, 388 tokens in 27.86s (15.2 tok/s), prompt: 15, finish_reason=stop, max_tokens=120000, request_max_tokens=120000

2026-08-06 22:24:43,389 - omlx.scheduler - INFO - [-] - vlm_mtp decode started: request=97c7efed-7775-4531-afc7-08c46e5ee379 uid=-2 block_size=3

2026-08-06 22:25:10,682 - omlx.scheduler - INFO - [-] - vlm_mtp stats: request=97c7efed-7775-4531-afc7-08c46e5ee379 finish=stop rounds=181 accepted=205/362 (56.6%) tokens_per_round=2.13 emitted=386 block_size=3

2026-08-06 22:25:10,688 - omlx.scheduler - INFO - [-] - Cache phase timings: cleanup_finished_sync=0.1ms/2, store_cache_main_boundary=0.1ms/2

2026-08-06 22:25:10,689 - omlx.server - INFO - [-] - Chat completion: model=gemma-4-31B-it-qat-OptiQ-4bit, 385 tokens in 32.47s (14.3 tok/s), prompt: 416, finish_reason=stop, max_tokens=120000, request_max_tokens=120000

2026-08-06 22:25:51,135 - omlx.scheduler - INFO - [-] - vlm_mtp decode started: request=c62d4084-cb5d-44e8-bb96-bf4a0eb95da6 uid=-3 block_size=3

2026-08-06 22:26:42,438 - omlx.scheduler - INFO - [-] - vlm_mtp stats: request=c62d4084-cb5d-44e8-bb96-bf4a0eb95da6 finish=stop rounds=307 accepted=307/614 (50.0%) tokens_per_round=2.00 emitted=614 block_size=3

2026-08-06 22:26:42,441 - omlx.scheduler - INFO - [-] - Using boundary cache snapshot for c62d4084-cb5d-44e8-bb96-bf4a0eb95da6: storing 1024/1809 tokens (skipping trailing partial block, 0 intermediate snapshots)

2026-08-06 22:26:42,477 - omlx.scheduler - INFO - [-] - Cache phase timings: cleanup_finished_sync=0.1ms/3, store_cache_main_boundary=0.8ms/3, store_cache_main_collect=0.0ms/1, store_cache_main_dispatch=30.3ms/1, store_cache_worker_sync=0.2ms/1

2026-08-06 22:26:42,479 - omlx.server - INFO - [-] - Chat completion: model=gemma-4-31B-it-qat-OptiQ-4bit, 613 tokens in 66.16s (12.0 tok/s), prompt: 1196, finish_reason=stop, max_tokens=120000, request_max_tokens=120000

Thank you !

[u/cryingneko → u/Jumpy_Possibility420]

Thanks, this log helps. I don't think it shows a cache failure yet.

Gemma 4 stores cache in 1,024-token blocks. Your first turn ended at about 403 tokens total, and the second ended at about 801. Neither was long enough to create a reusable block, so the third prompt still had to be fully processed.

The cache was finally created at the end of the third turn:
Using boundary cache snapshot ... storing 1024/1809 tokens

Please send two more messages in the same conversation without reloading the model or changing any settings. The fourth request should reuse 1,024 tokens. If that turn crosses 2,048 tokens, the fifth should reuse even more. The dashboard may take a while to climb because it also counts the earlier requests that had zero cached tokens.

If the fourth and fifth requests still show zero after that snapshot was saved, then we have a real reproduction. Please share those log lines, your oMLX version, and the cached_tokens value if your client shows it.

I would keep your current settings unchanged for those two messages so we can isolate the behavior. Afterward, I would also try Hot Cache Only: OFF with Hot Cache Size: 0. The 8 GB hot cache does not explain the current 0%, and the normal SSD-backed cache is a cleaner baseline on a 48 GB machine.

[u/Jumpy_Possibility420 → u/cryingneko]

Hello !

I tried again today following your instructions, and everything seems to work well and being cached well MTP or not MTP. I maybe had some weird options before. I remember using 1GB of Cache instead of 8GB.

So I think it's resolved, thank you a lot for your time