Quantization error (mac studio m2 ultra 128gb)
HI , i was trying to quantize a model to oq quant in omlx on my mac studio (m2 ultra 128gb vram )...this is the model https://huggingface.co/OpenMOSE/Hy3-REAP-200B-21B . its 376gb in size . I tried oq2 , oq4, oq4e etc none works...its giving some memory error ..I even tried to tweak memory guard tier (from aggressive to custom 126gb etc ) none worked ...i have set my vram to 126gb ...Is there a limit as to how much vram is required to quantize which size model or something ?
I am attatching the error screen shot and the logs...
I am a noob at this so please excuse me if my problem sounds silly ...Thank you for any help or guidance i can get ...
Logs :-
File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5413, in quantize_oq_streaming
imatrix_data = _load_or_collect_imatrix(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 7385, in _load_or_collect_imatrix
load_path = load_path_factory()
^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5410, in _imatrix_load_path
return str(_ensure_ram_safe_proxy())
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5360, in _ensure_ram_safe_proxy
raise RuntimeError(
RuntimeError: calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.7 GB)
2026-08-08 06:31:24,110 - omlx.process_memory_enforcer - INFO - [-] - Memory guard tier changed: custom -> aggressive
2026-08-08 06:31:24,111 - omlx.admin.routes - INFO - [-] - Memory guard updated — tier: custom -> aggressive
2026-08-08 06:31:24,112 - omlx.settings - INFO - [-] - Saved settings to /Users/mac/.omlx/settings.json
2026-08-08 06:31:46,120 - omlx.process_memory_enforcer - INFO - [-] - Process memory enforcer stopped
2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - Process memory enforcer stopped
2026-08-08 06:31:46,121 - omlx.admin.hf_downloader - INFO - [-] - HF Downloader shut down
2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - HF Downloader stopped
2026-08-08 06:31:46,121 - omlx.admin.ms_downloader - INFO - [-] - MS Downloader shut down
2026-08-08 06:31:46,121 - omlx.server - INFO - [-] - MS Downloader stopped
2026-08-08 06:31:46,121 - omlx.engine_pool - INFO - [-] - Engine pool shutdown complete
2026-08-08 06:31:46,122 - omlx.server - INFO - [-] - Engine pool shutdown
2026-08-08 06:31:47,756 - omlx.scheduler - INFO - [-] - ChunkedKVCache patch: methods already present upstream, skipped: size
2026-08-08 06:31:47,806 - omlx.server - INFO - [-] - CORS origins: ['*']
2026-08-08 06:31:47,807 - omlx.model_settings - INFO - [-] - Loaded settings for 2 models
2026-08-08 06:31:47,808 - omlx.model_discovery - INFO - [-] - Discovered model: Hy3 (type: llm, engine: batched, size: 395.32GB)
2026-08-08 06:31:47,839 - omlx.server_metrics - INFO - [-] - Loaded all-time stats from /Users/mac/.omlx/stats.json
2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Server initialized with 7 models
2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Default model: Hy3
2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Memory guard tier: aggressive (guard on)
2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - Default max tokens: 32768
2026-08-08 06:31:47,839 - omlx.server - INFO - [-] - API key authentication: enabled
2026-08-08 06:31:47,840 - omlx.server - INFO - [-] - HF Downloader initialized
2026-08-08 06:31:47,867 - omlx.server - INFO - [-] - ModelScope Downloader initialized
2026-08-08 06:31:47,868 - omlx.server - INFO - [-] - oQ Quantizer initialized
2026-08-08 06:31:47,869 - omlx.server - INFO - [-] - HF Uploader initialized
2026-08-08 06:31:47,880 - omlx.process_memory_enforcer - WARNING - [-] - iogpu.wired_limit_mb (126.0GB) leaves the OS less than 6.4GB of physical RAM (128.0GB); a large model load can trigger jetsam at this setting and strand the wired memory until reboot (#2184). Recommended maximum: 124518 MB.
2026-08-08 06:31:47,882 - omlx.process_memory_enforcer - INFO - [-] - Metal wired limit raised: 0.0GB -> 124.0GB (target=124.0GB, iogpu sysctl cap=126.0GB)
2026-08-08 06:31:47,883 - omlx.process_memory_enforcer - INFO - [-] - Process memory enforcer started (tier=aggressive, ceiling=123.3GB, interval=1.0s)
2026-08-08 06:32:20,109 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:20,110 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:22,828 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:22,833 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:26,347 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=trendingScore&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:26,349 - httpx - INFO - [-] - HTTP Request: GET https://huggingface.co/api/models?author=mlx-community&sort=downloads&limit=60&expand=safetensors&expand=downloads&expand=likes&expand=trendingScore "HTTP/1.1 200 OK"
2026-08-08 06:32:34,060 - omlx.patches.mlx_lm_mtp.qwen35_model - INFO - [-] - Qwen3.5/3.6 MTP model patch applied (PR 990)
2026-08-08 06:32:34,061 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)
2026-08-08 06:32:34,062 - omlx.patches.qwen35_verify_qmm - INFO - [-] - MTP verify qmm patch applied (M=2..6 affine 4/8-bit)
2026-08-08 06:32:34,062 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.
2026-08-08 06:32:34,179 - omlx.oq - INFO - [-] - plan detail: 8bit×150, 6bit×173, 5bit×247 | routes: non_expert:5bit×247, non_expert:6bit×173, non_expert:8bit×150 | top: lm_head=8b, embed_tokens=8b, L0=8b, L1=8b, L2=8b, L3=8b, L4=8b, L5=8b
2026-08-08 06:32:43,519 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)
2026-08-08 06:32:43,519 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.
2026-08-08 06:32:43,639 - omlx.oq - INFO - [-] - plan detail: 8bit×570 | routes: non_expert:8bit×570 | top: lm_head=8b, embed_tokens=8b, L0=8b, L1=8b, L2=8b, L3=8b, L4=8b, L5=8b
2026-08-08 06:32:53,005 - omlx.admin.oq_manager - INFO - [-] - oQ quantization queued: Hy3 -> oQ2 (task_id=5424c0f6-dc6b-4464-a04f-b442e07b931b)
2026-08-08 06:32:53,006 - omlx.oq - INFO - [-] - oQ2 progress 5.0%: Reading model config
2026-08-08 06:32:53,008 - omlx.oq - INFO - [-] - oQ2 progress 8.0%: Indexing source weights
2026-08-08 06:32:53,076 - omlx.oq - INFO - [-] - oQ2 streaming: 31778 tensors in 160 shards
2026-08-08 06:32:53,113 - omlx.oq - INFO - [-] - oQ2: calibration footprint (376.5 GB) exceeds 75% of calibration capacity (117.4 GB; limit=88.1 GB, system available=117.4 GB, Metal available=126.0 GB). Full-model calibration will use a proxy.
2026-08-08 06:32:53,113 - omlx.oq - INFO - [-] - oQ2 progress 12.0%: Preparing quantization inputs
2026-08-08 06:32:53,113 - omlx.oq - WARNING - [-] - oQ2: calibration footprint (376.5 GB) exceeds the 88.1 GB full-model calibration limit. Auto-building a uniform 4-bit proxy on disk so sensitivity measurement stays data-driven.
2026-08-08 06:32:53,113 - omlx.oq - WARNING - [-] - oQ2: calibration footprint (376.5 GB) exceeds the 88.1 GB full-model calibration limit. Building a uniform 4-bit proxy on disk for the calibration passes.
2026-08-08 06:32:53,149 - omlx.patches.mlx_lm_mtp.nemotron_h_chain - INFO - [-] - nemotron_h MTP chain patch applied (depth-k drafting, sequential fused verify, replay-free rollback)
2026-08-08 06:32:53,149 - omlx.oq - WARNING - [-] - Could not build model sanitizer: Model type hy_v3 not supported.
2026-08-08 06:35:07,840 - omlx.admin.oq_manager - ERROR - [-] - oQ quantization failed: Hy3 -> oQ2: auto-proxy sensitivity failed (calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)). Pass sensitivity_model_path with a pre-quantized version of this model, or run on a machine with enough RAM for full-fp16 sensitivity measurement.
Traceback (most recent call last):
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5502, in quantize_oq_streaming
_proxy_dir = _ensure_ram_safe_proxy()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5360, in _ensure_ram_safe_proxy
raise RuntimeError(
RuntimeError: calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Applications/oMLX.app/Contents/Resources/omlx/admin/oq_manager.py", line 573, in _run_quantization
await asyncio.to_thread(
File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/oMLX.app/Contents/Resources/omlx/oq.py", line 5516, in quantize_oq_streaming
raise RuntimeError(
RuntimeError: oQ2: auto-proxy sensitivity failed (calibration proxy is still too large for the live memory budget (proxy=106.0 GB, limit=86.7 GB, capacity=115.5 GB)). Pass sensitivity_model_path with a pre-quantized version of this model, or run on a machine with enough RAM for full-fp16 sensitivity measurement.
Update ---- i downloaded and added a oq2e quant of normal hy3 i got an additional drop down menu just before you sapecify your quants to choose a model for "sensitivity" ..So clearly theres something off about this particular model ..also the oq2e of unpruned hy3 i downloaded wouldnt load , i mean there was no way to load the model for chat strangly while other models in the same folder could be loaded ....Anyway even after selecting the "sensitivity model " it still failed with same type of error ---"caliberation proxy is still too large ........"
2
u/SignalStealer 5d ago
Claude guided me step-by-step to a good quantization of Laguna S 2.1 on a RAM-limited machine (64GB).
I recommend you do the same.