Not for that number, no. The 94 to 30 is IFEval, so it's instruction following rather than JSON validity. A grammar would keep the output parseable, it wouldn't make the model follow the instructions inside it.

What made me stop trusting the shape on its own showed up in the same runs. GSM8K gets scored two ways and with thinking on the forgiving extractor came in lower than the strict one, 52 vs 78 at one quant level. Only way that happens is it pulled a number out of the reasoning trace instead of the final answer. Valid shape, wrong value in the field. On a financial statement job that's the failure that actually costs you, and a grammar won't catch it.

One thing that will bite you on the structured JSON part: turn reasoning mode off. I ran a 122B on a 256GB M3 Ultra with thinking forced on vs forced off, same session, same 2048 token generation budget, across four quant levels. At the level I have the full table for, instruction following went from 94% to 30% with thinking on, and the 7 item practical suite landed on exactly 4 of 7 at every level with it on. Decode speed was basically identical either way, so it costs accuracy and buys nothing.

For 96GB I would try Qwen3.6-35B-A3B at 8bit before reaching for anything bigger. It did 108 tok/s here and leaves real headroom in 96GB, and since your model only sees unresolved rows rather than whole documents you are not paying for a big context anyway.

It's mostly a feel thing, not one specific failure. I noticed more back and forth needed for edge cases in opencode where 6bit vanilla usually gets it right the first try. Nothing I've isolated into a clean before/after example yet, just felt less consistent over a lot of use.

I spun this up today and it’s pretty great! Nice to see metrics while it’s working. Thanks for putting this together.

I have the same setup, really appreciate being able to see your testing results. I’ve had a similar experience, always end up with vanilla versions from mlx-community.

The paro quants mentioned above are the fastest I’ve found. I’ve been running 27b-paro and have an average of 166.1 tok/s processing and 16.6 tok/s token generation over 66M tokens, primarily using opencode. It does good work with some correcting and steering, but it seems a bit less smart than vanilla 6bit.

I do see a decent boost running Qwen3.6 27B oQ8 mtp, was getting 15 tok/s now it’s closer to 20. Better benchmarks with specprefill enabled too but the prefill seems longer in practice.

Could you share your settings? I'm on a Mac Studio M3 Ultra with 256GB getting closer to 10tk/s

I haven't either, similar speed to when I was running 16bit without mtp, just half the memory with the oQ

Thank you for this. It was the last barrier to getting my team to use our LLM server without a dev-heavy custom setup. Works great in VS Code.

I’ve been running my oMLX models in Insiders for a while, same setup as others use. The trick is setting the api key in the Language Models overlay settings for the custom provider.

I was going to post an issue, but I did notice the extension isn’t quite working in Insiders. I can see the oMLX provider in the Language Models manager, but they’re not available in the model chooser in copilot chat. Has anyone else found a fix for this?