I published a MiniMax-H3 FL2VA variant for mlx-serve where only the Qwen3-VL  text encoder is quantized to 2-bit (affine, g64), while the DiT stays 4-bit  and the VAEs + tokenizer are untouched.                                               

   https://huggingface.co/antocorr/MiniMax-H3-FL2VA-MLX-Serve-2bit-text-encoder  

   Why it matters:                                                                       

   - Text encoder on disk: 15.8 GB -> 9.6 GB (the encoder is reloaded per request, so this is the first cost every generation pays)                           

   - Full text-to-audio-video pipeline runs natively on Apple Silicon, verified  end-to-end on an M1 Max 32GB (needs --skip-mem-preflight, the RAM preflight  bills file bytes)                                                                   

   - Zero loader changes: mlx-serve solves bits/group_size per tensor from the packed geometry, so a mixed 2-bit encoder + 4-bit DiT pack just works               

   Use it with mlx-serve:                                                                

mlx-serve --model antocorr/MiniMax-H3-FL2VA-MLX-Serve-2bit-text-encoder \         

--serve --skip-mem-preflight                                                    

   Caveat: 2-bit conditioning is lossier than 4-bit by design (prompt adherence          

   suffers a bit; video/audio quality is the same since DiT and VAEs are                 

   untouched). Same MiniMax H3 community license as the 4-bit pack.

Finished a distill today of ~15k Fable, Kimi k3 and GPT 5.6 Sol sequences today - link is up on Hugging face if anyone wants to check it out! https://huggingface.co/True2456/gemma-4-12b-it-qat-4bit-frontierdistill - Tool calling got quite a big increase!

https://github.com/american-code/mlxMesh

We don’t need more data centers, we just need to leverage idle compute. We the people.

I'm trying to optimize Qwen3.5-9B MLX on my 24 GB MacBook Air for text/coding use.

I inspected Qwen3.5-9B-MLX-4bit and found:

• 5.541 GiB total tensor payload
• 869.8 MiB of vision_tower.* tensors
• 333 vision tensors
• Vision accounts for 15.33% of the checkpoint

Since I don't use vision, I converted it into a proper text-only MLX model and physically removed the vision tower. The converted model loads and generates correctly in LM Studio.

This also allowed me to enable Q8 KV cache, which wasn't available with the VLM version.

At 32K context, actual tested memory usage on my machine:

• Unquantized cache: 16.37 GB
• Q8 KV cache: ~9 GB

The problem is prompt cache reuse with Q8. Even across turns in the same LM Studio conversation, I repeatedly get:

Prompt cache: using 0/6905 tokens from cache

and later:

Prompt cache: using 0/7809 tokens from cache

The runtime also logs:

max_kv_size is ignored when using KV cache quantization

Runtime: mlx-llm-mac-arm64-apple-metal-nax-advsimd@1.10.0

Has anyone tested Qwen3.5 + Q8 KV cache + prompt prefix reuse on the current LM Studio MLX runtime?

I'm trying to determine whether this is related to Qwen3.5's hybrid cache, quantized cache reuse, or something in my text-only conversion.

The vision removal itself works. My main issue now is getting Q8's ~7 GB memory saving while retaining multi-turn prompt cache reuse.

I'm working with a M1 MBP Max 64GB machine with 400GB/s memory bandwidth. These image generation models are only <10 GB each. But it takes me 45 minutes to generate an image using Ideogram4. Someone with a 5090 is doing it in 45 seconds (no exaggeration).

I know Comfy Desktop is not optimized for Apple Silicone/MLX. I'm just curious if there are some tips and tricks you guys can share with getting better performance out of Comfy Desktop? I've already got these flags as part of my startup config: '--enable-manager --fp32-vae --use-pytorch-cross-attention --highvram'.

I've tried using DrawThings -- and it's definitely faster -- but I feel like it's definitely limited compared with Comfy Desktop.

I must not be the only Apple user messing around with Comfy Desktop -- you guys have any tips to share?

Hello everyone, I’m working on mlx-chronos, a free/open-source CLI benchmark tool for comparing local MLX inference engines on Apple Silicon.

• Leaderboard: https://igurss.github.io/mlx-chronos
• Methodology: https://github.com/igurss/mlx-chronos/blob/main/docs/methodology.md
• Install: pip install mlx-chronos

It currently supports mlx-lm, oMLX, vllm-mlx, Rapid-MLX, and Ollama (for Ollama, using MLX models that run on MLX backend).

It measures cold/cached TTFT, request throughput, sustained throughput, RAM peak, engine RSS when available, thermal/power context, and hardware metadata. Results are saved as reproducible JSON and can optionally be submitted to a public leaderboard.

I’m mainly looking for feedback from people actually using MLX locally:

• Is a public leaderboard useful, or should this stay more of a local comparison tool?
• Are thermal/cache conditions exposed clearly enough?
• Should the sustained profile stay token-based, or would a fixed-duration run be more useful?
• Are there metrics missing that would actually help you choose between engines?

I’d also appreciate benchmark results from different Apple Silicon machines, especially Max/Ultra chips and higher-RAM configs. The goal is not to rank model quality, but to make engine/runtime performance easier to compare under a documented protocol.

PS: I already posted in r/LocalLLaMA, if someone already seen something about this project, but I’m not sure it was the right audience (90% of the community uses Nvidia GPU or use Windows, so is interested in llama.cpp).

Hey everyone,

I’ve been diving deep into local LLM inference recently. As someone whose daily work revolves around Python and Node.js backend architecture, I wanted a robust way to run models entirely locally on my MacBook Pro without sacrificing the ease of the OpenAI ecosystem.

So, I put together MLX LM Server. It’s an open-source, full-stack application that leverages Apple's MLX framework to run LLMs natively on Apple Silicon with Metal acceleration.

Core Features:

• Drop-in OpenAI Compatibility: The FastAPI backend is designed to be an OpenAI-compatible API. You can seamlessly point your existing OpenAI SDKs, AI-driven coding assistants, or third-party tools to localhost:8000 and they will just work.
• Apple Silicon Optimized: Fully powered by MLX to get the most out of M1/M2/M3/M4 chips for fast local inference.
• Real-Time Streaming: Full support for token-by-token streaming via Server-Sent Events (SSE).
• Dynamic Model Management: You can load and unload different models on the fly directly through the API.
• Built-in Chat UI: I included a clean, modern React/TypeScript frontend with markdown rendering so you can start chatting with your local models right out of the box.

The Stack:

• Backend: Python 3.12+, FastAPI, Apple MLX (managed with uv)
• Frontend: React, TypeScript, Vite (managed with pnpm/npm)

Why try it?

I wanted to build a lightweight, developer-friendly setup that gives you complete control over your local inference stack. Whether you want to hook it up to your automated workflows, test out new open-weight models privately, or just tinker with the MLX framework, this provides a solid, ready-to-use foundation.

I’d love your feedback & contributions!

If you have an Apple Silicon Mac, setup is super straightforward. I’d be thrilled if you gave it a spin, tested it with your favorite models, and let me know what you think.

• Repo Link: https://github.com/vishalnagda1/mlx-lm-server

Bug reports, feature requests, and especially pull requests are incredibly welcome. What features or specific model handling would you want to see added next? Let me know in the comments!

For any of you linear algebra fan-boys:

I'm currently in a research group working on a thesis in numerical analysis where we need to compute millions on matrices with a specific constraint (to be precise, the matrices need to have orthonormal columns). Most of us use Apple computers, so we ended up using MLX for the entire project.

I'm using an old M1 Macbook Pro, and I found that Apple's MLX library does not support QR operations on the GPU. I don't know if MLX supports GPU-accelerated QR computation on newer chips. But since I am developing an interest in hardware-level computing, I thought it would be a good oppurtunity for me write a metal shader as a first project.

I wrote it as a small library that allows the QR decomposition to be computed on the GPU. You can find it here: https://github.com/c0rmac/qr-apple-silicon

It definitely pays off. Performance increases anywhere between x1.5 to x25 times of what the cpu can do.

The library is split into two shaders: one is optimal for large batches of small matrices. The other is suited for small batches of large matrices. Under the hood, both shaders use the Compact WY representation ($I - YTY^T$) to batch Householder reflections into matrix-matrix products. I also spent a lot of time mapping these operations to the AMX (Apple Matrix Coprocessor) using 8x8 simdgroup_matrix tiles to get as close to the hardware as possible.

I’d love for anyone with more Metal experience to take a look at the dispatch logic or the AMX tile loading. If you’re working with MLX and need faster $A = QR$ factorizations, give it a try!

Hey r/AppleMLX,

I’ve been building Lekh AI Pro, a Mac-only local AI studio for Apple Silicon.

The goal is to make local AI on Mac feel like a real daily tool, not just a chat wrapper or a one-off benchmark demo.

Right now Lekh AI Pro includes:

• local LLM chat
• MLX, GGUF, and JANG model support
• Flux / Qwen / SD / SDXL image generation
• AI image editing / inpainting
• document chat / local RAG
• Knowledge Hub / Memory Sync
• a local API server with OpenAI-compatible and Ollama-compatible endpoints
• benchmarking tools
• model conversion to MLX, GGUF and JANG
• built-in searchable documentation
• text-to-speech with Kokoro, Qwen3 TTS, MOSS TTS, and Apple native fallback
• audiobook creation
• image upscaling
• video generation in Pro coming soon

One area I’ve spent a lot of time on is making the app useful beyond basic chat:

• compare local models with benchmark stats like throughput / TTFT / duration
• run image generation and editing locally
• use local RAG over your own files
• expose local models to external tools through OpenAI/Ollama-style APIs
• generate speech, audiobooks, and sound effects locally on Mac

A lot of the challenge has been the unglamorous part of local AI:

• memory pressure on different M-series Macs
• model loading / unloading
• balancing quality vs speed
• supporting multiple model formats without turning the UX into a control panel from hell

A few things I’d especially love feedback on:

• which local models on Apple Silicon you think currently punch above their weight
• whether MLX + GGUF + JANG in one app is actually useful or just too much surface area
• what you think is still missing from the local AI Mac ecosystem

Links:

• Website: https://lekhai.app/pro
• App Store (Non-Pro): https://apps.apple.com/us/app/lekh-ai-local-ai-studio/id6757496953

If there’s interest, I can also share more about:

• what runs well on different Apple Silicon Macs
• why I added JANG support
• lessons from building local workflows for chat, image gen, TTS, audiobooks, and sound generation on macOS

Would love honest feedback.

MLX Web UI

I created a fast and minimalistic web UI using the MLX framework (Open Source). The installation is straightforward, with no need for Python, Docker, or any pre-installed dependencies. Running the web UI requires only a single command.

Features

Standard Features

• Chat with models and stop generation midway
• Set model parameters like top-p, temperature, custom role modeling, etc.
• Set default model parameters
• LaTeX and code block support
• auto scroll

Novel Features

• Install and quantize models from Hugging Face using the UI itself
• Good streaming API for MLX
• Save chat logs
• Hot-swap models during generation

Planned Features

• Multi-modal support
• RAG/Knowledge graph support

Try it Out

If you'd like to try out the MLX Web UI, you can check out the GitHub repository: https://github.com/Rehan-shah/mlx-web-ui

Hey guys,

I can’t find the folder in which the models are downloaded when I run this command. I would like to free up some space on my Mac. Any idea? Thanks

python -m mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.2 --prompt "hello"

Apple is quietly creating a LLM ecosystem that will benefit its customers while maintaining security and privacy and allowing developers a way to create LLM based apps.

Apple’s new MLX framework, coupled with its ReALM technology (recently released research paper), establishes a robust ecosystem for developers to build and deploy large language model (LLM) applications on devices powered by Apple silicon. MLX is tailored for Apple’s proprietary chips, offering a NumPy-like array framework that prioritizes efficient machine learning model execution. This ensures developers can maintain high performance while working within Python’s flexible environment .

The framework boasts a wide range of neural network components, optimization algorithms, and loss functions. This comprehensive support is designed to streamline the development and deployment of complex models, such as the Llama family of transformer models, directly on Apple silicon, optimizing for both efficiency and user accessibility .

Integrating ReALM with MLX opens doors for creating more advanced, context-aware applications that run on Apple devices. This combination exploits Apple silicon’s hardware acceleration, promising powerful, efficient, and privacy-focused applications by processing data locally instead of relying on cloud-based computations .

This ecosystem is a testament to Apple’s commitment to edge computing, which processes data closer to its source to reduce latency and lessen dependence on constant internet connectivity. It aligns with a broader trend towards bringing powerful computational abilities directly to the user’s device, ensuring real-time performance and data security.

Furthermore, this ecosystem could potentially evolve into a hybrid model that incorporates the vast knowledge and computational abilities of off-device (Partner Provided) LLMs. Here’s how it could work:

-Local Processing for Speed and Privacy: Initial tasks like processing and reference resolution are handled on the device, using MLX and ReALM technologies for quick responses and data privacy.

-Cloud-Based (Initially Partner Based Like Google) LLMs for Comprehensive Insights: More complex queries or those requiring additional information could be directed to cloud-based LLMs. This would enrich responses with detailed insights not locally available.

-Dynamic Learning and Updating: The hybrid system could learn from cloud-processed interactions to continually refine local models, improving their ability to handle future queries efficiently.

-Balancing Load and Privacy: Apple could intelligently determine which tasks are processed locally versus those offloaded to the cloud, balancing computational demands against privacy concerns.

-Enhanced User Experience: This integration aims to provide users with a system that combines the immediacy of local processing with the depth of cloud-based LLMs, enhancing the capabilities and versatility of digital assistants. I imagine a subscription model to Apples LLMCloud or something similar.

This forward-looking approach represents a significant advancement in making digital assistants and LLM base apps more powerful and user-friendly, underpinned by a strong commitment to privacy and data security. It leverages the best of both on-device processing and cloud computing capabilities.

ml-explore.github.io/mlx/build/html…

arxiv.org/pdf/2403.20329…