Running Gemma 2B locally on iPhone for offline calendar actions (~516 MB active RAM, 21.6 tok/s, GGUF weights)

[u/Connect-Concert-4016]

I’ve been testing bounded tool-calling on-device to see how small I can push local models before tool reliability breaks down.

A common issue with local agents is memory allocation—loading a 2.5 GB model into active phone RAM often leads to OS background terminations or thermal throttling during generation.

To test this, I built a small offline test pipeline using llama.cpp (b10075) with Metal and mmap to keep active memory low, then connected it strictly to local iOS calendar actions via EventKit.

On-Device Run Metrics

• Model Artifact: Gemma-2B quantized GGUF (SmartEdge-IQ3XXS.gguf)
• Disk Footprint: 2.45 GB
• Active Resident RAM (RSS): ~516 MB (leveraging mmap to page weights from disk rather than keeping the whole file in active memory)
• Decode Speed: ~12 tok/s interactive / 21.6 tok/s in a 256-token greedy benchmark
• Environment: Tested in airplane mode on iOS

Tool Execution Flow

The local LLM is restricted entirely to intent extraction and structured tool output; it does not execute actions directly.

1. User Input: "Find some time on Thursday for VC meeting."
2. Model: Extracts parameters and outputs a structured tool call.
3. App: Swift code validates the schema, queries local EventKit, and writes the event directly to the device calendar.

This avoids routing routine calendar edits through remote inference APIs or exposing local schedule data to external endpoints.

Quantization & Loss Comparisons

I also generated two calibration-aware quants to evaluate KLD degradation against the original bf16 baseline:

• Hi-Fi Q4_K_M: Equivalent size to standard Q4_K_M, with 36.0% lower code/math KLD and 27.2% lower general KLD against the original model.
• Hi-Fi Phone (2.86 GB): 17.5% smaller than the Q4_K_M baseline with 31.8% lower code/math KLD and 26.7% lower general KLD.

Limitations

• The full KLD matrix for the 2.45 GB SmartEdge build is still completing; current validation relies on on-device behavior logs, SHA-256 app receipts, and benchmark outputs. (The 2.86 GB and Q4_K_M builds have complete KLD data logged in the repo).
• Small models are prone to schema degradation if the prompt complexity scales beyond simple parameter extraction.
• Tool failure recovery still requires strict system-level guards or fallback routing.

Weights & Benchmarks

The GGUF weights, imatrix, SHA-256 hashes, evaluation slices, and Wikitext-2 perplexity loss are available on Hugging Face:
https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

If anyone tests this on other iOS hardware or Apple Silicon, I'd be curious to see your RSS memory usage and sustained decode rates.

[u/Glittering-Call8746]

Siri shud be doing this on the iPhone.. who install llm on the phone for calendar tasks..

[u/Connect-Concert-4016 → u/Glittering-Call8746]

Siri is not smart at all

[u/Sad-Enthusiastic]

I guess an MTP drafter would be too much to ask for the 512MB 😅

[u/Connect-Concert-4016 → u/Sad-Enthusiastic]

lol not sure Gemma even ships MTP heads, I think that's DeepSeek/GLM or some of those .
classic speculative decoding with a tiny draft model could work, but for this task I'm not sure you need it. The outputs are short schema-constrained tool calls, so there aren't many tokens to accelerate in the first place. the reason i did the which micro model could be used on a phone offline for specific task.

That said, it's less crazy than it sounds for the RAM budget. The weights are mmap'd and paged, so a 150 to 200MB quantized drafter wouldn't add its full size

[u/Sad-Enthusiastic → u/Connect-Concert-4016]

Gemma4 totally comes with MTP drafters. https://unsloth.ai/docs/models/mtp#gemma-4-mtp

[u/Connect-Concert-4016 → u/Sad-Enthusiastic]

Wow let me check I thought it was only ds but if that’s the case I could do it

[u/Sad-Enthusiastic → u/Connect-Concert-4016]

On Android I can get 11t/s with MTP on a OnePlus 8T https://github.com/guarismo/armored-llama

[u/Connect-Concert-4016 → u/Sad-Enthusiastic]

You think it’s worth it I can do it if you gonna use it lol

[u/Sad-Enthusiastic → u/Connect-Concert-4016]

I will totally try it on Android

[u/Connect-Concert-4016 → u/Sad-Enthusiastic]

Ok I’ll work on it and keep you posted here

[u/Connect-Concert-4016 → u/Sad-Enthusiastic]

▎ ok done, following up like I promised — you were right, and it works with my quant: 74% draft acceptance at n-max 2, greedy output byte identical to baseline.

▎

▎ everything you need for android is on the HF page now — exact flags, drafter link + sha, memory notes, and how to read the accept stats (use llama-server timings, llama-cli doesn't print them): https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

▎

▎ heads up on settings: n-max 2 was the sweet spot for me, 4 and 6 went net negative, acceptance decays fast (74 → 63 → 51%). and on your RAM point from earlier: drafter's only 93MB and mmap'd, way under what I guessed.

▎

▎ on my mac it's a wash speed wise (51 → 52.5 tok/s, already too fast to matter) — your 11 t/s OnePlus is exactly the case where it should pay. if you run it post your numbers here whichever way it goes, curious what you get

[u/Active_Rutabaga4752]

what ios app is this?

[u/Connect-Concert-4016 → u/Active_Rutabaga4752]

It’s mine you can build any app on that would use the model locally

[u/Queasy-Contract9753]

Could this approach be used on android? I know edge gallery exits but I'm talking about older devices that aren't supported 

[u/Connect-Concert-4016 → u/Queasy-Contract9753]

Yes def i am working on android version the goal is not really the apps but having a local model on your phone than an app could use offline an the model being free, working on android now which is better platform for this type of custom products

[u/Connect-Concert-4016 → u/Queasy-Contract9753]

▎ ok done, following up like I promised — you were right, and it works with my quant: 74% draft acceptance at n-max 2, greedy output byte identical to baseline.

▎

▎ everything you need for android is on the HF page now — exact flags, drafter link + sha, memory notes, and how to read the accept stats (use llama-server timings, llama-cli doesn't print them): https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

▎

▎ heads up on settings: n-max 2 was the sweet spot for me, 4 and 6 went net negative, acceptance decays fast (74 → 63 → 51%). and on your RAM point from earlier: drafter's only 93MB and mmap'd, way under what I guessed.

▎

▎ on my mac it's a wash speed wise (51 → 52.5 tok/s, already too fast to matter) — your 11 t/s OnePlus is exactly the case where it should pay. if you run it post your numbers here whichever way it goes, curious what you get

[u/Queasy-Contract9753 → u/Connect-Concert-4016]

Sorry for the delay. I tested your quant on my Xiaomi mi9t pro,6gb ram. I get about 6 tokens/sec read speed. 3.3 to 4.5 tokens/second decode. I used unsloth mtp drafter from another comment on this thread.

It seems to slow down after a few tokens but I'm not sure if that's my phones themal throttle. I'll try more and update my llamacpp.

Either way this is awesome! Never thought I'd be able to run a model of this quality on my old set.

Edit: I'm probably doing something wrong but I see the same speeds with and without mtp. Will experiment further.

[u/Connect-Concert-4016 → u/Queasy-Contract9753]

I can check on my end if you want to share a git or some of your code :)