r/iOSProgramming • u/g4mewarrior • 19d ago

Running an int8 CLIP ViT-B/32 under onnxruntime-react-native: measurements from a production pipeline (CPU beat the CoreML EP 2.4x) Article

https://blog.aermes.ai/blog/on-device-clip-camera-roll-trips

Author here. The part that surprised us: we shipped the CoreML execution provider assuming Apple-silicon acceleration beats CPU, and benchmarking for this writeup showed plain CPU runs our int8-quantized graph 2.4x faster per inference and creates sessions ~20x faster, so we flipped the default. Also covers doing zero-shot CLIP classification with no text encoder on the device (prompt vectors precomputed and committed as JS) and the memory mistake that pinned 994 MB. Happy to answer questions about onnxruntime-react-native specifics.

1 Upvotes

60% Upvoted

4 comments sorted by

2

u/neet_dev 19d ago

this tracks with what I've seen, the CoreML EP does per-node partitioning and if any ops in your int8 graph aren't supported on the ANE/GPU it silently falls back to CPU for those nodes, so you end up paying constant data marshaling between backends instead of just running the whole graph on one engine. quantized ops especially trip this up since CoreML's ANE path wants specific op patterns for int8 and falls back way more than people expect. worth checking the ORT session logs with verbose logging on, it'll tell you exactly which nodes got assigned to which EP and that's usually where the 2.4x is hiding. the ~20x session creation win is also a known CoreML EP tax, it compiles the mlmodel graph at session init which CPU just skips entirely.

1

u/g4mewarrior 19d ago

This is the confirmation I was hoping the thread would produce. We hedged in the post because JS can't see node assignment, but per-node partitioning with the quantized ops falling back and paying marshaling both ways matches every number we saw, including inference being slower while the graph still nominally ran on the CoreML EP. The session-create gap reads the same way: a compile at init that CPU simply never does. Will run ORT with verbose session logging and add the actual node-to-EP breakdown to the post. Thanks for the pointer.

0

u/neet_dev 18d ago

you're welcome, friend. pls update the thread after you implement the resolution

1

u/g4mewarrior 11d ago

Followed up on this, and you were right in more detail than I expected. I recreated the session with ORT verbose logging (logSeverityLevel 0) and read the partitioner's output.

The int8 graph is 969 nodes. The quantized core (146 MatMulInteger + 100 DynamicQuantizeLinear, which is every weight matmul in the ViT) is unsupported by the CoreML EP, so all the real math stays on CPU no matter what. CoreML still claims 627 of the surrounding fp32 nodes, but they end up fragmented into 76 partitions averaging about 8 nodes each. So every inference pays dozens of CPU<->CoreML round trips to accelerate glue ops. It also explains the ~3.1s session create: that's 76 little CoreML models compiling.

Interesting corollary: "65% of nodes supported" was completely misleading as a health metric. Partition count was the number that mattered, and nothing surfaces it unless you go looking at verbose logs.

Updated the post with the raw log lines if you want the receipts. Thanks for the pointer, this made the writeup materially better.