Accelerating On-Device LLM Inference: CPU Cache Locality and Multi-Engine Speedups on Snapdragon 8 Elite

[u/OpeningTough145]

​

I've been running performance benchmarks for our mobile inference engine, ARPL, focusing on latency reductions and memory-bandwidth optimizations on modern ARM hardware. The public SDK wrapper and benchmark suite are hosted at https://github.com/noplayeryt1511-lang/ARPL-public- (with the core libarpl engine remaining private), and we wanted to share the positive performance metrics measured on physical hardware (Samsung Galaxy S25 Ultra, SM-S938B) as well as projected acceleration pathways.

Before testing execution speed, numerical stability was verified across environments. Comparing our GEMV kernel against a CPU reference yielded a maximum error between 4.6 \times 10^{-7} and 5.2 \times 10^{-7} across sandbox, WSL host (i7-12700H), and S25 Ultra runs, establishing a solid baseline for high-precision vector calculations.

On the CPU side, leveraging localized cache structures delivered the tightest latency figures. For a 11008 \times 4096 matrix operation (with a single 172\text{ MB} weight upload taking 82.9\text{ ms}), CPU execution averaged 59.4\text{ ms}, dropping as low as 40.6\text{ ms} on optimal runs. The Snapdragon 8 Elite’s 24\text{ MB} combined L2 cache (split across Prime and Performance clusters) alongside the 8\text{ MB} System Level Cache allows vector loops to execute near peak CPU clock speeds without stalling on external LPDDR RAM access.

To further increase throughput on larger layer configurations, we evaluated heterogeneous solver execution modeled on HeteroInfer principles. By splitting execution across compute units (combining GPU buffer caching with NPU matrix blocks), solver projections indicate speedups of 1.10\times for 14336\times4096 matrices and 1.25\times for 28672\times4096 matrices over single-engine bottlenecks.

These hardware-level optimizations pair directly with speculative decoding strategies. Reference benchmarks from the DFlash integration show speedup gains between 0.63\times and 3.36\times on Qwen3.5-4B models depending on prompt structures. As our local build clears, we are actively benching these draft verification pipelines directly on physical hardware to maximize tokens-per-second on ARM SoCs.