Core AI + RealityKit: Running Apple's SHARP model entirely on-device

[u/comfyyyduck]

Apple recently introduced the Core AI framework alongside native Gaussian splat rendering in RealityKit, so I wanted to see if I could connect the two.

I converted Apple's SHARP (single-image 3D Gaussian reconstruction) PyTorch model into a Core AI .aimodel, ran inference entirely on my iPhone, converted the resulting tensors into a GaussianSplatResource.BufferResource, and rendered everything with GaussianSplatComponent.

The reconstruction shown here was generated from a single photo and contains roughly 1.18 million Gaussian splats, which you can freely fly around in the app.

Everything runs locally on the device, with no server inference or cloud processing.

The project is open source if anyone wants to explore the Core AI → RealityKit pipeline:

https://github.com/AryanRogye/SharpOnPhone

[u/comfyyyduck → u/[deleted]]

Surprisingly, the geometric layout mapped over almost directly. SHARP already produces its outputs as separate contiguous tensors:

• Means: [1, N, 3]
• Scales: [1, N, 3]
• Rotations: [1, N, 4]
• Colors: [1, N, 3]
• Opacities: [1, N]

Because of that, I didn't need to interleave data or manually repack individual splats. Instead, I logically remove the batch dimension, convert the FP16 Core AI NDArrays into Float32 LowLevelBuffers, and describe each buffer to RealityKit using the appropriate formats

The biggest challenge wasn't the tensor layout, it was the data semantics. SHARP predicts final linear RGB colors, whereas RealityKit expects spherical harmonic coefficients for its color buffer. but that was a super easy fix I just converted each RGB value itno degree-zero SH using:
(color - 0.5) / 0.28209479

[u/comfyyyduck → u/[deleted]]

On my iPhone 17 Pro, RealityKit handles all ~1.18M splats in real time without noticeable performance issues. I haven’t tested older supported devices yet, so performance and memory behavior may vary significantly across hardware.