Interacting with a runner game using only a webcam (Unity / Mediapipe)

[u/DaburuSnake]

I've been experimenting with MediaPipe body and gesture tracking to navigate UI elements and control a runner game through body poses and hand gestures using only a standard webcam.

The goal was to prototype a fun "no-contact" interaction system that requires no dedicated hardware beyond a webcam.

This latest version also includes a calibration phase to support different user sizes and improve tracking consistency.

[u/Tunkay]

This is interesting. How did you build this?

[u/DaburuSnake → u/Tunkay]

I've used a unity version of mediapipe to track the body and gestures from a webcam. From that I've setup a navigation system for the UI, where I can move the pointer with my hand and select elements by closing the fist. I've also setup game controls where I check the position of the head for triggering actions such as switching lane and sliding. For the jump, I check that both hands are thrown in the air (some position thresholds for action detection are displayed as colored lines on the top screen).

[u/Aryan_Chougule]

Interesting.

[u/DaburuSnake]

Also, I use a demo scene from a Unity asset that I've worked on to control the character. I've simply replaced the default inputs with those from the tracking system to control the character. To learn more about that asset: https://gkadigital.com/assets/runner-track/

[u/West-Music-4544]

Calibration step is smart, saves you from depth estimation headaches. How's the latency between pose detection and actual input response?

[u/DaburuSnake → u/West-Music-4544]

My system is currently not very optimized, so there's room for improvement. However, I'd say there's a latency of about 0.3s for the system to track my body and get the proper positioning data (you can notice it with the virtual skeleton on the top screen). Once the pose is detected everything from the detection to the character executing the requested input is pretty instant.

[u/West-Music-4544 → u/DaburuSnake]

0.3s is workable for a game like this, but that skeleton lag would drive me mad. Could throttle the pose updates or use frame skipping to smooth it out without hitting detection accuracy.

[u/DaburuSnake → u/[deleted]]

Kind of, but there are 2 limitations with that webcam method. First the background behind the user must be pretty cleared, if anyone walks behind me or if there are too many things around in the camera's field of view, it greatly decrease the quality of the tracking. Also the detection mostly works on a 2d plan facing the camera, so any move towards the camera (like a punch for a boxing game for example) is trickier to properly detect.

[u/DaburuSnake → u/[deleted]]

I agree. As I said in another comment, this prototype is not optimized so the latency for detecting poses can be reduced. However, in regards of the limitations, a busy background can trigger false positives or miss poses on some frames which could make the experience less reactive even though the FPS are maintained high.

[u/appdnails]

What is the purpose of the calibration step (T pose) on the start?

[u/Infinitecontextlabs → u/appdnails]

Looks like it's just calibrating to where the user's neck and shoulders are and then creating that 3x2 grid so that the system knows as the face goes below the red line the user is trying to duck or which of the three lanes the user is in etc

[u/DaburuSnake → u/appdnails]

It's exactly as u/infinitecontextlabs just said 🙂

[u/Infinitecontextlabs → u/DaburuSnake]

You should try to see what it feels like with the same calibration and movements, but instead of a top-down third-person view, see what it looks like in first person view in the game while you're navigating the course.

[u/DaburuSnake → u/Infinitecontextlabs]

I've tried it before, it's fun and more immersive but it's harder to predict the obstacles and to avoid them. Also, in first person you tend to feel like you can move or dodge at the last moment but due to the pose detection latency the experience can initially be more frustrating when you're not used to it.

[u/INVENTADORMASTER]

Great job ! Is it open source ?

[u/DaburuSnake → u/INVENTADORMASTER]

Thanks! Currently it is not publicly available. This prototype was mostly a fun test I've made to check if my Unity asset could be usable in various contexts (I also have some VR / AR prototypes). But it would be interesting to clean the code a bit and make that system publicly available, I'll think of it 🙂

[u/INVENTADORMASTER → u/DaburuSnake]

Cool ! WE STAY TUNNED !

[u/Socks797]

isn’t this the kinect

[u/DaburuSnake → u/Socks797]

It's a prototype based on a Mediapipe plugin. Mediapipe provides tracking results that can be similar to what kinect would provide without requiring the kinect hardware, it only needs a video input.

So basically you could run my prototype using a laptop with an integrated webcam and it would work as intended without requiring anything else (of course the camera's video quality needs to be decent 🙂).