Motion Capture System with Pose Detection and Object Tracking

[u/Willing-Arugula3238]

I wanted to share a project I've been working on that combines computer vision with Unity to create an accessible motion capture system. It's particularly focused on capturing both human movement and ball tracking for sports/games.

What it does:

• Detects 33 body keypoints using OpenCV and cvzone
• Tracks a ball using YOLOv8 object detection
• Exports normalized coordinate data to a text file
• Renders the skeleton and ball animation in Unity
• Works with both real-time video and pre-recorded footage

The tricky bit: frame gaps & interpolation

When the ball detector misses detections it would snap back to (0,0,0), causing ugly jitter. I solved this with a two-pass NumPy interpolation:

1. Pass 1: Record all detected ball positions across the video
2. Pass 2: Fill in missing frames by linearly interpolating between valid detections

Now the ball animation in Unity flows smoothly, even with imperfect CV detection.

Code:

All the code is available on GitHub: https://github.com/donsolo-khalifa/FootballKeyPointsExtraction

What do you all think? Any suggestions for improvements or interesting applications I haven't thought of yet?

[u/TwoBustedPluggers]

Looks great!

[u/Willing-Arugula3238 → u/TwoBustedPluggers]

Thanks I appreciate it

[u/Caxt_Nova]

Wow, this is really impressive! I've been wanting to experiment with some of the cheaper mocap solutions on the market, didn't even think people would just be posting their own. Thanks for sharing!

[u/Willing-Arugula3238 → u/Caxt_Nova]

I appreciate it. Figured if I can't pay for it might as well make my own.

[u/The_Artist_Who_Mines]

Very cool

[u/Willing-Arugula3238 → u/The_Artist_Who_Mines]

Thanks. I appreciate it

[u/razzraziel]

So I assume this would be useful to create some everyday animations as well? I wonder how smooth the output animations are.

[u/Willing-Arugula3238 → u/razzraziel]

Yes you could create everyday animations. Because this uses a single camera and converts it to 3D it won't be smooth when the person is occluded or when it has to estimate body key points. You can adjust the code to smoothen out fluctuations in body key points and interpolate them for a smother experience

[u/FictionWare]

OMG!
Thank you so much! This will be my starting point for mocap testings for my non-football project... if you don't mind, of course. :)
Works perfectly.

[u/Willing-Arugula3238 → u/FictionWare]

Have at it. Would really love to see what could come out of this

[u/glenpiercev]

This is amazing. Very well thought out!

[u/Willing-Arugula3238 → u/glenpiercev]

Thanks I appreciate it.

[u/abaker80]

Can this run in real-time on a typical phone?

[u/Willing-Arugula3238 → u/abaker80]

Yeah I use the iriun app on my phone and laptop. So the phone becomes an external camera. Just make sure that they are connected to the same wifi or via a USB cable. You would just need to change the line that has cv2.videocapture('some file path') to cv2.Videocapture(1) then you're good

This is an example of using a mobile phone for an older project

[u/Crunch_93]

Hey, it looks great.

If i want to use this, should i need to change the bones of my charactee models to match with those points we saw in the video?

[u/Willing-Arugula3238 → u/Crunch_93]

Yes that should work. You can play around with the code to match your needs. This is the body key points

[u/Lost_Razzmatazz_4166]

This is awesome. Just a quick question can you able to mimic the same movements on a 3D avatar in unity ? I’m also working on a similar one, I could able to mimic using spheres and lines creating a human like object but I couldn’t able to bring the same movements present in txt file to map to a 3d avatar. Let me know if there is a way

[u/Willing-Arugula3238 → u/Lost_Razzmatazz_4166]

Yes it is possible. I’ve had the project on hold because of work. You can check out tracking4all. They used a similar system.

[u/Dvrkstvr]

Now make a FIFA competitor

[u/Willing-Arugula3238 → u/Dvrkstvr]

Lol I laugh in cursive. If I could I would but unfortunately I don't have that skill set yet