Saturday, January 14, 2006

motion capture in python

You know those suits with the ping-pong balls attached to it? The kind they use in motion capture? I thought it would be really cool if I could build my own. Then I'd be able to film my kickboxing and analyze it afterwards.

The current state-of-the-art seems to be tracking a magnetic field with the sensors on the suit. Unfortunately I don't have the resources to do that. So I'm going with the older way: visually tracking the markers.

I capture frames using my cheap webcam and the VideoCapture Python library. Others have used it for simple motion detection but not motion tracking. I decided to track the motion of red objects. Why red? Well, it's one of the least common colours in my office and is usually concentrated to a small area (bottle caps, mugs, books).

To get my hands on the red pixels I can't simply extract the R from RGB images. If a colour has a red component it could also be white, yellow, magenta... So you take the R and subtract the G and B. Ignoring values below zero you now have a gauge for the "redness" of a pixel.

(Extra points: Correct me if I'm wrong, but I believe "R minus G minus B" will give you a "redness" plane that intersects the xy chromaticity diagram (see sRGB color space) as a line running from blue to green along the edge of the sRGB gamut. The triangular plane segment within the gamut reaches it's "peak" at the maximum red representable by sRGB.)

This is a simple way to find the maximum red point on the image. I drew a crosshair over that point for clarity. With the series of images captured by the webcam I constructed an animated gif. So the system tracks a red object. Cool.

However, what I want to do is track more than one red object. I did this by forming clusters of red dots above certain intensity.

How do you form clusters? Well I used the K-means algorithm, conveniently included with SciPy. Et viola! It seems to do a pretty good job of it.

Not bad for a days work. Only 150 lines of Python. Of course it is at this point that my fear is confirmed: My webcam doesn't seem up to the task of tracking the speed of my movements. So before I jump around with a bunch of red markers on my body, I'll need a better camera.

Edit: You can download a rough version of the code here: motion_capture.zip. Parts are commented out so you don't need a webcam. You will need Python, SciPy and PIL however. I hope this works...

Another Edit: Here's an alternative I wrote (vr_tracker2.zip) that uses numarray, Pycluster and PIL instead. I'm starting to dislike SciPy. If there are enough requests I think I'll have to do another write-up.

11 comments:

struktur said...

And you complain about the resolution of MY crappy phone camera!!! Maybe you should remove that huge block of timber out of your own eye before you poke a stick in mine. ;>

gumuz said...

that looks pretty cool, sooooo where's the code? :)

Janto said...

Your wish is my command :)

Anonymous said...

Oh man, we don't get to see the kickboxing videos?

Janto said...

The project was placed on hold. I need to improve on the update speed. So: no, you don't get to see any videos :(

phillip said...

I really like this code.
I have a bunch of ideas for programs that could incorporate this code.
the only problem is, I can no longer get it to work on my computer. I installed all the extra stuff (scipy, PIL, etc) but I get this error:
File "capture.py", line 12, in ?
from scipy.cluster import vq
ImportError: No module named cluster

I was wondering you have any insight to this.

Janto said...

I uploaded an alternative (see post) that removes the SciPy dependancy. You'll have to use Pycluster, however.

I'd love to see what you can do with this basic idea.

phillip said...

I don't know what is up, but I still can't get it to work. but that is fine.
you should check out my motion track script. It is a basic idea, let me know what you think. (phillipmates[at]gmail.com)
example video

swatch said...

my post above has broken links. python webcam color tracking (with a link to this post)

Anonymous said...

i instaled caapture.py and it says i need module named scipy....where can i download it and after i download it do i need anything elese for it to run??? my e-mail is vojinovicmiljan@yahoo.com
send me a mail please...
sory for bad english :)))

Janto said...

You can download scipy from http://www.scipy.org/. I don't think you need anything else for it to run.

It's been a long time since I looked at this code. These days I use the python bindings to OpenCV for my computer vision needs.