According to the web, "The depth sensor uses a monochrome image sensor. Looking at the signals from the sensor, resolution appears to be 1200x960 pixels at a framerate of 30Hz." It would perhaps be better to be able to run it at a lower resolution and a higher framerate; maybe it's possible. Processing 35M pixels per second does sound like something that could use up a CPU cycle or two. It also sounds like something that OpenCL would be good at handling, doesn't it?
http://openkinect.org/wiki/Hardware_info