Why do you say its the hardest part? Hand gesture tracking is quite hard, but tracking IR LEDs on a controller is much easier than tracking head motion as long as you have the camera setup to keep them in view.