Background subtraction works really well for these kinds of problems. You should give that a go, even better if you can make your background based on a rolling average of the image pixels.

As for the speed issue, your can often get massive gains by restricting the colour classification and blob detection to areas of the image which have changed since the previous frame.

I'm wondering whether you could apply an edge-detect filter to your input frames and combine it with the hue-based detection. I'd assume that irrespective of the ball color, an edge detect + posterize would give you a video frame with clear spots where the balls are, which should be relatively easy to find. I remember from the image analysis courses I took in college that spot detection is a very common operation in automatic analysis of medical images, and hence extensively researched. After you know where each ball is you could convert the frame to YUV and use the UV (chroma) channels to detect the ball colors.