I have only just started looking at the problem and (coincidentally) found this thread on the front page of HN so I couldn't resist asking the question!
I will look into OpenCV, is it the go-to libary for these kind of problems?
Yes.
Pretty much, but it has some warts (the matrix library is notably bad imo).
How easy the task is depends on the nature of your game. If your players have a red outline and that's the only red in the scene, no big deal.