It seems to me like what you're trying to do is analyze video, while Moviepy is mostly really good at programatic editing. Have you looked in to OpenCV?