I hit upon a similar idea back in about 2000... the key at the time was video encoding hardware that was designed to incorporate MPEG compression. Pulling motion vectors from the encoded stream could have given realtime optical flow way back then. Unfortunately I was nowhere good enough as a programmer to take advantage of it for real projects. A team at MIT had a paper about this that I was able to find.

Had a page about it on robots.net, but it seems to be down now