Didn't apply to YC but interesting idea. I thought of the same idea but there was one fundamental problem I couldn't solve. That is, the text on the side is too distracting. I hope you can think of some ways to solve that problem!
There's also user interface ways to make things less jarring.
Here is an idea you can play with: wikipedia for videos. So sometimes after you watched a video, you want to know EVERYTHING about it, and usually the comment are crap for factual information about the video. What you can do with that technology is allow users to learn more about the video when he or she pauses or when the video ends. Lots of possibilities here =).