I see where you're coming from, but that argument opens up a pretty big loophole to use any copyrighted material however you please. There's a big difference (in terms of both effort and value) between a documentary and some user-generated annotations of varying quality.
I'm not clear what you mean by "at best half of the work being used" regarding lyrics, since the full lyrics are reproduced.