I've implemented something similar with Gstreamer for my own work. Gstreamer has some of the most disorganized documentation known to man, though, and if something doesn't work, you have to do elaborate googling first to figure out what's wrong (error messages aren't generally helpful) and then experiment with how to fix it. There are also 2 distinct incompatible versions of it in the wild: 0.10 and 1.0.

Needless to say, I strongly suspect very few researchers have enough dev chops to meaningfully use Gstreamer. When it does work, though, it's pretty great, and it offers reasonable integration with GTK and can be made to work with Cairo. I got it to the point where it works for what I need to do, but I wouldn't want to release the code on GitHub because it's relatively high maintenance.

Interestingly I ended up with a similar layout: a set of frame producers (video file, webcam, sequence of jpegs), 2 video sinks (gtk and mp4), and the code to draw stuff in between (using PyCairo for vector and PIL for raster).