I also had that idea, and I made a crude proof of concept: https://github.com/iver56/cnn-clock