It would have been more tensorflow-y if you did this with convolutions (1x2 and 1x3)