If your goal is to get the network as small as possible, you could try using NEAT[1]. A cool example of a small network for a hard task using NEAT is MarI/O[2].

[1]: https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t...

[2]: https://www.youtube.com/watch?v=qv6UVOQ0F44