Deep mind just released a couple of papers about this:

https://deepmind.com/blog/neural-approach-relational-reasoni...

Here is an implementation of "A simple neural network module for relational reasoning" in pytorch: https://github.com/kimhc6028/relational-networks