There are more permutation invariant networks. Look for Bruno or for Neural Statistician. Both of these are RNN's with the exchangeability property. This means inputs and output can be permuted with each other. If you need something with less strong constraints (but still invariant under permutation of the input), there is also Deep Sets.