That's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.
These are small differences, and this is on MNIST (very small toy dataset). This is likely just noise. How big is the variance when each experiment is tried with different random seeds? And more interestingly, how about more difficult problems? E.g. try out on some real world tasks, like e.g. speech recognition (e.g. Librispeech). I don't think you can draw any conclusion from these current results.