CNTK user & contributor here. CNTK overall has very low framework overhead and has tensors with dynamic axes as first-level citizens. This means that sequences can be expressed without needing to do padding, sorting of the input data, or any other workarounds, and can be packed automatically by the toolkit in an optimal way. In particular, while it is laying out the rectangular structure it uses to traverse multiple RNNs of a minibatch in parallel, it fits shorter sequences into the holes and can reset the RNN state for these sequences while it is traversing this structure. This makes CNTK especially suitable for expressing RNN models (for CNNs many of the calls are just forwarded to CuDNN, so the difference might be much lower).

As for distribution, a) it has an extremely simple way to run data parallelism (for CNTK 1 it was just using MPI and starting the worker with a few extra options. I think CNTK 2 will add this in a week or so to the Python bindings), b) it has 1-bit SGD and more recently BlockMomentum, which are just dead simple methods to use for distributing the gradients, and they just work. All of these are open source (though 1-Bit SGD and BlockMomentum are patented).