Here Jeff Dean explains why there is no central parameter server in Tensorflow: https://youtu.be/90-S1M7Ny_o?t=28m59s