Edit: turning of guest mode on all cast devices makes things better (less wifi dropouts) but still bad. Turning of wifi on both the android phones in my house has cleared up the wifi problem (which was really annoying as even spotify kept failing to stream with phones on wifi).
I wonder if Google employees need to test their consumer devices outside the googleplex where the internet is super fast and the industrial grade networking gear deals with such issues automagically.