The problem is that the 1:1 conversations are a negligible weight relative to the massive 15,000 user rooms like #matrix:matrix.org - and spreading those 15,000 users over 15,000 servers rather than 1000 is going to just swap a client<->server API overload problem with an server<->server API overload...