> I feel like you'd miss out on so much collaborative effort and serendipitous & spontaneous conversations with colleagues that solve problems out of the blue

This happens for sure

In my experience the ratio of beneficial to detrimental serendipitous and/or spontaneous conversations that result in any benefit is extremely low, so low in fact as to not being worth it. The problem is that people tend to remember the useful one conversation and forget the other 100.

At my current job, never have I ever been involved in anything that has resulted in a solved problem from a casual conversation, yet I was regularly interrupted by other people before I started working from home.