I completely agree with you (although wasn't of course even aware of all the details, the company grew so fast during the last years). It's all about ownership and communication, which are somewhat easier with someone next to you (with obvious caveats as well), but it's far from black and white. Actually, remote in some ways makes it more explicit: you have to write stuff down, and thus, when you hire new people (or people leave) it's more than tacit knowledge.
But yeah, it boils down to individual in the vast majority of cases, much more than whether the individual sits next to you or not.
“did you forget our Kiwi friend, our first FTE? he's still here, rocking it.” Of course not, but he was a remote contractor as well through the early years :-)
Maybe that was one of the things that helped us to begin with: since all of the developers were remote originally, you just had to have the communication in place. I would imagine it's trickier if there is an in-house team originally and then remote devs added to that.
Miss you too!