Inevitably when you go out with co-workers or industry colleagues at a conference, you will end up talking about work. That is the one thing you for sure have in common (while someone may not even like sports), and it's an easy topic to fall back to.
But FWIW, I also think the small talk and getting-to-know your colleagues is valuable. Some of my deepest and longest-term friendships have come from former co-workers, former classmates, etc. While it's possible to cultivate a friendship via remote work, it's a lot harder.
Coworkers often share birthdays, weddings, random trips, and enriching experiences together that are not very easy at all to replicate over fully remote work.
I've noticed that as my career has gotten more remote, it's harder to even count on making new adult friendships into my 30s. I'm like a stranger in my own town, unless I really intentionally take the time to join clubs/groups outside of work. That's not a bad idea on its own, but it does take intentional effort and some degree of consistency that just sort of came automatically with working in an office. And some young people with less overall confidence / social experience may not even know how to go about that.
I think all of that def takes a mental toll. We're social creatures, even the introverts among us.