It's not actually hard to unfollow people on most platforms, though.
Twitter shows the Unfollow button when I hover over someone's name in my feed. It's one click to unfollow and I don't even have to leave the page to do it.
I think the real problem is a type of FOMO: People are afraid to remove any connections they've established, lest they miss something important.