In some case there is direct harm caused by the posts. Pizzagate was an example which led to a man turning up to a pizza shop with a gun based on the fake story and the business owners received death threats. Not everything is as easy as just unfollowing.

https://en.m.wikipedia.org/wiki/Pizzagate_conspiracy_theory

When it comes to hate speech, incitment to do harm or lies about elections a subset of people unfollowing doesn't mean there still won't be harmful consequences.