Number 1:
While it is milder, it will still kill a lot of people. Yes they're mostly elderly, or they are immunocompromised and have other health issues already, but for those people it still means premature death, and for their families and friends it's still a loss of a loved one.
Number 2:
Long Covid is still not understood. The prevalence, impact and cause is too nebulous. It can manifest itself 6 to 12 months after first infection, so there's not been data about Omicron for it yet, since we're only starting to approach some people recovering on the 6 month mark. There's conflicting predictions of it affecting 5% to 30% of everyone (all age) who gets COVID. And it looks like sometimes the effect are dehabilitating, where people can't even go to work. So while it's nice Omicron doesn't kill you, it isn't clear if it will make you unhealthy and suffer from chronic problems for the rest of your life, and possibly shorten your lifespan. We also don't know if the chances of long COVID exist each time you get Covid, so even if it were 5% but you get it year over year, and each year it's 5% chance, it might be a matter of time till we all have long COVID.
Number 3:
Healthcare capacity and supplies. Even though the percentages of cases that need medical care is smaller, if too many people get it at the same.time, the total number of people needing medical care could be quite large and breach the healthcare capacity.
Number 4:
Cost on the economy. When people say it's mild, it's often still like getting the flu, which means you're off work for 3 to 10 days. If everyone gets it once or twice a year, and if people get it all at the same time, it means supply chain issues, delayed business, it just puts pressure everywhere on the economy.
Omicron is slightly less deadly than Delta, but it's much more transmissible. That, combined with a lack of preventive measures, made this winter's Omicron wave the second most deadly wave in the US since the pandemic began.
If this was even remotely true everyone would be doing Covid parties now like they've done measles parties decades ago.
It's not. Getting Omicron doesn't give you any herd immunity at all, let alone against "the entire coronavirus family". That's beyond ludicrous.