I had the same feeling as you recently, until I read a site posted here a few weeks ago. I can't find it now, but it was a personal page where someone described in increasing levels of severity what can be done over an unsecured HTTP connection even if there is no private data transmitted over it. Can someone help?
Because unencrypted HTTP allows attackers to MITM the traffic, they can do things like put ads on your site, inject malware or bitcoin miners, or just replace your site completely. None of those affect you, the site owner, directly, but ìt subverts the purpose of your site and lowers the value and trustworthiness of your site to your visitors.