The adtech industry isn't full of idiots, it's thousands of people like you and me. Most people know how it all works, but unfortunately these companies are currently setup to make money on data and scale by any means - not by quality. The advertisers who hold all the money could do something by only spending it on good vendors - but who's going to tell them that? The agencies who sell them the ad campaigns and have inside deals with their own vendors?
Also the most annoying ads are the ones that show the most "engagement" - precisely because they are annoying. The ad that overlays the page and gets in your face will show more impressions and more time on screen than the less intrusive versions, and so these formats continue to get more money. Same with the outstream video (the ads that open up between paragraphs). Everyone wanted video, there wasn't enough inventory to go around, so these companies just created new spots for it. Unfortunately you can't escape from watching it so their stats look great and now these companies are massive.
What about data collection? Even the FCC/FTC hasn't really done much around proper disclaimers or data rights. Europe is leading the way with GDPR but that's still waiting to play out.
This is a tough industry where doing the right thing doesnt make you much money, and so you get the results you see.