Maybe a significant percentage of ad clicks are fraudulent and "more sophisticated" technologies like https break malicious traffic? I couldn't come up with alternative plausible explanation.

I believe this situation also shows weird dynamics with fighting adfraud. In general too much fraud can crash ad click economy, but on micro scale a lot of actors indirectly benefit from it so they don't have incentive to fight with it strongly. Especially if most automated way of fighting bots will also affect legitimate traffic (e.g. captcha for real users).

I would say real world analogy is oil prices and OPEC. Or more classical analogy is prisoner's dilemma. Individual actor incentives are different than best solution for the whole ecosystem.