It feels like there is fundamentally missing infrastructure here that is needed to make these problems go away.

Basically bots need to be (somehow) paying for the traffic they create, or prevented from creating it, or told to go away and then fined if they violate the request. No idea how to do these or even at what level in the stack they should happen, but they need to happen eventually somehow.