You're right that bandwidth caps aren't my main complaint about ISPs, but overall displeasure with internet service is relevant to the bandwidth cap discussion because bandwidth caps are a feature that contributes to that displeasure.
Fairly or not, I blame ISPs for taking us from "A world where everyone pays $50/mo" to "A world where everyone pays $80/mo". Now they're trying to turn that world into a world where you have the choice between $100/mo and $150/mo. There's a fairness question to be discussed (I personally think it's not unreasonable to have society-wide, fast, affordable, unlimited internet be the default) but there's also a value decrease there that deserves just as much attention as the fairness question. That decrease (and not the fairness issue) is where my frustration is directed.