Well in the base case of no debt forgiveness, the borrower suffers (though I think if it goes on long enough the exorbitant fees colleges are raking in would go down too, solving the other side of the problem).
In the case of bailout/forgiveness, there is the perception that every taxpayer...those who paid their debts for college or those that didn't even go to college, are paying for other's mistakes. So on one hand, the disinterest is on account of the issue not being seen as affecting some people while the bailout brings them into it.
i.e. the anger is "now you're making it my problem, to fix your problem."
In any case, to your point about punishing the colleges, I think is absolutely necessary. It's just that bailing out the debtors does not do that whatsoever, in fact in ingrains the moral hazard and keeps the colleges money train flowing (maybe, unless the government has some common sense to know they have to address the other end of the pipeline).