If more people did this, instead of waiting on hold for hours to fight with customer service for your own money back, the ISP would likely lose their merchant account for their fraudulent activity.
Edit: one other note when cancelling, make sure to take down the date, time and operator name & number during your interactions and note what was said. When I've had trouble with managing accounts, referencing that information strengthened my case dramatically.
I have a Google Doc that I keep for all these notes. While I'm on hold, I open up the doc and get it ready. Be clear that you're taking notes and they'll take the conversation more seriously too.