Slowness is just the beginning -- the end-goal is making the internet into multiple competing walled gardens where users are treated as silos that require permission to access. The global reach of the internet is at risk here.
Imagine if Netflix wasn't accessible to Comcast users from the beginning because they wouldn't pay the toll. Would it have thrived and grown as quickly as it did, or would it have died in it's infancy?
What about Skype? Skype stepped on the toes of incumbent ISPs' long-distance revenue streams, I wouldn't put it past AT&T to purposefully degrade the quality of Skype calls or even outright deny them from happening. Prior to being bought out by Microsoft, would they have had the revenue to pay for access to users? Would Microsoft have even bought Skype at all?