Should people have the right to communicate with one another without government interference? Does encryption undermine the state in a similar way as terrorism, or is it merely an extension of the existing right of people to be (to quote the US Constitution) "secure in their persons, houses, papers, and effects from unreasonable search and seizure" ?
I concede that reasonable people can disagree with where the line should be drawn (given different states and different philosophies about the proper nature of government) but I don't think it's that absurd to insist there should be limits on what any government can know about its citizens.
Unfortunately, there have been too many demonstrated cases of governments abusing the legal limits given to them, so I have no reason to expect that greater surveillance powers would be used responsibly. When it comes to weakening encryption, they're not "smacking the child," they're smacking every child and hoping they'll hit the right one sooner or later.