We'd tend to do that for stories, yes; take an overall story and assign it a number of story points. We'd then try and make time estimates for each of the story's subtasks, which is where I would have trouble. Time was a bit like a currency - "We have X number of hours available in this sprint, the sprint is big enough when we've 'spent' all those hours on estimates".

My experience actually matches what you're saying, in that there was always much better discussion around the number of story points and I never felt like my story point estimates were guesses. If there were any disagreements over story points, we'd at least be able to discuss it concretely ("here's why I think this task is more complex / more simple").

For what it's worth, before I was gone our process was moving more towards a velocity focus, where story points were getting more importance than time estimates. Generally Scrum was a new thing for our company and our group was one of the first to use it; I don't think they were using estimates as a whip, at least not intending to. However the business still needed/wanted concrete dates that the group had to commit to.