That's the biggest danger with estimates, that you're being held accountable for a padded guesstimate. One way of avoiding this is to give estimates in ranges or with +/- attached:

"It will take 2 days, + 5 days, - 1 day" <-- that shows a lot of risk

"It will take 1-2 hours, + 3 hours - 0.5 hours" <-- again, showing that there's some risk

Or you push back and tell them to give you some time to form a better estimate or that you will give them an estimate update as soon as you know more.

The issue is that most software developers are not trained in how to estimate and in how to deal with these time/cost boxing scenarios. We don't attend a class like managers do on how to deal with situations like this. We have to learn as we go along.

I highly recommend Steve McConnell's Rapid Software Development, fantastic book, it has a chapter on estimation and how to deal with client's and boss's that ask for estimates.