The problem is when you tell your manager it'll take you X number of weeks to finish a task, and your manager trusts that you're working hard and that the task simply requires X weeks of work... but in reality you're barely spending any time on it every day, and could easily have done it in a few days. Meanwhile, the rest of the project is moving slowly because they need your piece to complete.
If, on the other hand, the manager's trust has no relevance here, because you are not setting your own time-work estimates, then have at it!