Richard Hammings anecdotes of famous scientists would suggest otherwise... They may not look like they were working but They were always thinking on their problem. Working on a problem doesn't need to be done sitting in your desk but it can be done while on hikes with your colleagues or locked in the attic of your room.

Their minds were constantly thinking of their problems.