At "remote friendly" companies I've worked at in the past, most people were in the office at least 80% of the time- and because of that, there was sort of a general mindset that if someone wasn't in the office, they "weren't there"... which lead to the same sorta assumption that you mentioned.
On the other hand, at the place I'm at now, I would guess that less than half the company are in the office on any given day. This leads to an amusing inversion of what you're talking about, because if a normally-remote colleague needs to come into the office, it usually means they won't get as much done that day, because of the overhead of travel time! :P
At my current company, when I work from home, I'm expected to be working and available from 9 to 5. At a previous company, you only need to be available for a specific time interval during the day, otherwise you can work anytime you want.