When I was young I worked at a startup where the CEO was out of the office most of the time.
One of my coworkers was irate about how he had to work hard, while the CEO was out there doing who knows what. Being young and impressionable it started to rub me the wrong way, too.
Eventually one of the older guys informed me that the CEO was going out on sales trips to potential customers, meeting with investors, and doing a number of other in-person things that the company needed. That's why he wasn't in the office.
I felt foolish for thinking that his job and my job should look the same and occur in the office.
The disgruntled coworker didn't care, though. He wanted the CEO to be butts-in-seats and staring at a monitor because that's the only type of work he understood. I get the same vibes when people sneer at VCs for stuff like "meeting people all day". That's their job.