You are not being ridiculous and don't let people hate on you.
1. I see expert programming similar to Mathematics.[1] If you can do two 2-3 hour spurts that produces a ton, it can be draining and you need time to recharge. It's generally better to do a smaller amount of excellent work when programming than a lot of easier busy work. One star feature or optimization or whatever built right can have a much larger impact than hours of crap work in the wrong direction.
2. In this day in age of material production, why someone with a valuable skill can't earn a decent living with less than 40 hours a week work is kind of pathetic. Don't feel bad if that is what you want. If you are providing value to the company, that is the primary thing, not how many hours your butt is in a chair. Also, this relates to #4.
3. Time in the office can be valuable. I wouldn't dismiss it entirely. It makes communication a ton easier and can build a better sense of team in people, but remote should be acceptable too (at least part time). Inability to handle remote probably signals a weak team.
4. Most founders are closer to the VCs than to you. Know what the cap table looks like and what your reasonable exits look like. If you are as talented as you say, make sure you aren't getting screwed. First hire should at least be in the 2-3 percent range and that can still be a screw job if you are providing so much value as to build out their entire tech stack. Remember, most likely exit is none, and then most likely is middle of the road (Founders make $5-10M each, you make what? At 1/20 = $250-500k) Over four years, really as much as you are probably giving up if you worked for a big company (including stock and bonus and more salary). So 50% chance nothing, 40% chance same as you would have made +/- a bit, 10% chance you make double, triple? .1-1% chance FU money. The founders will probably be flying lear jets before you buy a house. There's a lot of fucking stupid people getting a lot of money, so keep this in mind. [2, 3] Just because people are founders doesn't mean they are competent, just means a slightly above average chance, like going to a good school.
The fact their previous team of developers left is a red flag. The fact that other employees have issues in a small team is a red flag. "Corporate" founders with other people building their tech? Your biggest issue may not be remote but the possible turd of a startup you are working at. 5 years? Startup years are like dog years, how is the company still alive and not growing. Sounds like a small lifestyle business calling itself a startup.
While other people are trashing your arogance, I sympathize. Keep moving until you find a place that feels right, where you respect the people you are working with, and where you feel you are valued and earning or learning or living the way you want. It's your life, don't let other people tell you how to live it.
[1] http://calnewport.com/blog/2012/08/31/henri-poincares-four-h...
[2] http://nextshark.com/8-reasons-clinkles-lucas-duplan-is-the-...
[3] https://www.quora.com/Why-do-so-many-people-want-to-see-Clin...