I'd have a similar attitude but there is something to be said for being able to go to an office when a pipe bursts in your house and it'll be unlivable for repairs for several months
You can just go to the library or a cafe, though?
Not really comparable. In my current office I have my own desk, multiple monitors, comfortable chair, a fridge, microwave, coffee machine, bathrooms that are rarely occupied, I can step away from my computer without fearing it'll get stolen, etc. I enjoy working from libraries and cafes on occasion, but it's definitely a worse UX than working from an office.
I pay for a coworking membership as a business expense to have a desk away from home or when on the road. Remote first orgs usually have a corporate account with WeWork, Regus, or both for their employees (or a coworking provider stipend).