We go into the office once a week, and i've found that on those days, just being in the office lets me overhear conversations about things that I can offer a solution to, or having a few people in the same area shooting shit often leads to a conversation about work and about a problem someone is trying to solve and often a solution is offered.
The biggest concept is that it's hard to "split" conversations while working remote. Either everyone is involved in the conversation (via phone or video meeting), or it's one-on-one. Getting the organic "handful of people in a group, and one conversation kind of breaks off to talk about something, then re-joins the group where someone else needed to leave for a call" kind of thing just doesn't happen. And that's where I find the most benefit from being in an office.
I spent the past two years working remote, and at this point I don't think I would ever want to do it again. Something I don't think I've ever really seen mentioned about working from home is that it can be incredibly isolating and overall mentally unhealthy. The isolation, lack of separation of work and home environment (in the physical sense), and even weird hours caused me to become very depressed and unstable.
In my case, my company was still located in-town and I even had weekly lunch/beer meetings. I pushed myself to get out and socialize with my friends, co-workers, and girlfriend several times a week. I still don't quite know what it is about working in an office with other people does that wasn't being replicated, but there's something there for me.
As far as the physical environment goes, I spent enough time reading different folks strategies. I tried many different strategies as well. I tried having normal office hours and a normal (home) office. I tried just floating about the house, working wherever I felt like. I tried working from coffee shops, bars, parks, and so forth. I tried working in blocks of time, all at once, having an inverted sleep schedule. As mentioned, I even had an actual company office I could work from. Still, I came to have a certain feeling of disdain, of wanting to escape the office at the end of a day of work. The only problem was my office was my home. You can't run from that.
I will say the convenience of having a flexible schedule is amazing. Being able to run errands as needed, or just go to the beach for a couple hours on a Tuesday because "why not" is great. However I always felt a bit weird about things, because "normal" people are in an office that I'm somehow doing something weird or wrong by being out and about during non-traditional hours. And of course among your friends and family there's always a little gentle chiding about how "You don't really work/have a real job", in a semi-jealous but not actually hostile way.
Working from home can be great, and I would enjoy the flexibility to do it a day here and there as needed. What I cannot do is work from home permanently.
That being said, the "you work from home so you should be able to handle X" really gets to me. I actually have less flexibility than it sounds like you do, it was never explicitly said, but most of us work 9-ish to 5-ish monday-ish through friday-ish.
It was something that my wife and I needed to work out on our own, because she would expect me to be able to run out and pick up prescriptions during the week, or would ask me frequently to go to the store to grab something for dinner. And I'd feel like an ass for saying no, because on some level she is right, I do have the ability, and it is easier for me, but it still impacts my day a lot to interrupt my work to run to the store.
We have found a good medium now, but it was a common argument for a while.
Doctor appointments and such may be worth breaking this rule, but picking up prescriptions or even your kids from school (assuming they have a way home) are unnecessary interruptions.
It has mostly been the same for me. I've worked 3 years remote currently.. and i loathe to give it up for my next job (which i'm hoping to land in the next week or so), i firmly believe it will be good for me.
Really, the only reasons i want to work remote currently is the checklist of savings. Saving time, gas, etcetc. Mentally, i all but feel it's worse off for me. I have a harder time focusing (i don't have a dedicated, isolated office at home), i feel disconnected, the days blur together, etcetc.
The place i'm hoping to land next is going to be half(ish) remote, which seems like a nice middleground at least. I can avoid traffic on some days, but hopefully feel less isolated.
I was recently in the office for a week. I heard so many personal discussions about things, I was able to weigh in on.
I walked away realizing I really am not contributing as much as I could, not being there in person.
What works for us is "status meetings" where we all go over a bit about what we are working on for the next few days. They seem pretty useless on paper, but it allows people to chime in with "wait, will you be touching system X", or "hey i worked with that before give me a call", and it gets i'd say 70% of the benefits of hallway conversations, but it's still not perfect.
I feel like a "sidebar" system of "not quite isolated, but not conversation-dominating like talking to the whole group" kind of in-between mode in group video chat software would be a godsend, but the UX of getting it right would be really hard.
Also, for something more programmer related, having peer code-review is really helpful as well. We don't do it as much as we should, but having a few people look over your commits, and you over theirs is a great way to get some of the benefits from that "hallway conversation" while also solving some other problems.