The key for this to work is that everyone has their mic + camera turned off except when we have a scheduled meeting (ie. standups, team meetings).
Throughout the day, people often pop in the hangout by unmuting themselves to ask the team a quick question, which also works great. In practice, this doesn't cause very many interruptions because people know to ping someone 1-1 if they have a question meant for a specific person rather than the whole team.
Even on days when everyone's mic + camera is muted all day except for standup, it's still effective at creating the feeling that you aren't sitting there working alone (I guess it acts as a reminder that you really do have teammates working alongside you, albeit virtually).