There are some highly talented developers with great interpersonal skills who just happen to have e.g. delayed sleep phase. They might leave "early" to have a normal evening with family or run errands, but then work more at night while they're awake with time to burn.
Of course, that doesn't accommodate people with severe delayed phase or whatnot, but often that stuff can either be worked out either on a team-by-team basis or arranged as part of the hire. Or maybe you only enforce the "scheduled" part, and put a "24 hours in advance minimum" on it to keep things humane. The important thing is to get team expectations in place.
What doesn't really work is "around whenever they feel like it," assuming there's any inter-team dependency at all. Even with an email SLA, that gets obstructive pretty quickly.
It has the virtue of assuring that we are all available in person to one another, which is actually pretty valuable.
In my own experience, those distractions are mostly due to open-plan and cubicle-farm offices. Given a proper one-man office, with a door, it's very easy to be productive.