I manage people, am managed, and work strange hours.
In the scenario you describe, you have set of inconsistent requirements:
1. A developer that works strange hours
2. A need to get everybody in the same room
3. The "perception problem", a.k.a. snide remarks from the peanut gallery.
The first thing to do is to be a good technical manager and admit that you do, in fact, have inconsistent requirements. Something on this list must change. You cannot satisfy all three.
The next thing to look at is what is the best (or least-bad) requirement to change.
The simplest option is to fire the developer. I don't actually recommend that, but it is simple. Embedded in that statement is the implicit assumption that you will never again hire someone who works odd hours, and in my admittedly biased experience, people who work odd hours tend to be good, so there is a hidden cost to that scenario.
A more complicated option is to figure out how to eliminate or reduce your dependence on in-person meetings. For example, having discussions via email or on GitHub. I would probably argue that if your software developers are having meetings more than 0.5 times per day, they are not being productive to start with, so there are reasons independent of this particular developer to study that problem. For more on this strategy, I would recommend looking at the GitHub model. [1]
Finally, to address the third issue, you need a performance review mechanism that isn't attendance-based. For example, consider mechanisms that are feature-based or deliverable-based. Attendance-based performance is bad for a lot of reasons independent of any particular developer's situation--it motivates working hard instead of working smart, and so on.
The thing to see here is that you're either going to have to change the environment or the developers you place in it. Personally, I would recommend the former, but in any case, something's going to move. The advantage of changing something now is that you get to decide what to change. If you let the situation run its course, it will change in a manner and at a time that will be less convenient.
[1] http://zachholman.com/posts/how-github-works-asynchronous/