> But we have to deal with the cards we were dealt

Are you sure about that? You're a manager. Responding to your parent comment about managing worker's perceptions of their team members: In the case that someone wants to leave at 3 because they got there at 6, why not just explain to everyone that it is, in fact, okay (or even encouraged)?