Holding the door for people is so ingrained as a polite thing to do. You really have to tell people not to allow others to tailgate - or install turnstiles/gates that make it impossible. I suspect this would be particularly bad in an office with a hundred people or so - small enough that you're at least slightly familiar with most of the faces, but large enough that you wouldn't necessarily know if somebody got fired. If they're wearing their no-longer-valid badge and acting like they're meant to be there, I can't see many people stopping them.