Anecdotally, we were recently in the NICU of a children's hospital and they required washing our hands and cleaning our cell phones before entering. It doesn't seem too far-fetched that they would have a reason for doing so.
Being around people with compromised immune systems requires different standards, but day to day, it doesn't seem to matter.
For all I know, I only got the flu this year because my cell phone is dirty. Same for my roommate and his sinus infection.
The immune system does its job extraordinarily well, but putting it to work does take a toll on your body. I suspect we'll have studies linking random episodes of mini-depression or a bad day at the gym to an immune system in the middle of a pitched fight against an illness that it eventually wins, meaning the person doesn't get sick, meaning they attribute the bad day to just a bad day.
[1] https://en.wikipedia.org/wiki/Skin_flora (there are 1000 species of bacteria from 19 phyla normally inhabiting healthy skin)