In real life people are just trying to get through their day, and if you're decent and polite to people they will usually appreciate and reciprocate it. Black people aren't walking around hunting for racists. Nobody is walking around looking to accuse others of things in real life. The guy has a job to do and he wants to do it. He appreciates people who are nice and friendly, and doesn't appreciate people who are nasty and mean, just like anyone with a job.
You are always vulnerable to accusations, no matter what you do, no matter who you talk to. You could be accused of anything at any time. If somebody wants to be a jerk and make your day worse they can do that. But the vast majority of people don't. Walking into an interaction with someone else, almost everyone wants it to be polite and friendly. All you can do is try to do your best to treat everyone like a person and be decent. Don't let toxic Internet culture stop you from developing friendships in the real world. The environment is not hostile, even if it seems like it is because us tech people are extremely online. If something bad happens, shrug it off as the rare one-off crazy event that it is and move on.