> I feel awkward when people talk about sports
You don't like talking about sports, but it's a topic of conversation, because by and large many people in the US like sports. That is different than assuming you must be an athlete because of the color of your skin (racism).
> Even as a middle-aged guy, I sometimes feel that the seat next to me is the one people are avoiding.
For whatever reason, sometimes people don't take the empty seat next to you. That reason isn't because the color of your skin.
> And I've definitely seen moms grab their little girl and march away when I smile at them
Yes, this is something all men should be aware of. Don't act creepy to little kids. Parents typically do not spirit away their kids when adults simply smile at their children. Something other than your smile is causing that parent.
> I'm sad that we're in a position where racial bias seems a reasonable default explanation for such things.
At the risk of telling you how to feel, how about feeling sad that people are actually victims of racism.
> At the end of the day, I think the only way to get past this is for everyone to be color blind. We need to guard against acting out of bias.
I disagree with the first part, but agree with the second.
A persons skin color is part of their identity, and we should recognize that, but not use that as a way to group, categorize, or otherwise use as the basis for how we behave towards that individual. We should also be aware that it's human nature to feel defensive against those that are different from us, and to not give into that behavior.