Ah yeah, anti-vaxxers (I'm not calling you one) grasping at straws including calling the vax mandates racist because of the low vax uptake from Black people.
If it was a policy that's based on something that the Black people can't change about them (e.g. refusal to entry to a club based on your address, with majority black areas of town being in that list), and it disadvantages them, yes that'd be racist. But in this case, the Black people (or whoever) can easily change it by getting vaxed...