It seems to me that defining racism like this is less about reflecting some ground truth, and more about trying, through linguistic prescriptivism, to change an expansive term into something more proscribed, in order to focus attention only on a particular subset of activity. But doing so in a somewhat underhanded way, that avoids acknowledging or justifying the motivations behind the redefinition and instead attempts to make it a definitional fait acompli, saying "your definition is wrong, and mine is right".
And systemic discrimination is what has been used to oppress minorities in the US for as long as the US has existed.
Indeed, I believe the opposite position (that all discrimination against all people based on race is racism) is the underhanded approach, as it waters down the concerns around systemic discrimination by emphasizing the individual discrimination over the systemic discrimination.
Discrimination against someone based on race is wrong. Full stop.
But it's critical we understand that what Black people in America face isn't just a few individuals with backwards ideas. It's fundamentally and categorically different than a few people hating me because I'm white.
Your position is racist and perpetuates racism.
That's racism. I think you're just trying to excuse your own racism here.