1. There's no good reason to assign the sweeping and inherently-fuzzy category "white" to mean "prejudice + institutional power". Even given your (oversimplified) observation about colonialism, that's neither a necessary nor sufficient reason to apply a label that will be inherently over-generalized. It is simple historic fact that many "white" (per current definition) peoples had no role in colonialism and enjoyed no (local or global) institutional power. If this claim is mystifying to you, you may need to learn about white people outside of America or western Europe. So, the proposed semantic shortcut is inherently unjust to a portion of humanity. To the degree that this (arbitrary aesthetic) choice does harm... it is, in itself, racist.
2. To make the argument above, does not require "pretending colonialism wasn't a thing." Those are separate debates.
3. But even setting aside #1 & #2, it is, of course, far more debatable how current elites got their status. What frame are you using? The US? "The west"? The world? I would say that even the presumption that "current elites" can reasonably be described as "white" would be quite wrong in all three cases (escalatingly so, in order). But the notion that dominance was wholly established by extractive colonialism is something that could only pass for true in a deeply incestuous echo chamber. By the time colonialism even started there were already wide gaps in technological and economic development of various societies, and given how development has unfolded, it's likely the gaps we see today would exist even without colonial interference. Differential development since colonialism ended also attests to the fact that colonialism and hard force are only factors in development, not determinants. Finally, all this ignores the broad strokes of human history: across all geographies and times, human societies have discriminated against and dominated each other. Maybe occasionally, maybe perpetually, but it's basically always there. At a certain point, it became possible to do so at longer distances, at larger scale and across larger cultural experience gaps -- but human awfulness didn't start 15th century. So again, making motivated aesthetic choices about naming descriptions of social systems and types of interpersonal behavior after an identity group that does not actually map to them is aptly described as racist. Not under the new, stipulative definition, which runs afoul of all these issues, but under the older and more useful definition that could be summarized as "harming or hating people on the basis of their race/skin color".