Colorism isn’t racism because racism isn’t about color. Color is merely an imperfect proxy for distinguishing different groups of people. (Many white people, especially Greeks or Italians, are darker than many East Asians, but that doesn’t change the racial categories.)
In some communities, colorism is a byproduct of racial categorization based on color. But in many places it’s just an aesthetic preference. Colorism in Asia and Africa long predates colonialism or contact with white people. An important thing to note is that in these societies, colorism doesn’t imply that white people are seen as the most beautiful. A person can be too light, just as they can be too tall or too skinny.