The Gibbs free energy of formation for chromium oxides and chromite is much more negative than for lithium-bearing minerals so Cr compounds are thermodynamically favored to precipitate out of melts and solutions, forming minerals with high concentrations that then get pushed up by other processes. Li+, with its lone valence electron, just doesn't form strong bonds or highly stable mineral phases in comparison. On top of that the diffusion coefficients for Cr species in magmas and rocks are generally orders of magnitude lower than for Li. Cr gets locked into crystal structures early and stays put, while Li keeps migrating and diffusing in the form of water soluble minerals. There's also a whole biogeochemical cycle for Cr involving microbes that can concentrate it in sediments.