This ignores Planck's energy-frequency relation. Things like photons are affected by gravity as well.