Sure, go ahead and ignore what people on the moon feel because Earth is more important. It's all fun and games until people start throwing rocks.
Even if/when humans again walk on the moon it will just be a short trip and they will head back to earth (Likely China, but India is realistic, other countries that could realistically send someone to the moon don't seem very interested in trying) Their missions will be planed from Earth, and they will otherwise want to match earth time so they can contact people back on Earth.
If in the distance future we put colony on the moon, the people will drift apart from earth and so I expect the kids born on the moon will not care about earth time so much and may eventually demand a switch to moon time. OTOH, I think what they really want out of moon time (light cycles are too long to be useful humans) probably isn't what we think today.
Of course this would be very different for planets or solar systems far from Earth.
Although I guess an equilibrium will necessarily be reached after a sufficient amount of rock-throwing.