1. No atmosphere means no weathering, so all of the moon dust is incredibly sharp and easily coats equipment when it gets kicked up. This dust quickly wears out joints and contact surfaces.
2. The Moon's low gravity and lack of atmosphere mean that you have to carry extra propellant to land.
3. Lack of easily extractable resources. It is comparatively easier to extract water and CO2 on Mars to generate methane for a return trip. A trip to the moon requires carrying all of the fuel for a return trip as well.
Nuclear-powered ships built in orbit would drop transit times significantly but I wonder if they'd significantly reduce Mars <-> Earth costs (you'd still need Starships to shuttle payloads through the atmosphere). They are probably required for mining the asteroid belt though...