SpaceX might be able to throw something on the moon by 2024. But for a manned mission...
Just look at Crew Dragon. SpaceX started development in 2014, and the first manned flight is planned for april 2020. 6 years, and it not designed from scratch.
SpaceX likes to set unrealistic expectations. And while they get the job done (SLS should take note...), they are not doing magic. This is not the 50s anymore, they have to work within a budget, and with much more consideration given to safety.