It has nothing to do with “liberal” or “conservative” fields of studies and everything to do with reproducibility and predictivity. If psychology could make predictions to the 13th decimal place like we are able to do for the gyromagnetic ratio of the electron (and confirm via experiment), there would be no complaints.
There is also a tendency for the outcomes of psychology studies to be “oversold” in a sense. To be fair, this is more often due to the media sensationalizing the results or reading something into them that isn’t there, but you don’t see this as often in physics (well, maybe a bit with quantum mechanics and black holes).