I think this “wisdom” is not really agreed upon, and in fact I’ve seen a few works successfully leverage synthetic data (including a cool one from Sergey Levine et al recently).