I'm not sure I agree. If P-value is really low then it implies either the existing data set is bad or your new data set is bad. Either way you haven't used data to get to a confident decision.

Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than using my intuition or not, since we really don't have a good understanding of the baseline.