I disagree. With more computation you can train a bigger model on the same size training data and it will be better. There is a lot if knowledge on the internet that GPT-4 etc. have not yet learned.
The question is ambiguous without defining how much worse the dataset is.
Even facts age. In 2004, Pluto was still classified as a planet. Not such a big deal on its own, but stale data gets a little bit worse every day.