Why didn't we get GPT-2 in 2005?

> IBM built a super computer for $290m. > To give a sense of progress, you can buy single GPUs today that can almost do that.

that's absolutely crazy!