Relative to compensation for the developers and researchers working on projects like these this looks a small component.
GPT was rumored to cost in the millions, perhaps the hours estimate is conservative?
And presumably one should also factor in the cost of experimentation before a good setting for the final training is found.
How feasible is it to evaluate the quality of chosen parameters/settings without going through an extensive training? I mean <BIGNUM> GPU hours to discover your results are meh, jesus.
(Small timer with absolutely no idea of practical ML here)