The author works at Anthropic, so probably idle capacity in Anthropic's datacenters.
If so, then the class of GPU used here may be significantly higher than mere mortals generally have access to simply due to cost.
Obviously nation states will likely have significantly more resources than this, but this is not script kiddie levels of GPUs.
"idle capacity" - aka subtle advertisement
If you look at the numbers, he managed about 50% utilization of those 2048 GPUs over 10 days, so he was probably sneaking in factoring work between training runs.