Ask HN: Why not use CPUs for LLM training?

What is the current performance gap between CPU and GPU training/finetuning/inference for Large Language Models, or transformers in general? Are there papers/benchmarks showing this from the last 2 years or so?

I think that with transformers being highly parallelizable and with a large number of parameters it would make sense to use CPU RAM which is significantly cheaper and using multi-core training could offset the performance difference between GPUs? (I assume 10 cores would perform as well as 1 GPU?).

Tell me why this is a stupid idea and won't work please before I go and blow my cash on a 70 core 512GB RAM machine.