> Do biases go away completely or they just get suppressed down deep in the model's "mind"?

Bias is a human term, and couching the conversation in that context does nothing to address the issue here, because it gets into the quagmire of social context.

Let's say LLM's had taken off 15 years ago at the point system d launched. All the answers given are going to weight toward the old init system simply because there is a lack of information.

LLM's are only repeating the data they are given, and it's cheaper to remove the data after the fact than it is to try to scrub it out of the training data.