I don't really think that's what I said. People who show time and again that they are ineffective at their jobs should not be there. I'm certainly not advocating firing someone for a single mistake or even for many mistakes as long as it's not the same one over and over again. Hell, making mistakes means the person is actually working and trying to get things done. That's not the kind of worker I'm talking about here. I'm talking about the worker who does little and consistently gets in the way of everyone else who is trying to push the boulder forward.
I also blame management because they need to be proactive in explaining to people that they are not cutting it and need to step up or will eventually be let go. I know if I was not meeting the needs of my job, I hope my manager would tell me.
it often is significantly more cost effective for a large company to keep them than to go through the expenses entailed in the hiring and training processes.
The cost effective you're thinking about is likely only in the short term. If you keep someone who is mediocre and consistently slows down other teams, and burdens other team members to make up their slack, then long term it is cost effective to let them go and get a quality person.
Rockstars may be 10x more productive individually, but anti-rockstars can be a huge minus on the productivity of the entire team. IMHO, it's much more important to not hire (or keep around) mediocrity than it is to hire rockstarts.