The mechanism behind engines were fully understood, any experiments with engines were reproducible and measurable. You could get an engine and create schematics by reverse engireening it.
LLMs, useful as they may be, are not that.
But the way that steam engines emergently transformed heat into work was not understood at the beginning of the Industrial Revolution. Figuring this out led to an entire new branch of physics, thermodynamics. Figuring out how big next-token predictors give rise to interesting systems is likely to lead to similarly new ideas.
If that's your rationale we have been replacing humans with atoms. But humans are also made of atoms. Nothing was ever replaced with anything.
Centuries later, we still learn new tricks for predicting and controlling the chaos of combustion, but those early engines already wrapped it up in a black box that we could more or less ignore.