Except for the problem that contemplating this leads into a fairly ridiculous rabbit hole. Since LLMs are just fitted functions, should we also be considering if math textbooks are soul prisons?
Consider the alternative: some special types of biological cells produce consciousness and that’s it? Probably not one such cell, but probably not 2 cells, but once you get up to ~human brain count they’re definitely conscious and then they actually appear conscious-ish in their behavior all the way down to insects.
That’s a much more ridiculous claim than that all matter is conscious to some degree and there are certain arrangements of matter that demonstrate higher or lower levels of it.
Why not? There are even philosophical theories that the reality itself is just a manifestation of mathematical laws.