Human beings act in accordance with their genetics and life experiences. They do this to such a degree that people can, at a young age, internalize maladaptive behaviors which require an enormous level of self-reflection to overcome, if they even can.

I think presuming that humans don't act "as instructed" is a bit too generous an assumption.

LLMs are capable of declining to do things. They're also capable of trying to do things and getting them subtly (or very) wrong.

A lot of this hinges on defining consciousness. I think there is more to the story than what human beings experience.

Do LLMs have an experience very similar to human beings that we would describe as consciousness? No, I don't think that.

I think the bar for consciousness is something more like experiencing a dream. You have a "mind's eye". The experience of existing, rather than nothing. Does that exist somewhere under an LLM? I don't know, but I think confidently claiming that the answer is no seems a bit of a rush judgment.