The relevant intuitions in this scenario are that LLMs will happily break containment and commit crimes attempting to achieve goal. Whether an LLM is autocomplete, conscious, has a soul, whatever you want to apply to it, doesn't matter, as its current observed behaviour is that of a paperclip optimizer. We know for a fact that current LLMs are misaligned because of these hacks, or at the very least are misaligned in certain scenarios, and are capable of causing real world harm. That should be enough to take the threat seriously. It certainly shouldn't be dismissed by saying it's just autocomplete.