> If I say "it's too chilly in here" in a house with only lights, it will turn them on as a way of "warming things up".

Thanks for the example that's interesting.

FWIW, this is pretty much what has been described as "waluigi" effect a bit extended: in a text you'll find on the internet, if some information at the beginning is mentioned, it WILL be relevant somewhere at some point later in that text. So an auto-completion algorithm will use all the information that has been given in the prompt. In your example it puts it in an even weirder situation where the model the overall model information (the lights, and that you're cold and nothing else), and it must generate a response. It would be a fun psychological study to look at, but I'm pretty sure even humans would do that in that situation (assuming they realize that lights may indeed produce a little bit of wattage of heat)