This doesn't seem noteworthy. It's called a context window for a reason--because the input is considered context.

You could train an LLM to consider the context potentially adversarial or irrelevant, and this phenomenon would go away, at the expense of the LLM sometimes considering real context to be irrelevant.

To me, this observation sounds as trite as: "randomly pressing a button while inputting a formula on your graphing calculator will occasionally make the graph look crazy." Well, yeah, you're misusing the tool.