They must have some sort of smoke tests for common operations, run in a test harness with the system prompts they force on users, right?
....Right?
What kind of Mickey mouse operation are they running over there?
....Right?
What kind of Mickey mouse operation are they running over there?
That said, I was sympathetic to the recent bug reports —- to trigger one, you’d need to have a session that waited an hour doing nothing and then very specifically tested for in-context retrieval. I don’t want to run that test, do you want to run that test?
They introduced a feature/optimization that triggered after an hour's idleness, so testing that the session continued properly afterwards seems kind of important. If nothing else, even the working-as-intended feature (context cleanup) could impact model skill in a current or future model version, so it would be well worth measuring any impact as part of the test suite.