Really love this. I agree with some of the comments here that adding encouragement to keep track of secret plans would be interesting— mostly from an alignment check angle.

One thing I thought of reading logs is that as we know ordering matters to llms. Could you run some analysis on how often “p1” wins vs “p8”? I think this should likely go into your Truescore Bayesian.

My follow up thought is that it would be interesting to let llms choose a name at the beginning; another angle for communication and levels the playing field a bit away from a number.