Hacker News Reader

They aren't good at Zork[1] and neither at newer and/or more obscure text adventures[2].

[1]: https://www.lowimpactfruit.com/p/zork-bench-an-llm-reasoning...

[2]: https://entropicthoughts.com/evaluating-llms-playing-text-ad...

Nethack has been widely used to test reinforcement learning agents, starting from at least 2020; there was a Nethack challenge at NeurIPS 2021. https://nethackchallenge.com/report.html

For a more recent test, see https://kenforthewin.github.io/blog/posts/nethack-agent/ .

To be honest, Zork at times makes precious little sense: you are supposed to die over and over before you figure stuff out. For instance, you have to grab the endless-light-source treasure very early on, or you mathematically cannot win. And the game does not spell anything out for you, you just have to "get it" by watching closely at how/why you die.

This is a tall order for an LLM: it needs a lot of context but most of the context will be just noise.