It should be noted that

1. this is nothing that should surprise anyone who has an intuition on control theory and the evolution of unconstrained markov chains

2. there appear to be relatively easy mitigations https://news.ycombinator.com/item?id=41061085 (made a separate post because it might be of independent interest to discuss)

3. you still won't get beyond the imititation game boundary without exploration & feedback, i.e. the recursive improvement doomers are, as of now, still wrong