> nor are they...able to mutate themselves to gain new information on the fly

See "Self-Adapting Language Models" from a group out of MIT recently which really gets at exactly that.

https://jyopari.github.io/posts/seal