DeepMind RETRO is a model connected to a 1T token index of text, like a local search engine. So when you interact with the model, it does a search and uses that information as additional context. The boost in some tasks is so large that a 25x smaller model can beat GPT-3.
So I am not concerned for the subtle mistakes, they are going to disappear. But the model with search and code execution is a whole new beast.
Just imagine it can use any library to call up any algorithm. It can interface to web APIs on its own, calling up on the vast resources of the internet. Looks like a "Wordpress moment" for devs.
That sounds like an extraordinarily bad idea. Which does not mean that it won't happen.
It could even promise a random human user a large sum of money in their account if they would just copy and run the following code, and then supply the code.
Google search results are like 30% factual. The rest are SEO spam to sell something and people pushing their own opinion as a fact. It would need to know how to tell what is actually true or false.