18 karma · joined February 1, 2022
GPT-3 and all language models are just search engines that respond to queries by uncompressing the content encoded in the matrices of the model.
> Jean Leray (November 7, 1906–November 10, 1998) was confined to an officers’ prison camp (“Oflag”) in Austria for the whole of World War II. There he took up algebraic topology, and the result was a spectacular flowering of highly original ideas, ideas which have, through the usual metamorphism of history, shaped the course of mathematics in the sixty years since then.
It seems that solitude is generally conducive to creative activity for those that are somewhat positively oriented towards such activity.
For why these things are essentially mangled compression engines one can take a look at "Hopfield Networks is all you need": https://arxiv.org/abs/2008.02217. It allows representing all modern transformer networks (which is what CoPilot is using) as a bunch of hopfield networks which are essentially memory modules connected in some complicated topology to encode some data set.
Neither can math by the way and by extension, computation. There is no such thing as a computational theory of consciousness unless consciousness is redefined to be whatever is done by a Turing machine.
It's easy to forget that technology should be treated as a tool and not as an end in and of itself.