joaogui1

Paxos is so mistifyingly hard that Raft was invented as part of a project to understand Paxos (and the advisor and proponent of the project was John Ousterhout, who's pretty badass). There are also I believe a few papers trying to trying to explain Paxos more clearly

When we talk about tree search we allow for backtracking, so if a node has 3 children all 3 will be explored generally, or at least a subsample of the children will be, in LLM sampling you generally pick a single token/child and then just go on with that until the end of the generation.

If DeepMind is indeed doing something similar to AlphaZero to language modelling one would expect they would generate multiple "rollouts" from the current context and then use some kind of function/network to predict which next token will lead you to the best final generation and then output that token. How to do all of that using a sensible amount of compute is what remains to be seen

I think the real comparison is "has the proportion of the population that needs to know how to take care of farm animals gone down?"

He doesn't use that much modern tech really, for example email https://www-cs-faculty.stanford.edu/~knuth/email.html

Using ChatGPT would require engaging with email and maybe more stuff that he's not inclined to

I'll link to my comment in another thread about this https://news.ycombinator.com/item?id=36023814

Before this potentially blows up, I'll say a little on what I hoped this was given its title. I saw many comments on Knuth's lack of curiosity with ChatGPT, and I wanted to give context on it.

1. Knuth's magnum opus is titled "The Art of Computer Programming" and he's the author of literate programming. Furthermore he has written a symphony for the organ, Fantasia Apocalyptica. That is to say, his perspective on programming is not only on programming as a craft, a means to an end, but as an art, crafting a beautiful and elegant solution to a problem, that is correct and efficient. Neural Networks aren't really understood, solving problems through them, especially large language models, is far from what Knuth would consider elegant, and for many problems it's also neither correct nor efficient.

2. Again on the point of correctness, Knuth's work is mainly on mathematics and algorithms, things were proofs are extremely important. To him correctness, and this includes bounding how far an heuristic is from optimal, is of supreme importance. With large language models we don't really have these bounds, there are some for neural networks but the way LLMs work different from the usual clasification or regression objectives studiend in these cases. So once again LLMs are pretty far from his interests

3. He has a lot of work to do. In the sense that both he believes he has an obligation to work on TAoCP and getting as close to finishing the first 5 volumes (he has admitted that he may not have anything interesting to say for the last 2) as possible. He takes his commitments seriously, and he does feel he has taken many detours from the original plans, detours that he has loved like TeX and metafont, but still detours

4. He has a lot of life to live. This is a man who deeply loves his wife and who has mentioned feeling guilty for focusing too much on work in the past, and who would rather focus a little bit more on his family

So from the first 2 points LLMs are in general far from Knuth's interests, and the last 2 points we get that he's also pretty busy, so it makes total sense for him to decide, "you know what? I think I'll pass".

The paper shows the equivalence for specific networks, it doesn't say every GNN (and as such transformers) are Dynamic Programmers. Also the models are explicitly trained on that task, in a regime quite different from ChatGPT. What the paper shows and the possibility of LLMs being able to reason are pretty much completely independent from each other

I don't get why the sudden fixation on time, the model is also spending a ton of compute and energy to do it

I'm not sure about any, but a lot of the persistent data structures seem to fit within that limit, and it's what's used in FP languages like Clojure. There's also some compiler research of identifying when something won't be used anymore and simply doing the mutation in-place, which can reduce even further the costs incurred by immutability

FWIW I work in LLMs and I consistently fail to do simple webdev stuff

You still have one vector per token, that's what they meant, also the fact that the vector associated with each token will ultimately be used to predict the next token, once again showing that it makes sense to talk about other tokens even though they're being transformed inside the model.

Kinda, there are GPU kernels that are sped-up by being non-deterministic, so you also gain performance with non-determinism

Someone needs to connect you to Stripe Press (maybe folks in HN can do it?)

How do you know it's pruned?

What's the name of the 3rd one?

2 issues in fact, one is TSMC being the only one capable of making these amazing chips and the second is Apple buying their entire supply. Now if any of that counts as a monopoly I don't think so

Uhm, sorry for asking, but are you sure you're not mistaking Janet and Julia?

Total Recall

It's how the software is known really (I think lots of people, including myself, don't recall the original meaning)

Python has a huge library ecosystem and the average Machine Learning programmer is not a CS geek and so prefers pythonic quasi-OOO then over FP (one of the hurdles for JAX adoption is it's functional paradigm)

You can send money through it (not sure how it's implemented on the backend) but Brazilians still don't pay for whatsapp

A* is more general than Dijkstra, as Dijkstra ia A* with the heuristic set to 0. Also it's not in the style of a psalm

In what way does AlphaZero use evolution?

This initiative is getting some pretty good practical results https://arxiv.org/abs/2209.01188, taking a look at their citations should lead you to other examples in the field

"We’ve very quickly gotten used to real-time collaborative software on the Web. We can write together in Google Docs, make interface mockups in Figma, edit computational notebooks in Observable, and we can share our documents simply by copying a URL. We’ve also gotten used to applications working across our devices. Spotify seamlessly lets you control music playing on your laptop computer from your phone, and Apple’s handoff lets you continue work when moving between devices.

However, building software that allows for collaboration, sharing of content, and distribution across devices requires the use of sophisticated frameworks and the ability to create mappings between an abstract data model and the concrete HTML of a page.

In our research lab at Aarhus University we have created Webstrates. We use Webstrates to explore what it would mean if these capabilities were an integral part of the Web, and could be leveraged using basic skills in Web development. Webstrates makes the Document Object Model (DOM) of a webpage collaboratively editable. What does that mean? It means that any changes made to the document of a webpage will be synchronized to other clients of the same page, and made persistent across reloads. We call such a webpage a webstrate (short for web substrate)."

This biggest most is high-quality data. Both their proprietary datasets (WebText, WebText2 etc), but also now their human-annotated data. Another secondary moat is their expertise with training models using PPO (their RL method), they can get results that are quite better than other labs. I say this moat is secondary because it's possible that you can get similar results with other RL algorithms (e.g. DeepMind using MPO) and because maybe you don't really need RL from Human Feedback, and just fine-tuning on instructions is enough

Generally something like "neurosymbolic"

RL is an immense field with lots of sub-areas and for the vast majority transformers are not SOTA. I believe they're only state of the art for Offline RL and even then there are some caveats

There's been work combining GA's and Architecture search for neural networks, the main keyword to search is NeuroEvolution, with NEAT being one of the first "good" algorithms for that (though scaling it up is hard)

Chinchilla is trained with substantially more tokens then Gopher, 1.4T Vs 300B