xianshou

If Waymo is truly as much better than Cruise as the statistics claim, this is a perfect opportunity to release the full video of the incident. Most collisions are avoidable, but a few are the result of a perfect storm of conditions that would be difficult to navigate for even the most skilled human driver. To know whether the truck's occlusion of the cyclist is one of those cases requires the actual footage.

Last May, Karpathy mentioned that the next step in LLM performance would likely take inspiration from AlphaGo's Monte Carlo-style tree traversal to enable deeper and more self-correcting reasoning:

https://youtu.be/bZQun8Y4L2A?si=RgD7NWfwDdh0bklK&t=1630

But if this approach holds up, it suggests that the most valuable part of the AlphaGo project that applies to LLM development is in fact reinforcement learning through self-play. Why not create a leaderboard of reward-generating language models that constantly "play" against each other, where model selection frequency is based on ELO, and update on the results? What if the Open LLM leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...) evolved into a constantly improving pool of such models? This also alleviates data scaling issues by providing a diverse and continuously changing distribution of new training inputs.

An excellent example of an anti-shibboleth, also known as a "frisco":

https://allthingslinguistic.com/post/45448571632/the-opposit...

These signal words act as a silent, passive test of competence in that once you hear them, you can be assured of the speaker's ignorance and discount their opinions accordingly.

More excellent resources from Jay and others:

The Illustrated Transformer - http://jalammar.github.io/illustrated-transformer/

Beyond the Illustrated Transformer - https://news.ycombinator.com/item?id=35712334

LLM Visualization - https://bbycroft.net/llm

Plagiarism detection will be extremely difficult to create any systematic standard for given the endless creativity of LLMs in generating variations.

I foresee a system of enforcement via name and shame, a la Sports Illustrated.

Morality and legality aside, there's a substantive difference between use of content and use of a model. Pretraining a GPT 4-class model from raw data requires trillions of tokens and millions of dollars in compute, whereas distilling a model using GPT 4's output requires orders of magnitude less data. Add to that the fact that OpenAI is probably subsidizing compute at their current per-token cost, and it's clearly unsustainable.

The morality of training on internet-scale text data is another discussion, but I would point out that this has been standard practice since the advent of the internet, both for training smaller models and for fueling large tech companies such as Google. Broadly speaking, there is nothing wrong with mere consumption. What gets both morally and legally more complex is production - how much are you allowed to synthesize from the training data? And that is a fair question.

Not a complete answer, but here are the most helpful resources for understanding transformer basics in particular:

Original transformer paper: https://arxiv.org/abs/1706.03762

Illustrated transformer: http://jalammar.github.io/illustrated-transformer/

Transformer visualization: https://bbycroft.net/llm

minGPT (Karpathy): https://github.com/karpathy/minGPT

---

Next, some foundational textbooks for general ML and deep learning:

Elements of Statistical Learning (aka the bible): https://hastie.su.domains/ElemStatLearn/

Probabilistic ML: https://probml.github.io/pml-book/book2.html

Deep Learning Book (Goodfellow/Bengio): https://www.deeplearningbook.org/

Understanding Deep Learning: https://udlbook.github.io/udlbook/

---

Finally, assorted tutorials/resources/intro courses:

Beyond the Illustrated Transformer: https://news.ycombinator.com/item?id=35712334

AI Zero to Hero: https://karpathy.ai/zero-to-hero.html

AI Canon: https://a16z.com/2023/05/25/ai-canon/

LLM University by Cohere: https://llm.university/

Practical Guide to LLMs: https://github.com/Mooler0410/LLMsPracticalGuide

Practical Deep Learning for Coders: https://course.fast.ai/Lessons/part2.html

---

Hope that helps!

The interesting point here is that the two founders had previously co-founded a similar AI company, Adept, that raised a much larger amount of funding ($415m vs $57m here) to build foundation models:

https://www.theinformation.com/briefings/two-co-founders-of-...

Why did they leave only to create a rival company in the same space? One could argue that Adept is more focused on native multimodality and agentic models (https://www.adept.ai/blog/act-1) developed with longer-term research horizons in mind, but Essential also seems to be aiming at frontier-level capabilities in models trained from scratch.

My bet: as with OpenAI/Anthropic, there's both a philosophical divide and a power struggle to be found in this backstory.

Marketing: Gemini 90.0% || GPT-4 86.4%, new SotA exceeding human performance on MMLU!

Fine print: Gemini 90.0% chain of thought @ 32-shot || GPT-4 86.4% @ 5-shot

Technical report: Gemini 83.7% @ 5-shot || GPT-4 86.4% @ 5-shot

Granted, this is now the second-best frontier model in the world - but after a company-wide reorg and six months of constant training, this is not what success for Google looks like.

Now that this and other articles have established that the board's actions were based not on a single event but rather on general, long-term, underspecified concerns, no one captures the essence of what happened more perfectly than Matt Levine:

"Well, sure, but [the board not trusting Altman] is a fight about AI safety. It’s just a metaphorical fight about AI safety. I am sorry, I have made this joke before, but events keep sharpening it. The OpenAI board looked at Sam Altman and thought 'this guy is smarter than us, he can outmaneuver us in a pinch, and it makes us nervous. He’s done nothing wrong so far, but we can’t be sure what he’ll do next as his capabilities expand. We do not fully trust him, we cannot fully control him, and we do not have a model of how his mind works that we fully understand. Therefore we have to shut him down before he grows too powerful.'

I’m sorry! That is exactly the AI misalignment worry! If you spend your time managing AIs that are growing exponentially smarter, you might worry about losing control of them, and if you spend your time managing Sam Altman you might worry about losing control of him, and if you spend your time managing both of them you might get confused about which is which. Maybe Sam Altman will turn the old board members into paper clips."

(Link: https://www.bloomberg.com/opinion/articles/2023-11-29/the-ro...)

First reaction: okay, should we be impressed?

June: flashy ML-perf demo with CoreWeave using 22k H100s

November: actually trained model using 5k H100s

June: we claim Inflection-1 is the best model in its compute class and are preparing a frontier model

November: we beat PaLM 2, which everyone else forgot about long ago anyway

Inflection got a ton of hype with its $1.3B raise (likely not cash but principally GPU compute credits) earlier this year, but now is starting to look like the next victim of inflated expectations.

Emu edit should be exactly what you're looking for: https://ai.meta.com/blog/emu-text-to-video-generation-image-...

Whether you side with Sam or Ilya on rollout speed and approach to safety, the breathtaking incompetence of this coup has already incinerated a stunning amount of trust. One could hardly ask for a better demonstration that it matters not only what you do and why, but also how - above all, how you treat the ones who rely on you.

Pour one out for the decline of human exceptionalism! Once you get material-reductionist enough and accept yourself as a pure function of genetics, environment, past experience, and chance, this conclusion becomes inevitable. I also expect it to be the standard within a decade, with AI-human parity of capabilities as the key driver.

The PoW camp example is interesting, but the way this article talks about risk and return makes no sense. The author gives an example of a lottery with a $1m prize, funded by selling 1 million tickets at $1 each, then introduces a "Mr. Behemoth" who can buy as many tickets as he wants. Since he can buy 600k tickets to win 400k, for a 66% return at 3:2 odds, or even 900k tickets to win 100k for an 11% return at 9:1 odds, the claim is that he "has complete control over risk and return."

...No? The expected value is 0 in either case.

.6 * 400 + .4 * (-600) = 0

.9 * 100 + .1 * (-900) = 0

The ability to slice a given risk-return profile into different pieces, or different amounts of leverage, is called "finance". But Mr. Behemoth in this case has the same expected return as everyone else, so the example has nothing to do with "control of supply and demand."

The actual claim of this paper is that any watermark can be erased, which it justifies by proving the existence of an Erase function that removes the watermark without degrading quality. Therefore, since any watermark can be erased, let's use the simplest one that's visually identical.

This is true, of course, but also vacuous. The problem is that there is no sense of the computational complexity or difficulty of implementing the Erase function. The proof holds even if the watermark can only be removed in O(e^n) or some similarly absurd time span. A good watermark, like a good encryption scheme or a good password, is one that can be verified quickly but must be reversed slowly.

The paper's stance is no different from saying "since any watermark can be reversed, none of them matter, and we should just use THIS WAS WRITTEN BY CHATGPT".

Which is why you should never assume that a paper makes sense merely because it is technically correct.

I would agree that summarization is a killer use case, but would actually argue for three:

1. Condense (summarize, synthesize, extract, etc.)

2. Expand (explain, copilot, suggest, etc.)

3. Transform (translation, style transfer, switch modality between text/image/audio, etc.)

All of the major consumer use cases of LLMs at this point boil down to one of these three core operations. To the extent that LLMs allow you to perform "magic", that magic is comprised by a chain of {condense, expand, transform} operations, each of which could be performed in half an hour by a reasonably informed college graduate with access to Google.

Even if chain length maxes out at 3 or 4 for production-grade applications, that solves a frightening number of problems.

One expects this to be satire, but it is stated entirely in earnest:

Freedom is control.

People have always been able to make false or misleading statements, at scale, with either false attribution or none at all. This is a matter of degree, not of kind. The primary difference in method is that now these statements can come from any source rather than only an approved mouthpiece.

Why should generative AI compel us to pervert the meaning of the word "freedom"?

Chiming in to confirm that I've tried several of the prompts verbatim and obtained perfectly good answers with sound reasoning from GPT-4. If I add any chain-of-thought markers such as "reason step by step and state your assumptions," response quality further increases. Assuming the author is reporting honestly (and did not, for instance, repeat queries with high temp until getting a bad response), this must have been an extremely unlucky run.

Not a single mention of existing quantization techniques? Ten bucks says this is just a wrapper around bitsandbytes or ggml.

Every facet of the article is designed to wring sympathy, but it's worth asking the real question: what is this advocating for? What kind of change (or stasis) would the author actually prefer to see?

The existence of call-center work, which is intrinsically dehumanizing and unrelentingly laborious, is considered a failure of our society.

The replacement or automation of that work, which creates displacement and impacts the livelihood of people who cannot adequately retrain, is considered a failure of our society.

What, exactly, would constitute success?

Behold, the true stochastic parrot.

No, this is precisely correct. To simplify further:

1. Start H4 and H7

2. Flip H4 when it runs out (4-minute mark)

3. Flip H7 when it runs out (7-minute mark, 1 minute left on H4)

4. Flip H7 back when H4 runs out again (8-minute mark, 1 minute elapsed on H7)

5. When H7 runs out again (after 1 minute), exactly 9 minutes have passed

GPT-4 (5/24 version) indeed fails to solve the problem if not given careful prompting, though I am not convinced this is a new development. However, chain-of-thought resolves the issue. Both prompts and responses included below.

---

Failure

Q: How can you use a 4-minute hourglass and a 7-minute hourglass to measure 9 minutes?

A: Here's a way to measure exactly 9 minutes using a 4-minute hourglass and a 7-minute hourglass:

Start both hourglasses at the same time. When the 4-minute hourglass runs out, flip it over (4 minutes elapsed). When the 7-minute hourglass runs out, flip the 4-minute hourglass again (7 minutes elapsed). The 4-minute hourglass had 1 minute left when you flipped it, so it will run for an additional minute. When the 4-minute hourglass runs out this time, 2 additional minutes have passed (for a total of 9 minutes elapsed). So, you've just measured exactly 9 minutes using a 4-minute hourglass and a 7-minute hourglass.

---

Success

Q: How can you use a 4-minute hourglass and a 7-minute hourglass to measure 9 minutes? Please reason step-by-step. You are allowed to run preparatory steps beforehand if necessary.

A: The problem can be solved by using the following steps:

Preparatory Steps:

Start both hourglasses at the same time. When the 4-minute hourglass runs out, turn it over to start it again. (This marks 4 minutes.) When the 7-minute hourglass runs out, don't flip it yet, but note that the 4-minute hourglass has now been running for 3 minutes on its second run. (This marks 7 minutes.) When the 4-minute hourglass runs out again, flip the 7-minute hourglass. (This marks 8 minutes.)

Measurement Steps:

When the sand in the 7-minute hourglass runs out again, it will have been 1 minute since its flip, marking a total of 9 minutes. So, the process will be: 0-4 minutes: first run of the 4-minute hourglass 4-7 minutes: second run of the 4-minute hourglass until the 7-minute hourglass runs out 7-8 minutes: rest of the second run of the 4-minute hourglass 8-9 minutes: one minute of the 7-minute hourglass after flipping it when the 4-minute hourglass runs out the second time.

Thus, the total time measured is 9 minutes.

The result is mildly interesting - improvement on an isolated task but none on the full benchmark - but what would be much more compelling is curiosity-driven replay in an LLM context combined with chain- or tree-of-thought techniques. This would be the machine analogy to noticing your confusion, a sort of "what do I need to know" or "what am I overlooking"? Anecdotally, language models perform better when you prompt them to ask their own questions in the process of answering yours, so I would expect curiosity to have a meaningful impact.

Shocked no one has suggested the XNOR life yet. What could be more expressive of the hell-yes-or-no mentality?

This is great, but suffers from the same issue as carbon credits in that any secondary market is likely to allow multiple pledging for the same underlying activity. If someone is already simulating millions of orgasmically happy fruit flies, for instance, they can earn utility credits by promising not to deactivate those flies - but how can we verify the same promise hasn't already been extended elsewhere?

I think you need some NFTs.

Self-aware commentary like this only works if used sparingly, but this one is extremely effective - acknowledges the mercenary motivations, implicit hypocrisy, and self-aggrandization of such announcements while nonetheless communicating all the important information and succinctly breaking down a real problem that their platform solves. On top of that, the legitimate uses of funding are clearly identified, answering any questions about both "why" and "how much".

Cheers, guys.

Great idea, but the implementation needs work - not enough time to type responses, and you should be able to send several messages in a row rather than one long one. Otherwise how can we suss each other out?

The key idea from the article is that recursive improvement functions best not as a one-step, but a two-step feedback loop. Rather than "plug the ethics module into the ethics module", allow the pre-existing conception of ethics to shape responses, then modify motivation and so forth.

This seems strikingly parallel to the concept of coherent extrapolated volition advanced by rationalists such as Scott but strangely neglected as a design component in early-stage AI alignment: "If I were the best version of myself and able to control my own values and motivations, what would I want (to want (to want (to want..."