a_wild_dandan

friction

So...nothing will change?

This conclusion is the lump of labor fallacy. It's not that simple.

Let's just define AI as "whatever computers still can't do." That'll show those dumb statistical parrots!

An additional citation substantiating that 10 MB/s figure: the 3rd sentence of TFA.

Expected utility, just like anything else. It's why our ancestors ate willow tree bark (aspirin), for instance.

Animation can look real though, e.g sci-fi vfx. But maybe you’re concerned about how prolific it may be? I could see that. Personally I think it’ll be fine. It’s just that disruptive tools create uncertainty. Or maybe I’m overcompensating to avoid being the “old man yelling at cloud” dude.

Reasons that I can't take this benchmark seriously:

1. Existing brute force algorithms solve 40% of this "reasoning" and "generalization" test.

2. AGI must evidently fit on a single 16GB, decade-old GPU?

3. If ARC fails blind people, it's not a reasoning test. Reasoning is independent of visual acuity. So ARC is at best a vision processing then reasoning test. SotA model "failure" is meaningless. ("But what about the other format, JSON?" Yeah, I would love to see the human solve rate on that...)

AI researchers get huge checks working anywhere. So how does your company distinguish its appeal? Play to a researcher's desire for recognition in their field. Lean on open source. (I'm just paraphrasing Zuckerberg.)

You can make highly accurate predictions of what contrarians will say by assuming that they define AI as "whatever computers can't do yet."

LLMs aren't truly intelligent. [No True Scotsman fallacy...] They don't really reason. [A distinction asserted without giving a falsifiable definition of reasoning...] They're just next token predictors! [Which must be mutually exclusive with intelligence, I suppose?] Etc, etc, etc. Find your favorite pretext to dismiss modern AI, ignore the holes in the argument, and satisfyingly conclude that it's all smoke and mirrors.

Consequently you see hilarious takes from skeptics, like comparing today's enormous investment in AI to when people sold blockchain cartoon monkeys. Or claiming that modern models aren't useful for anything, as if they exist in an alternative reality where hundreds of million of people don't use them daily, and there's no incessant firehose of new tools/products/results discussed in news/social media constantly.

All of machine learning is AI. It's a subset, by definition.

Important testing excerpts:

- "...for the closed (OpenAI) models I tried generating up to 10 times and if it still couldn’t come up with a legal move, I just chose one randomly."

- "I ran all the open models (anything not from OpenAI, meaning anything that doesn’t start with gpt or o1) myself using Q5_K_M quantization"

- "...if I gave a prompt like “1. e4 e5 2. ” (with a space at the end), the open models would play much, much worse than if I gave a prompt like “1 e4 e5 2.” (without a space)"

- "I used a temperature of 0.7 for all the open models and the default for the closed (OpenAI) models."

Between the tokenizer weirdness, temperature, quantization, random moves, and the chess prompt, there's a lot going on here. I'm unsure how to interpret the results. Fascinating article though!

The model meets/beats Llama despite having an order-of-magnitude fewer active parameters (52B vs 405B). Absolutely bonkers. AI is moving so fast with these breakthroughs -- synthetic data, distillation, alt. architectures (e.g. MoE/SSM), LoRA, RAG, curriculum learning, etc.

We've come so astonishingly far in like two years. I have no idea what AI will do in another year, and it's thrilling.

This could be revolutionary. The PPL/compute graphs are damning. If the Transformer is a function, then the TokenFormer feels like a higher-order function. Perhaps this approach is a natural direction for producing System Two reasoning? There's so much to digest here...

What were the major differences?

I suppose Valve, who trained a neural network to detect/ban cheaters exhibiting unnatural behavior.

The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]). It's just harder to ignore reality with hundreds of millions of people now using incredible new AI tools. We're approaching "don't believe your lying eyes" territory. Deniers will continue pretending that LLMs are just an NFT-level fad or bubble or whatever. The AI revolution will continue to pass them by. More's the pity.

[1] https://arxiv.org/abs/2110.09485

Evaluating cross-lingual transfer learning approaches in multilingual conversational agent models[1]

Cross-lingual transfer learning for multilingual voice agents[2]

Large Language Models Are Cross-Lingual Knowledge-Free Reasoners[3]

An Empirical Study of Cross-Lingual Transfer Learning in Programming Languages[4]

That should get you started on transfer learning re. languages, but you'll have more fun personally picking interesting papers over reading a random yahoo's choices. The fire hose of papers is nuts, so you'll never be left wanting.

[1] https://www.amazon.science/publications/evaluating-cross-lin...

[2] https://www.amazon.science/blog/cross-lingual-transfer-learn...

[3] https://arxiv.org/pdf/2406.16655v1

[4] https://arxiv.org/pdf/2310.16937v2

Yes, because transfer learning works. A specialized model for X will be subsumed by a general model for X/Y/Z as it becomes better at Y/Z. This is why models which learn other languages become better at English.

Custom models still have use cases, e.g. situations requiring cheaper or faster inference. But ultimately The Bitter Lesson holds -- your specialized thing will always be overtaken by throwing more compute at a general thing. We'll be following around foundation models for the foreseeable future, with distilled offshoots bubbling up/dying along the way.

Cursor is too! Mixing and matching specialized & flagship models is the way forward.

The speed with which AI models are improving blows my mind. Humans quickly normalize technological progress, but it's staggering to reflect on our progress over just these two years.

Are there any other companies which surprised you with their ubiquity?

I'm so jazzed to comfortably run Llama 3 405b on my Mac with less quantization.

I’m continually surprised at JavaScript’s speed. Seeing JS sometimes nipping at the heels of C/rust/etc in performance benchmarks blows me away. V8 is such an incredible piece of engineering.

In my work, it’s hard to justify using something other than JS/TS — incredible type system, fast, unified code base for server/mobile/web/desktop, world’s biggest package ecosystem for anything you need, biggest hiring pool from being the best known language, etc.

It’s just such a joy to work with, ime. Full-stack JS has been such a superpower for me, especially on smaller teams.

The dissonance between how the silent majority feels about JS (see, e.g the SO yearly survey), vs the animus it receives on platforms like HN is sad. So here’s my attempt at bringing a little positivity and appreciation to the comments haha.

Running a (405B) flagship model locally on a Mac Studio? Otherwise, no. Most of these products are fairly similar.

"Commoditize your complement."

Lina Khan's response to this argument. [1]

[1] https://www.youtube.com/live/L_QaZk5iJOA?si=ZkxBe1CHgagmcBcW...

Could Apple have Google-like restrictions in the future? Or are we kinda fucked because Apple already "won"?

More citations from their blog[1] too.

[1] https://www.liquid.ai/blog/liquid-neural-networks-research

"The Llama jumped over the ______!" (Fence? River? Wall? Synagogue?)

With 1-hot encoding, the answer is "wall", with 100% probability. Oh, you gave plausibility to "fence" too? WRONG! ENJOY MORE PENALTY, SCRUB!

I believe this unforgiving dynamic is why model distillation works well. The original teacher model had to learn via the "hot or cold" game on text answers. But when the child instead imitates the teacher's predictions, it learns semantically rich answers. That strikes me as vastly more compute-efficient. So to me, it makes sense why these Llama 3.2 edge models punch so far above their weight(s). But it still blows my mind thinking how far models have advanced from a year or two ago. Kudos to Meta for these releases.