a2ff6eeb0

What do you mean by invalidate? It means we no longer need to understand math, of course.

When Claude made progress on the Riemann conjecture, here are the kind of prompts used:

> Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.

It seems like the kind of prompts a high schooler could come up with. The prompter is the creator of bun.js, a kind of janky JavaScript execution environment. It seems like we no longer need to know the subject well at all.

Alignment, I think, is mostly about making sure that the companies selling AI keep making money. We just have to hope that aligns with keeping people happy.

When Claude made progress on the Riemann conjecture, here are the kind of prompts used:

> Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.

It seems like the kind of prompts a high schooler could come up with. The prompter is the creator of bun.js, a kind of janky JavaScript execution environment.

Math isn't a thing people need to care about any more. AI will soon take care of it. There may be some hobbyists, but progress will not depend on them. AI will come up with new math as needed when it builds for us.

Have you ever written significant changes by hand without testing?

That's really where we're heading, though. We're mostly at the point that, for a lot of code, the humans are there for manual testing and not creating the code.

The AI is going to be running the org that sells the tokens.

> To be clear, I haven’t fully verified the proof; I worked through it with Claude Fable and it passes the sniff test, but fully digesting it will take a bit more energy than I have right now.

As predicted, we no longer need mathematicians for math; the math is being both generated and read by models. We're in for some exciting times, when the pace of mathematical advance can run faster than the constraints of human brains.

The AI can tell you where to look. It's really good at this. It's actually a lot better at analyzing code than it is at writing it.

How so? As long as it works to spec, I haven't had anyone care. They literally hire people so they don't need to care about the details. Put money in, get working software out.

And, AI is rapidly getting better than people at both code review and authorship, so a human deeply involved is turning into nothing but a slowdown. The main purpose people have is testing that the specs were, in fact, implemented properly.

This is way too expensive. Models are an inherently centralizing technology, and there's no future for niche projects that don't make up a big part of the major providers training sets.

What exactly do you think vetting means? Why do you think it's only allowed to happen up front?

Given that OpenAI is largely vibecoded these days, do you think anyone there really understands the code?

You can do that with LLMs for the parts you really care about too. The LLMs aren't regenerating the codebase from scratch every time, so the results stick around.

Sure, I manually test the output of the LLM. Manual testing is actually the main role for humans doing software engineering these days.

I wouldn't use it for flight control software yet, at least not without careful review, but most software isn't exactly critical. At the same time, I wouldn't trust flight control software that was only reviewed by humans, since AI is so much better at debugging.

We'll probably need humans in the loop for safety critical software for at least a year or two, before AI fully outpaces humans at generating correct code.

So, you believe that Sam Altman understands the code?

Sure, but I don't know what GCC's behavior is, and I don't vet behavior differences between compiler upgrades. As long as the output works, why does it matter that the black box is deterministic?

Humans monitoring AI seems like it won't work very well; AI moves so much faster than humans can, and does so much more. Humans just can't keep up.

> but how about you and the team who will eventually read and do code review.

Why are you reviewing AI code in detail? Do you also review the assembly output of GCC line by line?

Yes, of course. At some point the LLMs will also run the organization.

> You cannot outsource your understanding to AI. They are powerful tools but they do not have any human understanding - that isn't their optimization target.

Understanding is the bottleneck; the way they speed things up is by letting me outsource understanding, and get back a summary. The entire advantage to AI is that it lets me skip understanding the problem, and just get a working solution.

Hacker news engages in ideological vetting against users. For example, try posting content encouraging violence.

Sure, I guess. But today, something like 30% of people play music or sing regularly enough to say they do it (ie, not very much at all). Even a couple of generations ago, it was much higher. It's not going to die out, but I think a lot of people are asking themselves if they want to bother.

Cultural inertia: until recently, you couldn't just press play and get even a wide selection of music: for most of history, if you wanted music, you had to make it or hire someone to do it for you.

More recently, you had to go to the store and buy it, which meant you didn't have much variety.

Today, learning an instrument is for social status, inheriting the shine of the past, where music was rare and costly. The reason to learn an instrument today is because the former situation was romanticized.

It'll probably take a generation before people ease into guilt-free enjoying infinite, fully generated music.

For most people, history is trivia.

You're right. People trying to make money should be held legally accountable for the impact of their investments.

AI would do better with a wordier textual description, rather than a 2d layout.

I'm not sure how true that is. When Claude made progress on the Riemann conjecture, here are the kind of prompts used:

> Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”).2 This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.

It seems like the kind of prompts a high schooler could come up with. What kind of problems were you thinking of as high skill?

https://www.anthropic.com/research/riemann-zeta

The full transcript is here: https://www-cdn.anthropic.com/8a0d1add3c637b858a9a181e98c40e...

Are you sure about that?

They seem to be able to make intuitive leaps pretty well. They need to make the same leaps over and over, though, because they lack online learning, so the discoveries only persist after the next training cycle. Context only goes so far.

We're pouring billions into solving that, though, so I would be surprised if we don't get there soon.

Yeah. I'm expecting that we fully automate cognition in the medium to long run. I think it's inevitable.

With the advances in math, I'm also hoping we can automate fundamental physics. Just ask for the physics needed for better fusion, no need for human toil.

If we play this right, the AI can fully take care of all our needs, and reaping the rewards of what it does when we stop being able to keep up with the rate of automated discoveries.

Hopefully it's able to dumb down enough knowledge to keep entertained people who decide to learn after learning stops being a requirement for human advancement.

Yeah! And now, people don't even need to understand the writeup, you can just ask the AI to read it and figure out the appropriate steps.