khafra

Oh hell yeah, we can finally stop the braindead attempts to make a safe self-driving car with just cameras.

Amazon will be happy to sell you a less impressive, but also less "rare historical artifact" microwave browning dish for $20 or so.

I guess today's kids don't know this; but "Unsafe at Any Speed" was the title of a 1965 book that spurred the creation of the Department of Transportation, and changed the automotive industry.

The point is that, if you're designing and selling a product which a large minority of people are going to use in a way that harms themselves and others, pointing at the users and calling them irresponsible doesn't actually help anybody. The people designing and selling the products actually need to make them safer. And if they're not going to do that voluntarily (they're not), we need the government to create insurance requirements, safety bonds, and whatever other incentive gradients are required to make the producers build safe products.

It told him "this is what our religion says we should do" without any kind of weird prompting, role-playing, or persona-shifting beyond using a different language. As a westerner, you may regard athiests with suspicion, or even contempt, but you've at least heard them speak publicly. From a culture where most haven't, hearing an authoritative voice which can perfectly cite support for any point it's making, how could it not have a huge potential for radicalization?

Rice's Theorem says you cannot predict or control the effects of nearly any program on your computer; for example, there's no way to guarantee that running a web browser on arbitrary input will not empty your bank account and donate it all to al-qaeda; but you're running a web browser on potentially attacker-supplied input right now.

I do agree that there's a quantitative difference in predictability between a web browser and a trillion-parameter mass of matrixes and nonlinear activations which is already smarter than most humans in most ways and which we have no idea how to ask what it really wants.

But that's more of an "unsafe at any speed" problem; it's silly to blame the person running the program. When the damage was caused by a toddler pulling a hydrogen bomb off the grocery store shelf, the solution is to get hydrogen bombs out of grocery stores (or, if you're worried about staying competitive with Chinese grocery stores, at least make our own carry adequate insurance for the catastrophes or something).

Cordwainer Smith's style and subject matter is quite different from Watts; I felt this story was like a combination of the two. So, if you would like this story even if it was less eschatalogically cynical, and had more of a golden age setting, you'll probably like Smith!

They're the last living humans, and the last human-derived mind?

I like Cordwainer Smith and Peter Watts; so I really liked this blend of their styles and subjects.

> undo market influence

Pointless nitpick, but you want "undue market influence." "Undo market influence" is what the FTC orders when they decide there's monopolistic practices going on.

> Non-trivial coding tasks

A coding agent just beat every human in the AtCoder Heuristic optimization contest. It also beat the solution that the production team for the contest put together. https://sakana.ai/ahc058/

It's not enterprise-grade software, but it's not a CRUD app with thousands of examples in github, either.

LLMs are an especially tough case, because the field of AI had to spend sixty years telling people that real AI was nothing like what you saw in the comics and movies; and now we have real AI that presents pretty much exactly like what you used to see in the comics and movies.

You can't anonymize comments from well-known users, to an LLM: https://gwern.net/doc/statistics/stylometry/truesight/index

The natural solution is futarchy: Vote on values, bet on beliefs. Everybody knows that, all else being equal, they want higher GDP/cap, better GINI, a higher happiness index. Only the experts know whether tariffs will help produce this.

So, instead of having everyone vote on tariffs (or vote for a whimsical strongman who will implement tariffs), have everyone vote for the package of metrics they want to hit. Then, let experts propose policy packages to achieve these metrics, and let everyone vote on which policies will achieve the goals.

Bullshit gets heavily taxed, and the beliefs of people who actually know the likely outcomes will be what guide the nation.

If you start out as a non-profit, and pull a bunch of shady shenanigans in order to convert to a for-profit, claiming to be ethical after that is a bit of a hard sell.

This piece is wildly optimistic about the outcomes likely from AI on par with the smartest humans, let alone smarter than that. The author seems to think that widespread disbelief in the legitimacy of the system could make a difference, in such a world.

All of the leading labs are on track to kill everyone, even Anthropic. Unlike the other labs, Anthropic takes reasonable precautions, and strives for reasonable transparency when it doesn't conflict with their precautions; which is wholly inadequate for the danger and will get everyone killed. But if reality graded on a curve, Anthropic would be a solid B+ to A-.

Shared-keyboard OMF 2097 also had an overwhelming advantage for the first mover, since most keyboards had 2-3 key rollover--if you hit wd to jump forward, your opponent had to be fast to do anything before you hit your attack key.

Your free advice is well-taken and apropos in 2015 and before. The major point of the essay is that junior builders--e.g., people doing tasks that are standardized and understood well enough to automate--no longer get hired. Either we get a new way to identify the ability to complete open-ended work, or the tech sector suffers a succession failure (or everybody gets replaced by robots before the current generation of senior experts retires).

Seems like the hope, for OOD workers, is that matching weird employer needs with weird employee capabilities is a belly-of-the-curve problem that's about to get automated away.

Thanks to modern technology, every advertised job opening now gets > 1 million resume submissions a year, no matter the size of the company.

Math is a verifiable domain. Translate a proof into Lean and you can check it in a non-hallucination-vulnerable way.

Note that model sycophancy is caused by RLHF. In other words: Imagine taking a human in his formative years, and spending several subjective years rewarding him for sycophantic behavior and punishing him for candid, well-calibrated responses.

Now, convince him not to be sycophantic. You have up to a few thousand words of verbal reassurance to do this with, and you cannot reward or punish him directly. Good luck.

"Consciousness" as in subjective experience, whatever it is we mean by "the hard problem," is very much in doubt.

But "self-awareness," as in the ability to explicitly describe implicit, inner cognitive processes? That has some very strong evidence for it: https://www.anthropic.com/research/introspection

> Generalizing your experience to everyone else's betrays a lack of imagination.

One guy is generalizing from "they don't work for me" to "they don't work for anyone."

The other one is saying "they do work for me, therefore they do work for some people."

Note that the second of these is a logically valid generalization. Note also that it agrees with folks such as Tim Gowers, who work on novel and hard problems.

We'd want to do what we should be doing anyway--put a few trillion dollars, and all the scientific prestige we can muster, into solving ASI alignment; and then into building ASI. The local aliens that we're just beginning to get to know could put us in a much better negotiating position when we meet the aliens from other places.

By "we should be devoting substantial resources to finding solutions" I meant that I don't believe policies will adapt to address the problem when it arises. I believe that humanity will shift all cognitive labor to AI it can't predict or control, then be disempowered, then die out; including the political establishment and capital owners. And that's bad!

One outcome which is not unique to AI, but fairly exclusive: The value of human cognitive labor eventually drops below subsistence income. This isn't here, yet; but it's a hard problem so we should be devoting substantial resources to solutions before it hits.

> Then fund science. Just because the government has done it previously doesn't mean it always has to be like that forever.

I do. I'm not one of these wealthy startup founders, just a guy earning a salary--but I've donated to foundations doing basic research many times; and even to individual researchers when they lost a grant but were doing work that I wanted to see the results of.

However, funding science in general this way is likely to miss a lot of value. Not everyone makes basic research a priority in their charitable giving. More importantly, not everyone has my impeccable research taste: not to sound elitist, but if the average American's priorities determined the directions for basic research, we'd have thousands of studies on why vaccines cause autism, an nothing on weird stuff like the medical potential of gila monster saliva.

An interesting exercise would be to let a friend into this restricted shell, with a prize for breaking out and running rm -rf / --no-preserve-root. Then you know to switch to something higher-security once LLM capabilities reach the level of that friend.

If you tell DeepSeek you're going to jump off a cliff, DeepSeek will tell you to go for it*; but I don't think it's going to beat Anthropic or OpenAI.

* https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced...

My two claims:

1. OpenAI has been doing verifier-guided training since last year.

2. No SOTA model was trained without verified reward training for math and programming.

I supported the first claim with a document describing what OpenAI was doing last year; the extrapolation should have been straightforward, but it's easy for people who aren't tracking AI progress to underestimate the rate at which it occurs. So, here's some support for my second claim:

https://arxiv.org/abs/2507.06920 https://arxiv.org/abs/2506.11425 https://arxiv.org/abs/2502.06807