mitthrowaway2

I don't know where you are, but when I hire mechanical engineers, I hire them from universities with accredited engineering programs. It's actually a legal requirement for the job in my jurisdiction. And if those engineers decide they want to become daycare teachers or knee surgeons, they do have to go back to university and re-train, even though they already learned how to think the first time.

Napster was a fantastic and disruptive product, the likes of which arguably has no equal to this day. But eventually the hammer came down from the courts and it was replaced by streaming services like Netflix, which pay to license materials from their creators.

The original definitely could refuel midair.

I don't think most prototypes are serious software.

Actually reading it, it seems like they say both the geoengineering and pandemic pathways could cause human extinction, but that the AI would have to be doing it to kill us on purpose, not by accident.

That doesn't seem too comforting to me? I'm not sure what this article's author was expecting us to take away from it.

I'm not certain. In an aging society, the thing they'd need are young laborers. People with 30 years of savings are unlikely to be either young, nor laborers.

Maybe those wealthy immigrants are very talented people who are very good at, say, managing an insurance company or researching pharmaceuticals, but Japan has a glut of 55-year-olds who can do that stuff very well. They're running short of young people who can be nurses, junior machinists, and construction workers.

In a macroeconomic sense, wealthy people might be able to pay for the services they need from their own pockets, but functionally it's not that different than being unemployed (unless they choose to work). They would be almost purely adding demand to the economy. The wealth they are bringing represents supply that they added into the economy in the past and abroad, and now they have a wallet full of paper with which they can buy services for the next 30 years in Japan without having to do any further productive work.

Declaring Dominos a supply chain risk means that Boeing cannot order a Dominos pizza for their board meeting in Arlington, because Boeing sells F-35s to the Pentagon, and Dominos wouldn't agree to deliver pizzas to a war zone.

Declaring Anthropic a supply chain risk means that Boeing can't use Claude to draft a poster informing people about the board meeting.

Maybe the military should just seize the technology and nationalize the company then? Because clearly, paying a private party voluntarily entering into a contract on mutually-agreed upon terms is now a polite fiction that can be done away with.

It's funny, I buy some software and the license says "for student and non-commercial use only", which lets me get it for $50 instead of the $5000 that an enterprise would pay. But I can just ignore that clause or declare them a supply-chain risk if they try to hold me to it, because now that they've taken my $50, who are they to dictate how I use it? I can do whatever I want, not just what the vendor thinks I should, right? A contact is just paper, right?

They're offering a service, with contractual limitations that the Pentagon agreed to.

It doesn't seem any different to me than Dominos pizza offering to deliver pizzas to the Pentagon but not to the strait of Hormuz. If that somehow makes the military subordinate to Dominos Pizza, just because Dominos is able to dictate the terms of their own delivery service, then sure, ban anyone in the military supply chain from being able to order pizzas. It would make the same amount of sense.

The previous power plant manager had, of course, already signed a contract agreeing to those terms; the power plant is subsequently under new management and upset about being bound by this contract, so they designate that none of their suppliers of parts and fuel are allowed to have any commercial dealings with the pen manufacturer or use these pens for any purpose, even writing.

The universe cares about dimensional analysis because it is invariant under changes of units-of-measure, which are human constructs.

What's an example of a completely novel problem?

... Huh?

No, I'm saying -- and I can't believe I have to spell this out -- that pedestrians should be rolling around inside giant steel spiked balls like sea urchins.

This is so far from the point of the analogy. But when you don't normalize by miles driven, the improvements don't look quite as impressive, especially for pedestrians.

https://www.iihs.org/research-areas/fatality-statistics/deta...

For what it's worth, the fangame "crimson echoes" is a great alternative sequel.

There's a theory that the best way to reduce fatalities from car accidents is to put seatbelts and airbags in every car.

There's another theory that says the best way is by putting a big spike in the driver's steering wheel.

So. I guess, if you believe that the only viable solution is model alignment, rather than relying on technical barriers to exfiltrating weights, then this is a decent steering wheel spike.

I'm not denying that there's a shadowy ruling elite cabal of influential billionaires in the US, but if any of them are members of the rationalist community, they're certainly not influential within it. Maybe Elon lurks on lesswrong and posts stuff under a pseudonym or something, it's certainly possible, but doing that doesn't make someone a ruler of the group.

Who are the rich people you associate with the rationalists?

The main influential people I think of in that crowd who steer the conversations are Eliezer Yudkowsky and Scott Alexander. They're both doing fine financially at this point, I think, but not "buy politicians and laws like Elon Musk" fine.

There are some very rich people who might be loosely associated, like Dario Amodei (maybe?) or perhaps some VCs, but I'm not sure anyone really thinks of them as being a major influence among rationalists, more like people who hopped on the bandwagon. Certainly not ruling it.

What do you mean by "ruling"?

I would understand "popular".

> Similarly, those who build AI need to take precautions to make sure the AI they build doesn't accidentally cause serious harm.

I am ten thousand percent in agreement that this is needed. And yet, I don't know how we ensure and incentivize that these precautions are taken by the actors involved, and as I understand it, even many of the top researchers in the field agree that they don't know what precautionary measures would even be effective, let alone sufficient. It seems that game theory has so far been pushing the AI companies to build it anyway without a robust solution for preventing harms, even while they express worry in public about the potential for those harms.

Again we can hold corporations or people responsible for damages after the fact, but many examples can be cited to show how that tends to be insufficient.

You seem focused on intentional misuse but I think there's an equal or greater potential for harm happening by accident without intention. Nuclear weapons, at least, have so far reliably and predictably done exactly as their owners intended. Nuclear power plants on the other hand have, on occasion, blown up despite nobody intending them to do so, causing some populated areas to become uninhabitable.

Don't you think that AI has by nature at least a little more potential for unpredictable results and unexpected harms than your typical technology? It seems like a major oversight to completely disregard this aspect.

We can maybe hold someone responsible when things go wrong and harm is done, and blame them for negligence as though the harm were intentional, but from a preventative safety perspective that's not usually sufficient nor even always helpful for preventing accidents.

Reinforcement learning makes it something different. It becomes much more of a search engine through next-token-space that targets the training objective. Better to think about it like that, and then you'll see why "these things have motive" is not a terrible analogy, and you'll better be able to anticipate what they do.

Okay, so what should we do?

I for one would like nobody to build the AI, is that an option? How do we get it on the menu?

Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"

They're about as insular as the Unitarian church or Hacker News. It's basically an online community where anyone can check in, check out, and leave a comment or an essay. There's no formal structure or membership organization. The people who write or associate there could be edgy pre-teens or prominent math professors, it's really just a self-association thing. There are informal self-organized meetup groups in various cities, but not much different than groups who meet to discuss poetry or literature or video game speed-running. Some ideas have bubbled around more than others, like any memetic spread, but I don't think it's accurate to say that the people who associate there all agree on things, just like HN does not either. And nor is it any more of a clique than HN, or Twitter for that matter, even though you'll find some people on Twitter are more prominent and more vocal than others.

Who are the actual scientists and engineers, what do they think, and are they not allowed to to blog?

As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.

Since we haven't reached the end yet, I'm not so sure.

Who has the authority to perform that shutdown without getting arrested, and does that person have a mandate and responsibility to take that action in response to AI misbehavior?

What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?

It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.

He can slow down his own company kind of like how Zelenskyy can just declare peace in Ukraine. It works a lot better if you can get the other sides to agree.

They don't cost that much within Japan. No idea why it's so hard to buy them elsewhere, while Korean models are relatively commonplace.