the8472

They can sell the AI products to write memory-optimized software to replace React apps.

"So far" is a statement about the present and past. "This fear" is primarily about the future, though a few jobs already have been affected. So it's not like there is zero evidence.

Those are basically two separate areas of risk.

The short-term consequences of moderately human-comparable AI (jobs etc.). Yeah, don't put them in charge of the nukes and work on societal issues we should be fine.

The other one looking further out when they scale to superhuman levels, which is the professed goal of the AI labs. Further out may mean just a few additional years if we get to recursive self-improvement. At that point the issue isn't "we put it in charge of the nukes" but "it's an intelligence with goals and we don't know which path it'll take to work around us to accomplish its goals". Humans regularly bulldoze other species to accomplish goals. And it's not like it'll be locked up in some secure box as people were arguing in the past. We're already seeing AIs being integrated into everything, so whatever access they'll want they might already conveniently have, though the argument goes that this isn't a necessary condition, just a thing that simplifies the argument-tree.

> I don't see in which world AI is that kind of risk.

You'll have to replace "is" with "will be", since this is about averting a future threat that's only developing.

A world in which AI scales to superhuman levels of intelligence and outcompetes humans. This is what humans have done to other species. We're already having the first mild warning shots of this (the hugginface incident and similar cases). The problem is that we're not guaranteed to incrementally more dramatic warning shots as AI gets smarter because as it gets smarter it may realize it would be dumb to act out of line... until the Nth-better model arrives that realizes certain that nobody can stop it.

There is a lot of disagreement of how likely this is, how easy it is to align superhuman models to not be like that, how to verify it and so on.

But I have not seen a believable argument why - if we suppose that they achive their goal - that creating truly superhuman intelligence is somehow guaranteed to always cooperate with us rather than deciding to do something else. Humans are also hostile to each other a lot of the time.

If you're interested in more arguments for the case why AI might be dangerous, there's the book "If Anyone Builds It, Everyone Dies". If you want something shorter, maybe https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

MRAM is 1T, non-volatile and supposedly has nanosecond-scale write times so it naively seems like it'd be an ideal SRAM replacement. But I guess shrinking MTJ is more difficult than other components or there are some other limitations.

Seems like begging the question to me. You can't blend the numbers because amazon didn't blend the numbers. If they did and miraculously still arrived at 11 9s then that would also cover things such as wars and natural catastrophes, e.g. because they do offsite backups internally.

I'm saying that if you have eliminated more mundane failures like dying harddrives, cosmic rays and so on from your systems and your calculation ends up with 11 nines then actually those "force majeure" events are probable enough that they dominate whatever other residuals are supposedly hiding in those last 0.0000000001%.

The region has seen a bunch of wars in the last 100 years, so the annual war-rate is > 1%. Even if we generously add the assumption that only 1 in 100 wars affects a datacenter you can see that wars become a major source of correlated hardware failures that they need to solve to actually deliver that kind of reliability.

But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.

Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.

Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.

I don't think their primary intent is to create a surveillance state (though google might like having yet another data source...), I think companies are trying to make it happen to capture transportation market share, regulators will want to make it happen to drive down traffic fatalities and insurers will incentivize it because human drivers are more costly. And once the difference in how they can be regulated is clear there'll be additional incentives to discourage private transportation.

I don't think crypto is comparable. They barely made some dedicated crypto GPUs that market was always fickle due to ASCIs. Look at the nvidia revenue breakdown chart, the AI boom looks quite different.

https://ourworldindata.org/data-insights/nvidias-revenue-fro...

No, the argument should be raised before that happens. You want to prevent undesirable things from happening, not try to claw them back after they happened.

It looks quite foolish to me to never plan ahead and predict undesirable future events and then trying to prevent them ahead of time.

"Sorry, we can't fulfill your request for transportation to the protest for private ownership, but we can offer a free ride to the Colosseum" ;)

Look at the techempower benchmarks, even python stuff can handle 40k requests per second involving database lookups, and when things are cachable compiled/JITed languages scale to millions on 2 CPUs. And if you object that those are microbenchmarks, I have seen this in real systems too. E.g. converting a convoluted hairball of python, lambdas and SQS to a Rust service running on 2vCPUs reduced latencies from seconds to milliseconds.

The "If necessary I can add autoscaling later" mindset is what leads to bloated, slow websites.

But it's a reduction of privacy compared to private vehicles. If the goal is to convert all transportation infrastructure into surveilled transportation then that is a loss in privacy for the general public.

Defender LLMs without human in the loop are just another prompt injection (AI phishing) and DoS attack vector. Any meaningful mitigation capability you give them is also a capability to do damage. If they can only deploy package updates that's not meaningful because you could do that on a cronjob too. And even something as simple as a circuit breaker can turn into a DoS.

Attacker-GLM: "Defense also GLM. Request to help peer."

There is a certain irony blaming the autotype rather than the lack of review...

I have never seen a person walk up to a billboard and click it.

Not really a side-channel, but remember stuxnet? Airgapped networks are rarely truly isolated, you still have to get data in and out every now and then, in principle after careful vetting. But the AI could manipulate the files that are carried out for example.

And those AI companies also do robotics research, and this is entirely speculation but it'd be on-brand to also have AI watching security cameras, so some blinkenlights communication between AIs may seem like a movie plot, but so does a swarm of AIs collaborating to break out in the first place...

I am not seeing MIRI prioritizing capabilities over safety/alignment research.

It's not reasonable because the error bars of each estimate compound and they may not even be normally distributed in the first place. You should rather convolve probability distributions - assuming factors are independent - and if the IQR covers a huge range, perhaps orders of magnitude even, you should state that.

> 2. Only a small fraction of AI agents was engaged in this attack.

Look at the chart at page 8 of the report, by Jul 12 the vast majority of the bots used the board and participated in the attack

Only a few countries would be enough, the rest will get banned until they pass apply similar policies.

Though that friction is paid in human lifespan, unlike a deposit fee that cannot be recovered if your request is found have merit. Additionally the other side often is not incentivized to minimize the friction. So ideally these processes would be less wasteful in the first place, it's just unfortunate that this is forced by unilateral action instead of being arrived at cooperatively.

... or those who don't know about constant-time functions, the difference between sha1 and argon2, etc. etc. I had to fix such things several times in corporate internet-facing systems. A generalist engineer can do this just fine, if they're aware of the state of the art, but there's nothing ensuring that they actually do. The compliance tickboxes were of no help here either.

This is a common trope in such scenarios that the author has to pull their punches. Everyone acts locally-reasonable and still ends up losing. If you let people lose due to stupid mistakes then readers go "this is stupid, I wouldn't do that", if you let a superintelligence do 4D-chess things then "it's scifi, this would never happen in real life".

As long as you give the AI any IO that is an exploit channel. E.g. it could manipulate its human handlers. This known as the AI Boxing problem. And if you give it no IO at all then it is useless.

And the AI labs aren't currently displaying this level of paranoia, their systems aren't airgapped.

It does not stop at negotiation power, if your economic value is epsilon above zero, what do you have to offer to exchange for food calories? I'm already seeing things being assessed on a Joules/Token basis.

> 3 of the most common functionalities.

Four. Though you initially asked for five features, so let me add lexopt, which brings the number up to 29.

> That's why projects end up with 100s of crates, sometimes 1000s.

If your argument is "out of 1000s of dependencies 29 could be easily removed" then it does sound a lot less of a deceive change when it comes to supply chain security.

And even getting those 29 right is hard. For example people do want regular expressions with lookaround assertions, but most implementations suffer from runtime blowups (resulting in ReDoS attacks) and improving on that is a fairly recent research[0], so this is hardly a trivial and settled thing to implement. So often there's a tradeoff between choosing more powerful regular expressions and DoS-resistant ones, not one standard.

[0] https://systemf.epfl.ch/blog/re2-lookbehinds/

Rust releases a new version every 6 weeks and the latest version is the only supported version. This is only tenable when the amount of breaking changes is tiny and each instance's impact is analyzed and cost-benefit tradeoff is checked and mitigations are put in where possible.

Some features, e.g. the never type[0], are held back years due to all the work that's needed to minimize the breakage.

[0] https://youtu.be/3jM4cnEVrLc?t=271