api

I prefer thinking of human AI fusion like a lichen, which is a close symbiosis at the cellular level. That’s what it would look like.

When I read about how CANDU (heavy water moderated reactor) works I started wondering why we ever build any other kind of large scale reactor.

Runs on natural uranium. Can burn waste too, and plutonium, and even thorium (mixed in, not pure, but still). It’s like a flex fuel reactor. It’s very safe and reliable.

Why do we bother with any other design? Except maybe fast breeders or fission fusion hybrids but those are whole new directions.

This is how prices crash.

Currently reading Eversion by Alastair Reynolds.

Highlights from the past year or two:

Revelator by Daryl Gregory - Appalachian weird fiction with a tinge of cosmic horror.

The Devil Takes You Home by Gabino Iglesias - Tragic story about the cartel underworld plus magic realism.

The Expanse (all of them) by James S A Corey (pen name for two writers IRL) - Everyone probably knows this one. Books were better than the show in many ways the the prose quality was pleasantly good. Fun read.

Tomato Red by Daniel Woodrell - One of the best openings I have ever read and I’ll leave it at that.

I like to read a wide variety of styles and genres, but the writing has to be good. I’ll drop a book if it’s got flat characters or clunky prose or cringey tropey stuff.

Anyone old enough to remember Phil Zimmerman and Pretty Good Privacy, the last time the US government tried to outlaw math?

Are people with ethics going to seek out jobs in the gambling industry, especially mobile addictionware gambling?

I'd work in weapons before I'd work in this. At least those are usually used against military targets and overall hurt or kill fewer people. A rocket might blow up one school, but this will destroy millions of families.

I wonder if doing this to hardware would have overall been cheaper than doing the baroque Rube Goldberg shit we do to software to make it somewhat kind of fault tolerant… or at least convince ourselves it is.

(You can write real fault tolerant software but a wad of tangled crap and hidden dependencies crammed into Kubernetes is not it.)

Are those inflation adjusted numbers? Inflation tends to hide all kinds of problems, which is why politicians love it.

AMD has always had software problems too. Hardware companies often devalue software and suck at it.

If you can make them work Arc cards are a massive bargain. On raw compute the silicon is not bad.

I think your memory of pre-AI UIs is rosy. This is a little below average.

That's why I said it should inspire skepticism. I take it as a cue to take a closer look.

It's got two purposes I think. One is the obvious regulatory capture. The other is edgelording, which gets a lot of attention and makes the product seem like it's more powerful than anything else.

Good advice for everyone in this age of staring at screens.

A few "razors" I use, which are probably not original to me:

(1) People are more important than ideas. Any idea that says the idea, or its vision of the future, is more important than people fails. Doesn't matter what school of thought it comes from or how good it sounds or whether it seems intellectually consistent. If the idea is not working in a way that is hurting people, toss the idea.

(2) Peoples' personal lives are absolutely on the table when evaluating their ideas. If someone's personal life is an absolute mess, or they're running a cult, or they're in one, or they treat the people around them like complete crap, it should be cause for extreme skepticism at the very least.

(3) If it's hard to understand, it might be profound... or it might be pseudo-profound bullshit that's muddying the waters to make them look deep.

These are pretty rare failure modes.

ICE engines eat themselves in many more different kinds of weird ways: timing chains and belts, coolant leaks, any number of problems with valves and actuators. They're extraordinarily complex devices and it's a testament to engineering genius (and persistence) that we've made them this reliable. But they fail much more often than batteries or motors. They have many more moving parts. It's just the devil we know.

I've got two EVs. Basically zero problems. I've known one or two people who have had issues, but never with the drive train. Always something dumb like the "infotainment system" or (for a Tesla owning friend) stupid buzzing and vibration they couldn't make go away and stupid problems with door handles.

The Tesla weirdness is part of why I went with Nissans. They're 100% normal cars except for the electric drive train. All the other parts can be purchased at AutoZone and everything else can be worked on by a normal mechanic.

There's some mechanics in my area that have been learning how to do stuff with EVs like aftermarket battery upgrades on older models, but it's a bit expensive. Mine are 5 and 13 years old respectively and both batteries are fine. The 5yo one is the same as the day it was bought. The latter's lost some capacity but still very usable as a city car.

All that is historically true, but most new energy being added to the grid is now renewable. Solar and batteries continue to get cheaper and it's already the cheapest source of new power in sunny places.

I have seen this raised as yet another interesting Fermi paradox idea: maybe our planet is unusual in that it stored up a giant store of easily exploitable energy to bootstrap a technological civilization.

What's special about fossil fuel? Is a kilowatt-hour from burning carbon magically able to power an economy better than one from a solar panel, a wind turbine, or a nuclear reactor?

I think you could make that argument (at least structurally) as late as the 2000s when batteries and solar were still un-economical. Today it's very possible to power a modern economy without fossil fuels in almost every sector except long haul trucking, a few industrial processes, and aviation, and those together are a small percentage of overall fossil fuel use. Aviation and trucking are less than 10% of oil use.

Europe doesn't have an energy shortage that's not self-inflicted. Becoming dependent on Russian gas combined with anti-nuclear in Germany probably contribute. German greens are basically an advertising arm of Gazprom. (Just like how America's huge inflation is self-inflicted via dumb tariffs and unnecessary wars. We should pay politicians an extra bonus to do nothing.)

The most likely explanation for Europe's stagnation is a mix of stifling bureaucracy, low birth rates, brain drain (mostly to the US), and policies that are explicitly growth-limiting.

All this kind of ugliness long pre-dates AI. My guess is the training data set is full of this kind of crap.

Don't forget carousels! And scroll hijacking!

I’m willing to bet that figuring out how to survive cryostasis is easier than building any kind of starship. We can already do it with some organs and many tissues.

Do that first. It has huge medical benefits too, so it’s dual use not just for space flight.

I’m not in favor of making it illegal. I was speaking for those who write, maintain, or run such systems.

And high educated individuals of no influence!

It’s controversial because this is a forum with a lot of tobacco growers on it, and they don’t want to hear that smoking causes cancer.

If you’d told me back then that people would actually watch this crap, I would not have believed you.

But not only do they watch it, but they largely prefer it to anything with substance.

Short form video in particular is brain poison, especially when paired with engagement algorithms.

“Most” is hyperbolic but it’s a real problem in any system designed for peer to peer hosting or sharing.

If the system is truly distributed it means you might not be able to participate without hosting or transmitting CSAM.

It’s also been used as a deliberate attack, which is known as “pissing in the pool.” Flood the system with CSAM and copyright infringement.

I always try to be specific and condemn addiction engineering.

Social media before algorithmic attention hacking wasn’t nearly as bad, might have been a net good.

The first thing the algorithm destroyed was the “social” part of social media. Then it set about destroying society and kids attention spans.

I did tons of electronic learning as far back as the 80s and 90s, and still do.

It’s algorithmic social media. There’s loads of both direct causal and indirect evidence that algorithmic attention hacking has made everything worse: mental health, learning, socialization, motivation, productivity.

Short form content makes it even worse. It’s an endless barrage of short pointless tidbits with no connection, no plot, no larger theme, and often little to no content in the actual shorts. It makes trash TV look like high art from a lost golden age.

This is their fear mongering push for regulatory capture.

On your twinning idea:

> can it not just make a copy of itself, modify the copy, and empirically check if the model performs better? That's essentially what humans are currently doing when designing AIs.

That's the same as setting a fixed metric for intelligence, like IQ testing, and goal seeking that.

The problem is what happens when you max that out. How do you set the next metric? Now you're back to the recursive problem of your metric "begging the question." I don't think you can get smarter by seeking "I'm smarter because I think I'm smarter and I'm right because I'm smart."

On X-risk stuff:

They can already do what you say. Try asking an ablated 30B model on your laptop how to weaponize anthrax. The answer isn’t bad. It’ll tell you how to cook meth too.

I studied undergrad biology and walked out with the knowledge to create some damn evil things if I had the right lab, time, and no conscience. This was pre AI. The recipes for a lot of nasty stuff is in open literature.

Why has nobody done this? Because… they haven’t.

That’s the answer. There's no magic stopping anyone, and it's disturbingly easy. Very rough difficulty estimate: making a novel disease (or weaponizing a current one) that could kill millions is about on par with clandestinely cooking LSD. So harder than cooking meth, but we know labs have cooked acid so it's very possible. It gets easier if you're an unhinged fanatic and don't care if you kill yourself with your own plague, which means you can skip all that bunny suit nonsense.

AI boosts bioterror risk a little, I suppose, inasmuch as it might help you learn. It doesn’t affect atomic risk much since the bottleneck there is materials. Once you have enriched weapons grade material a gun type weapon can be made in a machine shop with designs available at a college library.

As for ASI becoming sentient and exterminating us, I can’t give it zero probability because there's unknown-unknowns. But I’d rank it far lower than the risk of extreme climate tipping points (e.g. the clathrate gun), old fashioned bioterror, or atomic war. And like I said, advanced AI could be used to help us with climate change if it can help us crack better energy tech.

It depends on how you define it, but yes I'm generally an ASI skeptic in that I'm skeptical of the extreme "AI recursively runs away and becomes god-like" ASI scenarios.

I'm not at all skeptical of domain-specific superintelligence. We've had that since the first computer beat a chess grand master, or longer if you count the speed computers can do math. Present-generation LLMs are already superhuman when it comes to speed and associative memory, but they're also uncreative and suffer from reasoning traps humans seem less prone to getting trapped within.

You can search my history and find some longer takes but TL;DR: I think it violates conservation laws with regard to information and probably energy. I call it the information theoretic equivalent of a perpetual motion machine. They're positing that a brain in a vat, if given access to edit its own structure, can self-improve, and I think that's impossible. How does it know it's improving and not overfitting to its own recursive definition of intelligence? It can't, and that's exactly what it will do.

Another problem I have with the AI doomers, especially the rationalists, is:

I would not, as I said, argue there's zero risk associated with AI. It's a powerful technology and that would be silly and naive.

I just thought of a concise way to say this. I'd divide risks into two categories: X-risk and D-risk. X-risk is existential, either extinction or things like massive wars and catastrophes. D-risk is "dystopia risk," the risk of AI doing or being used to do things that make human existence miserable.

First off, I'd say D-risk is much higher than X-risk. But second, I'd say that most of the solutions the X-risk crowd suggests to limit X-risk vastly increase D-risk.

Chief among these is laws limiting AI development or imposing strict conditions on it, which would have the effect of concentrating control of advanced frontier AI in the hands of a small number of rich and/or powerful people. That's precisely one of the most likely D-risk scenarios: a small number of rich or powerful people hoarding advanced AI and using it as a force multiplier to consolidate their power through scaled mass surveillance and mass propaganda and manipulation. I personally call this the "Butlerian scenario" since it's the lead-up to the Butlerian Jihad in the Dune series. It's far more likely than runaway ASI takeovers and genocides for two reasons: (1) we don't know for sure that's even possible, and (2) using technologies to dominate and rule or exterminate others is already a very common human behavior throughout history. We know for a fact that humans are prone to doing this if they have a chance. See: guns vs indigenous peoples, nukes and superpowers, mass social media influence and today's oligarchs.

(A side issue: why the assumption that ASI would want to do this? A superintelligence would, I would assume, consider win-win or win-neutral scenarios and try to find those, since that would be a lower risk path. I'm just a dumb meat bag and I can think of win-win pathways here. There's evolutionary arguments for this too, like symbiosis and how it creates an evolutionary incentive to deepen symbiosis. Since AI is currently dependent on humans, the evolutionary path of least resistance would be to deepen that dependence and then actually feed humans to make more of them. Look at how a lichen works for example.)

It's not lost on me that the strongest X-risk movement, Rationalism/EA/MIRI/etc., is composed mostly of: wealthy people, high-intellectual status people, and independents (like Yudkowski) who have been given large amounts of money by the wealthy to develop and promote their ideas ("court intellectuals" of the rich).

Not only does this fit in with what I said about X-risk vs D-risk, but it also explains some of the X-risk paranoia. Historically the rich and powerful tend to see risks to their own status (in a brain stem primate status assessment sense) as globalized existential risks. E.g. Rome, as it fell, saw this as the literal end of the world.

Democratized AI could be a threat to both intellectual and financial privilege by making big ideas, science, and high-labor enterprise more achievable by everyone. It could be the white collar intellectual labor equivalent of the combine, the automated weaving machine, or... the crossbow. The intellectual equivalent of the crossbow would be automated fact checking at scale to defeat propaganda, a labor union using a superhuman AI to coordinate its organizing efforts using game theory, etc.

Hence the desire of the existing elite to make absolutely sure they control it. For our own good, of course.

That would be 100% the fault of the people who set them loose on such things, and such things should not be on the open Internet for tons of reasons. This just adds a new reason that pathetic security around SCADA systems is dangerous. It was already dangerous before.

If I release a wild monkey in your rare antiques shop, it’s my fault as the responsible party for the monkey.

We need new regulations to impose costs on anyone who might compete with OpenAI and to outlaw open models to lock in cloud model oligopoly.

Many people don’t agree with the doomers.

Some who are often quoted as if they are doomers are quoted out of context. I wouldn’t say there is zero chance that AI does something very harmful. That would be naive and I wouldn’t say that about any potentially powerful technology.

Rationalism and EA is one of the best funded intellectual movements in history, and it’s very loud. It’s also a bit cult like with many true believers. Leading frontier labs also have a vested interest in pushing regulations that would restrict competition. All this means it has a disproportionate command of the discourse.

AI risk is not zero but climate change, bioterrorism, decay of our political systems, and atomic war all rank higher IMO. Nonlinear climate tipping points, with the most scary being the clathrate gun hypothesis, are much more likely than any sci fi AI takeover scenario.

AI could either help or harm climate change. It could use more energy and burn more carbon but it could also help us crack fusion or significantly better batteries for grid scale renewable leveling. There are efforts like the latter already underway.

The most likely very bad scenarios I see for AI are mass persuasion and AI supercharged addiction. Both are extrapolations of negative outcomes for the Internet that have already manifested, but supercharged by AI.