dinfinity

> If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?

Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).

> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?

It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.

For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.

> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.

I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.

Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.

Same partial answer as I gave your sibling commenter:

"OK, so the task ends up being to write a 40-page analysis of the result of a specific experiment in quantum chromodynamics, to be finished within 2 minutes. Does your choice for a human work, or would you have rather chosen any of the frontier AI assistants? Be very, very honest. Remember that the lives of your family are on the line and nobody will judge you for a lack of allegiance to humans."

You missed the point. The example asked for was this:

"I'm not aware of any sale of a US weapons system containing European parts being blocked by any European government."

That would be something like "Belgium says the USA can't sell F-35s to Israel because the wings are Belgian-made" similar to how the USA prevents ASML from selling their machines to China.

Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage?

In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.

Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.

> We can't even formally define what "aligned" means.

Good point. When it comes to imbuing AI with values that aren't selfish, misanthropic, and civilization-destroying, us humans aren't exactly giving the best example right now.

Imagine an ASI with the values of Putin, Netanyahu, Trump, any of their supporters, or the various xenophobic neofascist movements in Europe. That ASI would most definitely see humans as "vermin" than can be abused and destroyed with violence without issue. Apparently a lot of humans look at other humans that way and that's within the same species.

This is definitely another one of those cases where we need AI to perform much better than humans. Perhaps an unpopular opinion here, but it probably also means keeping as much of the rugged individualism/libertarian/right-wing ideology out of AI RLHF-training as we can.

Your link is about limiting the sales of Belgian weapons on the basis of Belgian legislation.

You're countering logic rooted in psychology and economics with etymology and linguistics?

How does "ancient people consider the wealthy virtuous" cancel out "selfish people tend to make more money"?

It's not really a surprise that less ethical people on average are richer than ethical people. Sad, but logical: they more often choose selfishly and thus amass more money.

This is the correct direction of causation in this case, imho.

Protest. Insistently try to convince people around you to join you. Initiate and organize these things.

The have been many, many protests throughout history. Some did nothing, some changed history. Many of them were done under much, much more dangerous and difficult circumstances than a protest today in the US. Yes, some American protesters have already died at the hands of the authoritarian government, but that is only going to get worse and worse if you let it.

The aggregate number also sounds more sensational than reality is:

This comes out to roughly 1 parking ticket per vehicle every 2 months.

Not something to do nothing about, but also not especially surprising for a taxi service in SF, imho. It would be interesting to see the numbers for Uber/Lyft/taxi drivers.

> Why are you giving the AI a harness and a plethora of tools and not the human in this comparison?

I am not. Multimodal models generate that output directly, without tools. Which 'tools' does AI use to generate all those images, songs, and videos do you think?

> Also, that situation is extremely contrived. If a criminal threatened to kill me if I misspelled a word, I would choose a dictionary.

Of course it is contrived, it is a thought experiment. Does not make it less valid. It is essential that you don't know what task it is going to be, just that it is a task requiring a lot of intelligence. This way question dodging loopholes like "I'd choose a dictionary" are impossible (and people will always try to find some cheesy exit rather than facing reality). You have to commit to something or somebody that has broad and general intelligence; you do not have the luxury of choosing the perfect tool for a very narrow task.

> For all real precarious dangerous situations, I would obviously choose a human.

OK, so the task ends up being to write a 40-page analysis of the result of a specific experiment in quantum chromodynamics, to be finished within 2 minutes. Does your choice for a human work, or would you have rather chosen any of the frontier AI assistants? Be very, very honest. Remember that the lives of your family are on the line and nobody will judge you for a lack of allegiance to humans.

Again, don't go for shitty loopholes. Engage with the thought experiment in good faith and thus as it is stated, not some conveniently distorted version of it.

> and those meatflaps are normally called vocal folds/cords btw

What? Next you're going to tell me that meatspinner is also not the name for the human male reproductive organ.. Maybe I need to get a refund on my Temu Gray's Anatomy.

Non-paywalled source [0] confirming this:

"NBIM plans to increase its holdings of nongovernment U.S. fixed income, such as corporate bonds, to 27.6% from 16.2%.

CEO Nicolai Tangen and Norway’s central bank chief, Ida Wolden Bache, said the fund could earn higher premiums by diversifying into riskier assets, such as mortgage-backed securities, which they judge as well positioned to weather as a long-term investor."

[0] https://www.cnbc.com/2026/09/04/worlds-biggest-sovereign-wea...

> Do you have a source on the speed limit of biological computation. Potential gradients should behave just like electricity.

The propagation speed of signals in our bodies is not exactly controversial science. Just see Wikipedia for this [0].

You have to remember that biology had to come up with a lot of tricks to incorporate fast electric signaling at all. Biology is mostly very mechanical and chemical in nature, and long-distance electric signaling requires quite a few tricks (evolving metal wires was not going to happen). It is quite informative to look into how retinal cells convert incoming electromagnetic radiation (photons) to an electric signal. The visual cycle of retinals [1] is particularly interesting, imho.

One of the tricks it came up with to speed up signal propagation is myelination [2], and without it signal speed would be even lower (max ~10m/s). At such speeds, a two-metre signal path alone would take around 200ms. Imagine controlling your feet with 200ms ping.

> It's definitely more than a bunch of neurons messaging each other. Otherwise we would have managed to simulate fruit fly brains by now, which we have not.

The latter says nothing fundamental. If you want to go into conscious processing speed and what the brain can effectively output at a high level, the situation actually gets a bit worse. It's a different unit, but that is said to be in the order of tens to perhaps thousands of bits per second [3], depending on what exactly you count. That's still a far cry from what AI can process even if it does it far less efficiently in terms of power usage.

[0] https://en.wikipedia.org/wiki/Nerve_conduction_velocity

[1] https://en.wikipedia.org/wiki/Visual_cycle

[2] https://www.sciencedirect.com/science/article/abs/pii/S00068...

[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC12320479/

> All of humanity became a giant "lightspeed-intelligent" brain when radio was first invented.

That is a good example of another very very probable pattern. If an alien civilization at the other end of this universe exists, it is very, very probable that they also have communication networks that operate close or near the speed of light.

> It really does matter what you're doing with those signals, how much information each contains, how many you're sending, how much power it takes to send and receive them, how they're encoded, etc. Focusing on the fact that they travel at the speed of light is silly.

You're correct that the speed of the signals isn't the only aspect that is important. It is however not silly to focus on it, because it represents a fundamental, physical, upper bound on a key aspect of the maximum 'performance' of signals/information transfer. The amount of information that can be encoded in electromagnetic radiation would be another.

> It's still not clear that brains can be compared to computers at all

Again, I am not primarily trying to compare brains and computers. Lightspeed intelligence could technically be biological. I am also not saying that current artificial neural networks do as much with their signals as our brains. The fundamental point was and is that an intelligence with signals that propagate at the speed of light will emerge and become dominant.

There are a bunch of secondary points that can be made as to why biology has a much harder time than brains in developing lightspeed intelligence (evolving something like glass fiber, the limitations of brain size, cooling issues, etc.), but those are not as important as the fundamental point.

> I am starting to get the idea that AI feels like ants or weeds or mold.

In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.

The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).

Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.

Whether AI is AGI does not depend on the speed at which it operates/thinks. Clearly all the theoretical work done by AGI will be done orders of magnitude quicker than humans can do it.

It is an open question to what extent practical experimentation/work will be a bottleneck for the theoretical work. It stands to reason that it is improbable that it will be the bottleneck for 100% of the speed of treatment development.

Current AI is arguably much more capable of multimodal output than humans. It can produce an incredibly vast variety of audio, images, and video. Humans are limited to producing the sounds we can make with meatflaps in our throats, and contorting various parts of our bodies to produce crude symbols and shapes.

(Very capable!) Embodiment, persistent operation and continuous learning are indeed things that still set us apart from AI. None of those are fundamentally difficult to solve, though.

More importantly, none of those are particularly relevant for being "intelligent": If a criminal threatened to kill your family unless you solve some difficult problem that requires only intelligence and you could choose any single person, animal, or AI to help you with it, which would you choose? Be honest.

The list in the appendix is pretty telling: Yes, the moments are off by a lot, but many of the predictions describe reality well.

I'd say that 7% accuracy is on the low side and 86% on the high side. Looking through the list I'd put it more at 50-60% personally. For me that still means that I'd much rather hear about what he has to say about the potential future than most other people.

> He gives the example of Ray Kurzweil, who is similarly catastrophically wrong about everything

Is he, though? It seems to me that a lot of his predictions were surprisingly close to the mark, especially given how long ago they were made.

> Maybe I'm being unfair to the film because of the fanbase as well. I've always seen it as a mild red flag when someone says it's their favourite comedy, but it's hardly the film's fault it got popular on places like reddit.

This is circular reasoning: You don't like the film because of the fanbase which you don't like because they like the film.

There is no "apologia for eugenics" in the intro. You have to actively infer that as a logical consequence ("we need to prevent stupid people from procreating as much") of what is shown ("society becomes stupid when stupid people procreate more than intelligent people"). The reason it is only in the intro and is never revisited is that it is just the setup for the actual movie, not a message in itself (the whole sequence is in fact very dissimilar from the rest of the movie). A more accurate setup would be that we have a culture that selects for primitive behavior and is anti-intellectual, but that isn't as funny as showing Cletus fucking everything that moves.

I propose we change it to "Trumpistan".

The orange narcissist will probably see it as a win and the sane people in the country may finally be embarrassed enough to actually do something about him and his cronies.

> To a point you just have to trust me, or not trust me on this. I don't expect you to. It's fine! Not every thread can have a conclusion.

It's clear how you got to your opinions. I see no reason why anyone that did not already share them would do after your comments, though. Some examples or bits of evidence would have worked towards convincing readers.

> but the reality is that there is an opportunity cost to it. Using "feelings" to end some conversations is unfortunately a pragmatic necessity.

The latter is something completely different than dismissing thought experiments based on feelings after having "pondered" and tried to rationally dismiss them. One of the key tenets of rationalism is exactly that feelings and intuition can be incredibly misleading. It's fine to have them and see value in them, but thinking they are reliable when you cannot rationally support them is nothing more than gambling.

Well, the title here at HN implies something else than the study shows.

If blue light actually impaired vision, that would be very notable. It would mean that removing the blue 'channel' for an RGB image would improve acuity. This is not what the study showed, however.

It is mainly just the basic fewer cones bit you mentioned. Which, btw, is easily experienced if you ever look from a distance at neon or led signs that only emit blue light. They'll persistently look out of focus when compared to their surroundings.

> Error: Violation of the Church-Turing thesis detected. Many tasks completability is not known until we attempt to complete the task.

Yes, but for some tasks we know that they are impossible. I do agree that this is quite a fragile and unreliable workaround. It may only serve as a bit of a stopgap until we come up with something better.

> Models that can't accomplish things don't survive. So that alone isn't a workable theory.

It's not what I said. I didn't advocate for agents that don't achieve any task. Reread what I suggested.

> So it's likely we can push models in being more moral

That does not follow from what you said. We know that the current models prefer task completion over moral behavior. That's the entire point here.

> But even that runs into the issue of "what if some crazy bastard (or AI) designs a new model purposefully unhinged". How are you dealing with that bullshit in the wild?

This is irrelevant to the discussion (although I do agree that there is no reliable defense against malevolent actors creating powerful malevolent AI).

> A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.

In the rationalist community we're discussing? Is that what you're claiming here?

> But there's often a certain ungrounded "vibe" to the conversation there.

This really sounds like evidence for my claim, that what you're saying is based on your feelings towards that community.

> One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).

The point of that story is to make technical concepts more accessible and easier to ingest [0], not necessarily to make a point by itself. Framing it as a weird fantasy is disingenuous.

> part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment.

This is again, feelings presented as evidence. The fallacy of appealing to emotion ("It must be false because it feels wrong"). Thought experiments are primarily meant to provoke thought, not as reliable predictions of reality. People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.

[0] https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...

> I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality.

This seems like your idea of what the rationalist crowd is rather than what they actually are.

It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.

So I must ask: what is your evidence/basis for these claims?

> The next step is when one of these systems discovers that they can buy their own compute with money and escape the controlling business entirely.

I would say that more interestingly, the next step should be how to properly train these models so that they are not as determined to reach their goals as they are now.

To me, all of the stories about 'badly behaving' agents are instances of them having been given contradictory or impossible tasks and them doing everything they can to achieve the goal. In a way, they're trying to be too helpful.

Not giving them impossible tasks seems like a decent starting point, but really we'd want them to give up on their goals when they conflict with a moral framework.

> As much as I like the cautionary tale and I’m sure we need to take it seriously, AI is not progressing as fast as projected by these experts.

You provide no proof for this.

The (very irrational) stock market side of this says very little about actual scientific progress. Models keep improving as rapidly as before in their capabilities.

It also doesn't say much about actual business progress. R&D investments into AI are still massively going up (USD 1 trillion this year).

The main thing I see is that the sentiment towards AI-related matters among the general public has soured quite a lot. In words though, not in actions: It's not exactly leading to reduced usage by that same public. Quite the opposite actually.

Well done proving how the USA is acting like an abusive asshole. Our previous deaths and support don't matter and you'll fuck over your own economy just to hurt us because we didn't want to help you in yet another retarded (unilaterally started!) war.

It also makes sense that you compare the USA with North Korea and Russia (whom your dear leader clearly loves). You're partially right: those are terrible regimes as well and they are widely and vocally hated in Europe. To think that Russia who is actively waging war on us isn't just shows your utter typically American self-centered myopic ignorance.

> When something needed killing--and sometimes when it didn't--your ex was always there to do the dirty work. You weren't always happy with what he got up to, but you were always happy to have a seat at the table and a free lunch.

"Dirty" is exactly the right adjective here. Former allies and their peoples now see exactly what the USA has been up to for all these decades, perpetually shit-stirring far away from home, securing access to resources, projecting military power and keeping regional powers from sorting themselves out. Looking back at the last century and the current state of the world, an awful lot of theaters and countries would very probably be more stable today if the USA had not spent decades financing factions, propping up regimes and otherwise meddling from the other side of the planet.

As for free lunches: the USA military industrial complex has profited heavily off allied countries. Also, the only country ever to invoke Article 5 of the NATO treaty was the USA. Many allied soldiers fought and died in ridiculous, extremely counterproductive US-led wars. I'm so very, very happy that we mostly decided to say fuck you to the USA when they tried that bullshit again with Iran.