lxnn

> Anthropic in particular has developed this almost Orwellian like veil of condescending rhetoric that on the surface suggests they're looking out for you while underneath they're taking actions that suggest they do not trust you, Mr/Mrs Ordinary Person. All the safety rhetoric, never supporting open weight models, the lockdowns on harnesses outside claude code, etc.

This line of reasoning is really very silly. I can be looking out for my young child by choosing not to put them behind the driving wheel of my car.

There are both stupid and malicious people in the world. If you have a tool that you believe may be genuinely dangerous in the hands of the wrong person, it is not the smart thing to do to give the public unrestricted access to that tool. Do you think it would be sensible to give every individual on Earth a nuclear weapon to do whatever they want with?

Note that we only got to observe outcomes in which we didn't die from nuclear annihilation. https://en.wikipedia.org/wiki/Anthropic_principle

I hope they are out-competed by a better alternative.

Technically, I think it's not that instrumental goals tend to converge, but rather that there are instrumental goals which are common to many terminal goals, which are the so-called "convergent instrumental goals".

Some of these goals are ones which we really would rather a misaligned super-intelligent agent not to have. For example:

- self-improvement;

- acquisition of resources;

- acquisition of power;

- avoiding being switched off;

- avoiding having one's terminal goals changed.

You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.

As the pre-amble to the statement says: they kept the statement limited and succinct as there may be disagreement between the signatories about the exact nature of the risk and what to do about it.

The emergence of something significantly more intelligent than us whose goal are not perfectly aligned with ours poses a pretty clear existential risk. See, for example, the thousands of species made extinct by humans.

The problem is that's unilateralism.

Why are you so confident in calling existential AI risk fantasy?

I think Mastodon solves this problem to some extent, with their federated way of doing things.

It's essentially when information that was used to identify the suspect is presented as confirmation of their guilt.

> The witness described a tall, white, black-haired man with a tattoo on his right ankle. Only .001% of the population match this description, so there is only a .001% chance that the suspect is innocent.

Of course, this ignores the fact that they compared the description against a database of one million people, among whom you would expect to find 10 coincidental matches.

For the same reason more data is required to select a model than to validate a model in machine learning.

Also, the way mathematics is taught at high school might not prepare you well for the way you use it at university.

Maths in high school focuses more on applying methods (taking a derivative etc.); maths at university (even on a CS course) is more about constructing mathematical models and arguments.

It can almost feel like a completely different subject.

Yes. It is odd that in the British system at age 16 you can choose to study mathematics and physics and literally nothing else. (I blame this for my poor history knowledge.)