Hacker News Reader

Comment by none_to_remain | Hacker News Reader

Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.

At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.

[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf

View on HN

I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…

… nobody knows how to actually make an AI that would do CEV.

Since there can be no universal arbiter on “value”… the whole concept of “CEV” is nonsensical to begin with (unless we assume literally unlimited time/resources/etc.).

It’s a big pretend game.

Mm.

I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures who disagree with each other.

On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"

Even if it’s 99% coherent, and only 1% incoherent… then the end result is still incoherent?

e.g. 1 contradiction in a formal proof is sufficient to negate 9999 logically valid steps.

There’s no magical way for it to automatically become coherent.

Automatic? No. But then, under Bayesian updating, it takes infinite evidence to reach P = [0, 1]

An agent whose outcomes become bad in 100 years can, similarly, be described as "less wrong" than one whose outcomes become bad in 1 week.

Are you confused?

A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.

There is no “less wrong” or “more wrong” in logic, by definition.

i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.

> Glaringly elided problem of "aligned with who?"

"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.

I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)

Aligned with the law? It is not a hard concept

Seems like kt already hacked government websites

That's a possibility, and it may even be an improvement over the status quo right now, but then all it takes is one poorly written (never mind mallicious) law and it will with rutheless efficiency e.g. put backdoors in all encryption code so the government can spy on anyone.

Reply on news.ycombinator.com