Hacker News Reader

Comment by NoblePublius | Hacker News Reader

No, because HAL isn’t a person. The people who programmed HAL likely committed manslaughter, though it depends on which state they did the programming.

View on HN

My first thought. I'm a licensed Professional Engineer. If I build something and "stamp" it for approval I am legally held responsible for negligence of work.

https://www.nspe.org/resources/ethics/code-ethics

This is something I've been wondering about the Software Engineering profession. As software becomes more "blackbox" or AI if you will should this profession be required for licensure as well. Especially if the software design is used in public service.

NASA has lost multiple extremely expensive missions due to software errors despite investing hundreds of millions of dollars and dedicating entire teams to software safety.

You want to put a guarantee stamp on something at the risk of going to jail for missing a bounds check at some point in your life? Go ahead, I'll pass. This kind of requirement would kill the industry overnight.

Software is not engineering. Software engineering is a hilarious oxymoron, and I say this as an accredited engineer. There is no way to guarantee that software will work 'properly', and 'properly' is only a small subset of 'as intended' which is a small subset of 'as envisioned'. It could be that software never becomes engineering. And until it does, holding it to the same standards as engineering is pointless.

> Software is not engineering.

It can be. There are ways to prove a program is correct.

There are ways to prove the program will reach certain outputs within a certain envelope of inputs under a certain set of constraints.

This in itself is rather limited. For example, a formal prover would know nothing about something like rowhammer, and would happily accept code that seems to work as intended but also executes a rowhammer attack on the side which completely changes the output.

Ignoring that, this definition of correct is entirely divorced from the level where requirements exist, which is as envisioned. An error in requirements or an error in design can still very well yield formally 'correct' code.

There's no way around this today, and there's a pretty strong argument that this is an inherent problem to which no solution exists. This is precisely why NASA has lost missions despite being one of the most prolific users of formal provers.

They proved the code works as designed, but that says nothing of the design being correct, which says nothing of the intent of the design being correct.

Also see: https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

I think saying software can't be correct because of things like row hammer is like saying a wood structure can't be safe because of fire or termites.

I think myopically selecting the easiest to dismiss part of what you're replying to, a part that was a deliberately simple example of where provers can and have failed, is not arguing in good faith. The rest of the argument stands even in a hypothetical world where provers are prefect.

My argument holds even when you extend it to the rest of what you say.

Mechanical and electrical systems can also be built correctly to an specification that's ultimately inadequate.

If being able to kill is part of the design, then you aren't negligent if it works as intended.

This is a "carbon fascist" POV - to borrow a line from Ian M Banks.

Q is, is HAL a person

Reply on news.ycombinator.com