Hacker News Reader

Comment by mdhb | Hacker News Reader

There’s really no parallel whatsoever between pushing code to production and a military operation like this. It’s a terrible analogy.

View on HN

I agree that it’s not a great analogy but would caution that some of the same concerns about complacency apply. If this tool produces some decent looking output, it’s not hard to imagine a mistake being okayed because it looked good similar to someone hitting “LGTM” on Friday afternoon because it’s been fine the last 500 times OR miss something else because they were assuming that the AI would flag the important areas.

I know that people in the military are very familiar with problems like that so I think the main concern here is making sure that vendors don’t successfully oversell their products’ capabilities.

It's not hard to imagine, but then the military already uses loads of automation that that applies to. I don't think this increases that risk.

At the very least, it would increase the risk by expanding usage. I think there’s an additional angle, however: the better an automated system like this gets, the more people assume it’ll always work. Tesla drivers get in FSD crashes they’d never get in with a basic cruise control because it’s good enough to handle enough road situations that they assume it can handle all of them, despite periodic fatal reminders that it isn’t.

I get that unfamiliarity can breed trust. Same as drivers who blindly follow GPS into a lake. But I don't think the military should stop using GPS.

There are parallells to human complacency, and there have been military incidents due to it.

Reply on news.ycombinator.com