Hacker News Reader

Comment by bcrl | Hacker News Reader

This is why I think that we're getting to the point where software developers need to be trained in ethics and licensed in the same way as professional engineers are around the world. The people building the tools need to be held accountable when design choices kill people.

View on HN

Most software development is not much like engineering. That's probably the right choice for low-stakes consumer products, but wouldn't be accepted in avionics and shouldn't be accepted in medical software that impacts life safety.

The hospital software that I'm familiar with is absolutely terrible and would never have been accepted in an avionics environment. It is an endless pile of nightmarishly wrong UI/UX choices coupled with opaque documentation and updates that confuse the users. They also make money hand over fist.

Ethics is applicable to many situations outside of life critical systems. Engineers take ethics very seriously because it is important, and I think it is important for software developers as well. The first time I worked closely with a Professional Engineer on a project, it was eye opening as to how the concern of doing things the ethical way shaped the interaction... Everything from identifying potential business conflicts to ensuring data collection for inputs to the design process was done correctly and traceable.

Corporations already push enough random HR requirements on tech employees. I can't see how having a semester or 2 of ethics courses is particularly onerous.

I think including ethics in the curriculum at both the high school and university level is a great idea. I do not think requiring software developers to be licensed is a good idea at all.

Safety-critical software projects should have a licensed engineer in a supervisory role.

Licensing takes many forms. Some are onerous, some are not. Working At Heights training takes 0.5-1 day of training every 3 years where I live, and it's mandated because it saves lives. Yes, it's a form of licensing, but it's an easy box to check because it's accessible.

For example: running experiments on humans without their prior consent is considered unethical. However, large tech firms routinely run A / B tests on their users without providing Informed Consent. If software developers were trained like engineers, they would be ethically obligated to obtain Informed Consent prior to engaging in this kind of conduct. More importantly, when software developers realize they were being experimented upon (as happened in the Linux kernel community a while back), they were justifiably outraged.

I know this isn't going to be a popular position to hold here, but there's a lot of harm being done by unethical practices that are currently widespread in big tech. Ephemeral ads that prey on the elderly would be considered Not Good by anyone that has seen a parent fall prey to them, yet there doesn't seem to be any concern whatsoever amongst the industry giants themselves to prevent this practice. So long as an ad brings in money, it's good to run seems to be the bar for advertisers at present, and I don't think that's good for society.

When there are no ethical considerations given to the consequences of an action, unethical outcomes are inevitable. Fixing that starts with learning, which is something every great developer already does.

> More importantly, when software developers realize they were being experimented upon (as happened in the Linux kernel community a while back), they were justifiably outraged.

Wait, what happened here? Do you have more details?

I didn’t read the whole thing but the intro seems even handed: https://hendrik-erz.de/post/lying-to-your-research-subjects-...

Name a consumer product that doesn’t impact life safety.

Phone notifications do. Logging out software while someone’s driving does. Every button on your car impacts life safety. Every interlock in your kitchen does.

Remember, each of these things gets rolled out 100M’s of times, so one-in-a-million scenarios kill 100’s of people.

Modern engineering exists because of the people that were maimed or died to give us the knowledge that lets us live the way we do today.

Bureaucracy won't help anything. You can't train people to be more ethical.

Bureaucracy and accountability are two separate concepts.

Doesn't the FDA already regulate medical software?

To an extent, but only if the software meets the criteria for being legally classified as a "medical device". And at the lower classification levels the regulations are very light. You don't necessarily have to prove any level of reliability or usability.

https://www.fda.gov/medical-devices/classify-your-medical-de...

Ex employee of the division of imagine devices and software reliability here. I have nothing to add, and I'm not qualified to confirm or deny your comment. Just interesting to see a mention of my past life

But thats part of the problem, as well as part of the solution. The FDA regulations say the software must do this and must do that, and you end up with thousands of pop up alerts. I think the regulations are well meaning but they way they translate into software is not realistic. In the article an FDA-regulated pop-up is part of the story.

There are no FDA regulations that require EHR software to display pop-up alerts for possible sepsis.

It is doctor’s responsibility to know the patient might have such a condition, software system is just for helping

Reply on news.ycombinator.com