Hacker News Reader

Comment by samcat116 | Hacker News Reader

During my civil engineering studies there was an emphasis on ethics throughout classes, even the more specialized, technical ones. This is one huge difference I've noticed compared to the software engineering field. There aren't these governing bodies such as ASCE or NSPE with these ethics canons. I suspect we'll see that change in the next several years.

View on HN

The reason the engineers are given ethical training is because engineers (like accountants, appraisers or similar people) have historically had an institutional independence from their employers. A civil engineer in particular can choose and will not to sign off on a given project that they know will be unsafe. Given that engineers are certified by their profession, their employment situation is not precarious and so they can reasonable make this refusal (this is degrading like all professions naturally but still).

Programmers have no such institutional support. If a programmer refuses a job, it goes to someone else that's it. Programmers may have ethics but ethical training a la engineers isn't going to give them any leverage for choices.

Exactly! You seem to get it, the issue is not the lack of knowledge of ethics, but the lack of power. That's one of the many, many reasons why strong unions are so important and in this country we see the effects of not having them.

Without protection for people refusing to do bad things, you create a system when there's always someone desperate, hungry or unethical enough to do things that shouldn't be done.

This sounds reasonable, but I don't quite understand that last bit. If a civil engineer refuses to sign off on a project, won't the same thing happen? Why won't the job go to the next civil engineer down the line?

It's a collective action problem - the company can't fire the first engineer if they know everyone else they hire would also refuse orders. But each individual engineer has an incentive (not getting fired) to break with the group's strategy.

This collective action problem is solved by coordination, through the means of the licensing body. That body can impose severe penalties (not just firing you from your current job, but from all future jobs) for anyone who betrays the group strategy, so an individual engineer can feel some more safety refusing orders in the knowledge that the whole profession will back them up.

EDIT: In civil engineering, this system is propped up by the state, which requires plans to be signed off by a licensed engineer. The guild functions in this capacity as a subcontractor of the state, taking on a regulatory burden and allowing rather more severe punishments (barring someone from a profession) than would be acceptable from a purely state organ. In software, this could be enforced by similar means for safety-critical applications - the ACM, for example, could be required to license any software engineer, with the understanding that they would revoke licenses for negligence or malfeasance that didn't rise to the level or criminal liability.

While skilled software engineers are still rare enough to be coveted, programmers are a dime a dozen these days, many desperate to get their foot in the door to break into the industry. Licensed engineers, on the other hand, are hard to come by. Professional engineers spend years in school, then years training under a licensed engineer, and then finally take an arduous test with an abysmal passing rate to get their license. They aren't about to risk their entire career to satisfy some asshole manager, because the entire business depends on the engineer, not the other way around. It's a small community and word gets around.

The problem with programmers is rather incompetence than obeying unethical instructions. It’s not a business manager who tells a developper to concatenate a SQL string rather than use a parametrised query, to leave a datastore wide open rather than setting up a permissioning, to store passwords unencrypted or not validate inputs from the client.

Yeah, there is a handful of cases where software developers have been given bad instructions from their management, and perhaps Boeing is one of them. But the real problem is developers being unaware of the most basic good practices.

The reason the engineers are given ethical training is because engineers (like accountants, appraisers or similar people) have historically had an institutional independence from their employers.

Imagine the state of the tech world today if all of the "engineer" programmers at Google, Facebook, etc... practiced at the same ethical level as actual engineers.

Seeing how google is firing people for trying to unionize... I would guess there would be less spyware in ads?

Actual engineers make weapons of war; do you think they would not also make adtech?

Balancing that out, of course, is that professional engineers (I'm talking about PEs here, not programmers) can be held liable if they sign off on something they knew, or should have known, to be defective.

They would be held liable even if their boss ordered them to do so.

It's a very different set of incentives than we have in software, but maybe its time we introduce real PEs into software development.

Membership of the ACM has a code of ethics https://www.acm.org/code-of-ethics

I wonder if the MCAS software coders are held to any ethical safety standards by Boeing's engineering department or are they so siloed that it would be impossible.

They were held to the standard of "Get it done, yesterday."

And will likely eventually be held to the standard of "How dare you let this happen?" as they're fired, to demonstrate how seriously Boeing takes safety.

The programmer is told to implement a spec. They didn't write the spec. If the spec is flawed, do they even have the necessary domain knowledge and experience to identify those flaws? And, if they can identify them, and they speak up and nobody listens, what do they do next? Implement it anyway, or quit and let somebody else implement it instead? Especially if they don't even work for Boeing, but for some subcontractor.

The logic follows that if a malicious company deliberately subcontracts all their development, then they've completely indemnified themselves against all claims. While simultaneously providing no opportunity for subcontractors to bring noticed issues to their attention.

Which seems a pretty bad state of affairs to trust when building things like airplanes.

It is. that's why strong labor laws protect everyone, not just the workers, and why well run Unions are so fundamental to the sustainability of an economy.

That is why high risk industries have regulators like the FAA instead of just relying on the fear of liability.

In terms of liability, Boeing can try push it to the subbies and the subbies can try and push it back to Boeing. Both are trying to bamboozle the non-technical lawyers.

The purpose of the FAA is to cut through that crap and enforce actual, effective change through sanctions or otherwise, and they didn’t do that. That’s what fell apart here.

To wit:

In my state, Gambling is legalized. I remember my surprise when a friend who worked in the compliance side of the business actually knew what MD5 was (Back in 2007.) She wasn't a 'technical' person either.

She explained that they actually had to audit the slot machines to make sure that the code running on them had a hash that matched a codebase that had been audited and approved by the state regulatory body.

So, the practice for auditing code by a regulatory body is nothing new. If we do it for money, FFS can we do it when there are actual lives involved?

Someone weighed that the short-term money here was better than long-term. I think that quarterly earnings reports is the wrong optimization for some industries. If only the execs at Boeing could imagine each of their planes carrying suitcases of their Bonuses.

As for the FAA, they appear to be in the throes of regulatory capture on this issue.

https://www.economist.com/business/2019/03/23/regulatory-cap...

Which is why it is so important to keep safety-critical software in-house. Because then the developers shouldn't be limited in knowledge about their prime domain, software engineering, but also about the domain the software is used for. So real aeronautic engineers need to be part of the software teams. People who would at least have a chance to detect fatal flaws in the specs they are supposed to implement, if they were not part of the spec creation from the beginning.

I haven't seen any evidence that the coders were doing anything other than accurately implementing the system design that the control systems engineers dictated.

What ethical safety standards would you have them implement?

I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.

I don't implement anything I can't/don't understand.

I'm a generalist, so I make it my business to know a bit of everyone else's business. If I can't look at a spec without seeing issues down the chain that the spec makes no mention of, I end up feeling that it is my duty to make sure to raise the question until I am satisfied with the answer.

I don't always get the most satisfying answer, and I haven't had to put the career on the line by doing so yet; but I'm prepared to do so nevertheless.

I will not be part of the next THERAC-25/MAX fiasco. And if I've learned anything from this decade, it is that engineers as a whole may need to organize against those that would seek to have us do unethical work.

It wouldn't stop the practice, and God help me, I don't want the field locked behind accreditation/licensure...

However, I don't see any other defense or measure that would allow for putting the kebash on bad work. There has to be a price for bad corporate behavior in terms of ruthlessly pursuing performance that can only be met through wink wink nudge nudge style inducement to unethical behavior. At least, no way besides publically outting a company's dirty laundry. That really isn't satisfying though, because that requires a sacrifice of somebody's integrity every time, and no one wants to touch you after that.

I just can't converge to a satisfying middle-ground with the right incentives. Besides maybe anonymous whistleblowing to an appropriate watchdog agency. Even then though, issues are raised in that you are leaving the regulation up to people who feel insecure reporting something when they have everything to lose.

It is a frustrating issue to say the least.

I've worked on systems in 3 categories:

1. Safety

2. Non-Safety

3. "Safety"

For #3 I mean it's "we realize that failure has bigger repercussions than a fail-whale, but we can't afford to do any of the ISO processes that have been proven to work." Sometimes I feel like my only job on those sorts of systems is to bang the "Normalization of deviance is not okay" drum in every meeting.

All failures need to go to the PM and get signed off on, otherwise the PM has a false sense of the actual reliability of the system. If the PM wants to get more budget for safety concerns, they should be able to hand a stack of 100s of pages of papers to whomever controls the purse strings and say "These are the failures in the last N days" If all they can say is "some of my engineers have expressed concerns" then 0 change will happen.

> I assume that you would agree that the control system specification is the ethical responsibility of the control system design engineers.

It is the primary responsibility of the control-system, but there is still also a responsibility with everyone who interacts with that spec to speak up if any flaws are noticed.

One of the big things that tight deadlines do is give tunnel vision to the engineers, so "just implement the spec" becomes the goal and the forest can be missed for the trees.

There were probably dozens of engineers that saw the MCAS specification as part of their duties; here's a few possibilities for what happened:

1. Nobody considered the case of improper MCAS engagement under normal flight conditions; this should clearly qualify the system for "Hazardous" classification under DO-178, which would require redundant AOA sensors.

2. Someone considered this case, but didn't speak up (was very junior, or it was way out their specialty).

3. Someone spoke-up, but was told by the person they spoke to disregarded it for the same reasons as #2, so it never made it to the control-system design team.

4. Someone spoke-up, it made it to the control-system design team, and business pressures caused the concern to not be investigated.

#4 would be significant ethical issues for the control-system design engineers, but I think it to be unlikely compared to the others.

#1 can be indirectly caused by time pressure. The certification process is supposed to slow things down, but there is some indication it did not sufficiently do so in this case.

#2 and #3 show ethical lapses outside the control-system design department, and are not just isolated to the individual in question, a safety culture needs to include cultural norms of speaking up about potential problems even when you think you are wrong.

Seems unlikely given that they were likely contractors: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...

You don't need to take an ethics class to see that cutting corners when human lives are involved is a bad idea.

I think I heard somewhere that the entire purpose of Engineering as a whole, is to know which corners are safe to cut.

"Anyone can build a bridge that stays up. Only an engineer can build a bridge that barely stays up."

(I mean, it's obviously exaggerated for effect, but still.)

I took ethics courses as part of comp sci. Definitely left an imprint on me.

I don’t know if ASCE or NSPE are governing bodies as much they are societies. I think the Core of Engineers has way more power there from my experience in Civil.

For Software that would be Homeland, FBI, and even the FAA for governing bodies specifically. But software doesn’t have ethically governing bodies because we have enough laws to cover a lot of mandatory auditing.

For example: HIPAA - Medical Data Protection & Compliance Gramm-Leach-Bliley Act - Financial Data Protection & Compliance FISMA - Federal Data Compliance GDPR - PII Data Compliance PCI DSS - Credit Card Data Compliance

Specific example of use: Code that is deployed that involves PCI data being handled requires a code review.

There are heavy penalties for lost of these things, and I promise there are many companies focusing on this. But it could be very much improved with a proper governing body.

Reply on news.ycombinator.com