Microsoft President: We Need a Hippocratic Oath for Software Engineers
capitalandgrowth.org
capitalandgrowth.org
The Hippocratic oath sounds more altruistic than the alternatives, but good legislation, including business audits and incentives, will have far more impact than a software engineer swearing they won’t be evil.
But ultimately there's always a software engineer involved in the creation of software - and that's not true of any of the other roles you mentioned. Since software engineers are necessary and sufficient to produce software, they should always be held responsible, and any oath should fall on engineers.
> To say it’s on the engineer to do no harm puts them in the tenuous position of doing the job or being replaced by someone who will.
Well, yes - if there were no tradeoffs there would be no point in having an oath to begin with. But there are software engineers today, including some on HN, who do things more harmful and unethical than medical malpractice, and they are personally culpable for the decision to do so - just as their replacements would be if they refused. I would also like to see laws criminalizing those individual engineers' conduct - maybe you're alluding to the same thing? - but an oath is a good start.
I've quit jobs in the past because of ethical concerns about the way in which those above me have been acting. In one case this involved bribery of senior government officials to push through a project that put at risk the privacy of hundreds of thousands of people.
If you go along with shit like that, you're an accomplice and share partial responsibility. As professionals we have a responsibility to stand up for what is right. It's not good enough to fall back to the lazy excuse of "just doing my job".
I agree to a very limited extent about the hierarchical nature of a typical corporation, but I also disagree. Software engineers at a certain level of their career and with relatively uncommon skills can pick and choose what companies they want to work at. In my opinion people of good moral character and conscience need to be prepared to refuse to accept a position at companies known to engage in activities against their principles. And further need to be prepared to resign if they are asked to do something clearly unethical.
From my particular specialization in network engineering, I would never accept a role at an ISP in an environment where I had to implement something like the GFW in China, or further walled-garden/censorship of the global Internet. It's directly contradictory to my principles. I sincerely hope that the best and the brightest of my colleagues would never choose to aid and abet internet-fuckery by autocratic regimes. If people from my field look at a project and could reasonably say "Vint Cerf would be really disappointed if he saw me implementing this...", I hope they will choose to walk away.
the idea of Hippocratic Oath reminds me of Asimov's Three Laws of Robotics in "The Naked sun" (SPOILERS ahead): the detective realises that the normally quoted First Law of Robotics ("A robot may not injure a human being or, through inaction, allow a human being to come to harm.") is actually just an approximation, he argues that the real Law is "A robot may do nothing that, TO ITS KNOWLEDGE, will harm a human being; nor, through inaction, KNOWINGLY allow a human being to come to harm."
This is important because even though robots really try their best, different robots could perform sub-tasks that look very harmless by themselves, but combined kill a human being:
- A robot is instructed to pour this bottle of poison into a caraffe of water and then leave the room
- Another robot is instructed to enter the room, take the caraffe of water and give it to a human to drink
The human is poisoned, but none of the robots are directly responsible (in the first law sense). Is the act of connecting the two dots the evil deed.
Isn't that what a hipppocratic oath would solve. They'd be accountable to the oath before their bosses, and that would give them reasonable grounds to refuse unethical work.
A system of ethics within the health system are necessary for customers to retain trust in the health industry. It's also strongly aligned with the selfish interests of workers who must enact that system of ethics. These properties do not neatly translate to software engineering—mostly because the most difficult ethical dilemmas in technology are rarely obvious when looking at source code. The problems with Facebook (for example) are not always inherent in code; many are only revealed after being deployed at scale and external groups begin exploiting the system.
This is where a strict licensing requirement, like Canada's P. Eng, can empower the engineer. If you think what you're being asked to do would violate your professional ethics, not only can you decline to do it, but you have a system to ensure that you won't just get replaced by someone who will do it.
Software engineers are accountable to themselves before they are accountable to their bosses.
- developing treatments for chronic symptoms instead of curing diseases
- being unprepared for pandemics
- making health care unaffordable except through employer plans
- promoting wrong nutrition guidelines for decades after the evidence was in
and more.
To have good outcomes you need ethics at both individual and system-wide levels.
After the 2008 mortgage crisis, Netherland required everybody working at banks to take the banker's oath, which is mostly about balancing the interests of the 4 main stakeholders of the bank: shareholders, customers, employee, and society. It's pretty broad, it doesn't magically fix everything, but it does make everybody more aware of their responsibilities. Maybe software companies should require something similar, where everybody needs to be aware of their responsibilities towards, well, primarily user data, I guess. And that goes for not just software engineers themselves, but for everybody involved in the process.
Organizations that went through true iterative process to reduce failure rate like NASA figured out that they needed to allow true authority to specific domain experts to blow the whistle and not face reprisal or suffer for it. Oaths fix nothing, you need organizational change, and if someone is going to do that, its the management in charge.
If we were to introduce an oath we would have to take further inspiration from doctors, e.g having a certification required to do the job, or, failing that at least having an industry-wide union/guild protecting the position.
This sort of thing works in other professions like medicine because malpractice can cause doctors to lose their license. Same with civil engineers. This changes things because the choice is now quitting or possibly never being able to work in the field again.
Perhaps principal software engineers in charge of life or death software should be licensed for accountability, “engineer on record”.
obviously for a hippocratic type oath to work you need the same kind of system in place for qualifying engineers that you do for doctors and not allowing anyone to work as an engineer who failed the ethics board.
They are responsible to their bosses as well as the public who are the end users of their designs/products
This was not always the case.
How the hell did we let that happened?
The nazi officers who committed most of the atrocities used similar arguments. "I was just following orders!"
I expect better from a software engineer on hacker news. You've single handedly convinced most here - through your weak logic - that such an oath is necessary.
A software engineer is more like a chemist working for the pharmaceutical industry than a doctor treating patients. And chemists typically don't have an Hippocratic Oath. Pharmacists sometimes have their own version, but it is mostly about giving good advise to patients and respecting them as human beings.
But it doesn't stop the pharmaceutical industry from being heavily regulated, and while their business practices are often criticized, the drugs that come out of it are generally safe and effective. Many countries also have regulations making important drugs (ex: vaccines) accessible to everyone.
Some health practitioners are literally bought by Big Pharma, by their hospital accountant, etc. How would an oath fix that? Same with engineers or any other discipline.
You need to make sure that everyone in the process has skin in the game. For me it's less about control (legislation) than about responsibility and accountability (assessments, eating your own dog food).
This is a hypocritic ode. If somebody is acting unethically at MS then it is management. All the innovation that is not happening because MS is abusing their position. Two times they have killed a universal software platform to preserve theirs: Java and websites. Ironically they are pushing websites now that the platform has shifted to mobile with objective c and Google's variation of Java.
>According to Brad Smith, just like it is the Pope’s job to bring religion closer to today’s technology, it is the software developer’s job to bring technology closer to the humanities.
The Pope is to religion as is the President of the biggest software company to software development. It is his responsibility, not theirs. Or does he see himself as that software developer? I guess it is more a Balmer developer and he means software engineers.
He could start by handing out software licenses / EULAS that take full responsibility for any damage the software does cause, like any other sold product has to do. Then, by business processes, management will take care of the ethical issues to minimize risks.
Microsoft executives seem more in need of lessons in ethics than their engineers. Just one example from last year:
>'We did not sign up to develop weapons' say Microsoft employees protesting $479 million HoloLens army contract
https://www.pcgamer.com/we-did-not-sign-up-to-develop-weapon...
>They build the weapons
Talking of weapons, while we speculate about what AI might be used for, Microsoft executives have literally decided to build actual weapons.
Not building weapons for the war effort is not always right. That is an intentional double negative because I think it's the most clear if you read it twice. Building weapons for the war effort is sometimes right would be the boolean negative of that statement.
>Microsoft executives have literally decided to build actual weapons.
Yep. Literally they did. Clearly all US weapons are evil in your opinion because you disagree with all US weapon usage I'm guessing? You have to combine the argument that they are literally making weapons with the fact that those weapons are being used in a way you don't agree with.
Keep in mind that most of these advanced weapons they are literally making are not designed against the current wars you most likely disagree with. They are built, to include AI, to keep pace with advanced threats from other countries. Allowing us to fall behind technologically, due to perceived moral black/white issues of current wars, could lead to a whole new world in 40 years as you make your arguments in a well protected environment. Not researching advanced topics will lead to an asymmetric fight... not in our favor... if the enemy so chooses.
Reference our usage of nuclear weapons. If you think that was evil, then you wouldn't want an evil country / group of people to gain such an asymmetric advantage. If you think it was necessary, then you want to have an asymmetric advantage when it is necessary against an evil group. Yes I recognize the inherent cyclical issue with the above statement. Either way, allowing all people to gain an asymmetric advantage while we just discard all research in hopes that others will follow is ignorant of history - war theory is a thing.
I have some friends who have worked at MSFT for a long time, about 20 years or so. There was a time when they used to talk about open source as if it was cancer (~2011). When MSFT started embracing the cancer, they didn't really up and leave. Now they are all talking about how great this open source thing is.
But even funnier was when they used to complain about Google's rampant user tracking. And then one day they added targeted ads into Windows 10. Did these people suddenly decide "enough is enough" and go and join the EFF? You already know the answer to that.
Not saying I’m in favor of this oath, just that it seems silly to distinguish different roles in the engineering process.
The broader point is that in most companies engineering decisions don't come purely from the engineering department. They are often decisions made as part of bigger projects or efforts. For example, it's probably not up to engineers in most companies whether the any of the tech giants sell to the military. If it is, it's up to people who were engineers at some point and might still exist up at the top of the "product" part of the company, but who for all intents and purposes stopped writing any code or even managing anyone who writes code a long, long time ago.
But more to the point: Why are we trying to shift focus from the wrongs large multi-nationals do to individual software engineers? Plus what would the result be if this "oath" conflicts with a manager's instructions?
Maybe we should start with Microsoft, Google, Comcast, Oracle, and similar taking an oath to do no harm, before we push engineers under the bus for not fighting hard enough against what they're ordered to do.
You have incorrect information. A an abridged or modernized version of the oath is still taken upon graduation of most American MD schools.
While it is often referred to as "The Modern Hippocratic Oath", I would argue the Lasagna oath contains significant differences from the original Hippocratic oath, and it is worth treating them as separate things.
P..S. I remember because I was like "Mmmm, Lasagna...." both times.
> Doctors no longer take the Hippocratic Oath (because it is incompatible with a lot of difficult situations doctors are placed in).
The original Hippocratic Oath is no longer used (as both the original post, and you yourself readily admit). Why it is no longer used it highly relevant to this discussion because someone is calling for a Hippocratic Oath-like thing in a different area.
The fact doctors have moved to a less idealized Hippocratic Oath should be a historical lesson, not something we should seek to emulate.
Because software engineers are the ones doing the actual work. By imposing an ethical standard on the people doing the real work the multinational executives then either accept that limitation or knowingly accept risks from intentionally violating the spirit of that limitation.
What risk? Since it's the engineers, not the executives that take any oath of professional conduct, there wouldn't be any risk for any executive. All this would get us is a legal framework for throwing engineers under the bus via an ethical commission if they do something silly like blowing the whistle on an unethical decision higher up in the hierarchy.
I'm very much for greater personal responsibility in the field of software engineering. Until not too long ago, I used to work a in a field (medical equipment) where accepting the burden of potentially catastrophic mistakes came with the job. But I also know -- based on that same experience -- that personal accountability is meaningless without organizational ability.
Unless this hypothetical "Hippocratic Oath for engineers" is backed by a "Hippocratic Oath for executives", a "Hippocratic Oath for product managers", and "Hippocratic Oath for engineering managers", (edit: or by a legal framework that requires companies to enable it) all it'll do is reduce the PR effort involved in cleaning up a mess like Volkswagen's emission test scandal to pretty much zero by providing an exceptional -- and very mythical-sounding! -- framework for scapegoating.
I mean, don't get me wrong, there are clear scenarios where I think many of us would choose to lose the job. For instance, I'll go unemployed vs directly causing someone to die. But those slightly more ambiguous scenarios are where we need to be enforcing it on a legislative level and the onus should be on ALL levels of the company (engineering and management).
SWE would need to first unionize to protect workers from being fired/deported for pushing back before anything like this is even considered. Or tackle the problem where it begins: with the organization.
Are you now going to ask for more software import/export laws?
Also, people change positions fairly often and avoid work in areas they consider to be unethical. This means that the people in a position to actually make the call are people who don't find it unethical, because the people who would be concerned about it avoided the whole area. Like, if you don't want to work in ads, you'll probably find a job in some other division, or at least not directly on something you consider unethical. The people who came up with AMP (to pick something controversial on Hacker News) were true believers who sold it to management.
But people still care about the company's reputation as a whole, and as a result you get conflict between the people not actually working on the controversial thing and the people who are, but that mostly results in a lot of drama and cynicism.
The politics is complicated. I can't think of a generic oath that you couldn't rationalize your way out of.
This "ethics" business doesn't make sense because the stated goals of the people pushing this idea aren't their actual goals. The stated goal of this "ethics" push is to reduce the harm done by software to society. As you point out, it won't work. The actual goal of the "ethics" push is to entrench a certain politically-contentious ideology in tech by branding this ideology as "ethics" and thereby making it immune to criticism.
The only legitimate binding code of ethics is the law. If a practice is harmful, we can all talk about it together and agree to enact a law against it. The "ethics" people are trying to use elevated moral rhetoric to bypass this democratic process, and we shouldn't let them get away with it.
Lol. I'm married to an MD. Doctors definitely still take the Hippocratic Oath. Perhaps some places no longer do it, but I am not aware of any of here peers that have not taken the Hippocratic Oath.
Reality, we have virtue signaling.
.... soap box
because its and end run. the idea being that if software developers had this as a whole then corporations could not force them to do whatever it is that was then decided as against the oath but it amounts to nothing more than virtue signaling
the issue with a software developer Hippocratic Oath is that you can damn well bet it will be subject to the whims of whomever is loudest on social media or whatever political group wants to use it to damage the other party.
The Hippocratic Oath is protected by history and pretty much limited to interpretation but any such oath or rule today is not worth page it is printed on.
That said, this might actually work! If a software engineer can suffer personal harm by working for a business with iffy ethics, then they are incentivized to play it safe by avoiding working for those types of businesses -- thus correcting the market by internalizing the externalities. I doubt anyone would work for Facebook in a world with a Hippocratic Oath for Software Engineers that has real teeth.
Put another way: pointing to decision makers instead of individual engineers is a simple rephrasing of the Nuremberg defense, "I was just following orders!" It is obvious that we should hold leaders accountable. The question here is whether we hold individual software engineers accountable too (they're not mutually exclusive) and the answer is probably yes.
This sits a layer down in the defense-in-depth stack. And the idea is that if there's a recognized code, and consensus on what constitutes a violation, that employers will conform because if they don't they'll risk not just one "activist" employee leaving but most of them, out of a shared sense of communal ethics.
Would it work? No idea. My experience is that software people tend to be pretty squishy on matters of personal ethics.
When Bezos fires every single warehouse employee, what happens if the job they start retraining for also gets automated away before they can even start? And the next one, and the next one. If nobody is making a salary anymore, then it doesn't matter how much lower the prices are on Amazon (due to being produced in automated factories and shipped from automated warehouses) unless Jeff decides to reduce those prices all the way down to 'free'. At that point the assumptions underlying the world's economy would break down in a way that makes a corona shutdown look like a mild hiccup.
No software engineer is going to be able to do anything to help alleviate this. If you want to do something about this, you need to go into politics, not tech.
It's interesting to me that this just presumes developing these autonomous weapons systems in the first place is ethical. I understand there is a difference of opinion on this ethical point, but it immediately frames the discussion pretty far away from the Hippocratic oath's requirement to abstain from causing harm.
- Borenstein
So does abortion and euthanasia, and probably plenty of other practices as well. Both of those are without doubt harm-causing practices, with their related points of controversy primarily revolving around whether the harm that is caused is worthwhile in the context of the alternative being a potentially greater harm.
Putting aside the fact that the Hippocratic oath is not actually a relevant part of modern medicine (modern doctors are accountable to comprehensive, codified sets of ethics), the fact that there is no such thing as a set of common ethics by which people choose to live their lives kinda points out the futility of this idea.
One person could say developing weapons is bad because they cause harm, another could say it’s good because they can be used to reduce harm that would have otherwise been caused. Who’s right? Neither of them. That’s just two people with different opinions. I would personally suggest that establishing moral authorities like can often be harmful, because lacking any objective truths, it’s a topic people should generally be left to make up their own minds about.
Am I right or wrong? Who’s to say? I’m just a person with an opinion, and so is anybody who would want to agree or disagree with me.
I think the main motivator in almost all of those things is money.
The reason for wars is money, they just get justified by "the greater good".
Same for all the involved technologies.
A less controversial example would be something like chemotherapy. In fact, a lot of treatments for terminal and chronic ailments are pretty harmful.
The way its asked looks like an attempt to shift the Overton window until autonomous weapons of all kinds are treated as a mundane inevitability not worth worrying about, with just the niggling details subject to ethical questioning.
But big shifts like that are exactly the sort of thing serious ethical codes should be used to watch out for. Not the niggling details afterwards.
There is no human in the loop (no pun intended for snares). It decides when to strike using physics and the answer is always "yes" if it is triggered.
What makes the new "autonomous" weapons different is that they attempt target differentiation. Mobility becomes useful then when weapons systems can say "no" when presented a target. Since even the Military Industrial Complex, purveyor of unneeded bullshit which wantonly takes lives would find it impossible to sell a drone that goes around shooting missles at all targets after launch.
It just seems wrong for someone that has been asked to "write a web browser" to be at fault for anything.
What about someone asked to code up a voice prompt on something that answers the phone for a telemarketing company. And said company later uses the code to do illegal spam robocalls instead of what they told the developer they were doing with it?
What about if a software developer that writes code to turn off and on a sprinkler system by phone is later convicted of writing the code for a bomb that blows up a building?
What about if a software developer writes code that matches human faces for the purposes of automatically unlocking his door at his home, ends up being open sourcing it, but the developer is arrested for that software being deployed on a drone that murders specific people?
What about if a software developer that is asked to install the above Face Rec system on a drone but is told it's designed to take pictures of people it knows at a birthday party, and is later switched out to trigger a machine gun.
In any case, we already have laws like this! And they're not controversial! If you commit war crimes by killing people with a shovel under the orders of your superiors, both you and your superiors are responsible, but the manufacturer of the shovel obviously isn't responsible for what you did unless they advertised the shovel's skull-bashing capabilities.
Feel free to tell us how your professional ethics as a lawyer enabled you to make change at Microsoft before you were general counsel and president, Brad. Or after you got the top jobs.
Sounds brilliant!
In other words, oaths aren't worth much in the real world, apparently, as much as we might like to think they should be.
It would be troubling if Brad Smith isn't aware of this. He wouldn't be the first once-respected person to go off the deep end: https://en.wikipedia.org/wiki/Francis_Collins.
I have to give Brad the benefit of the doubt and assume something was lost in translation; possibly he is being misquoted or quoted out of context.
But even assuming we could agree on a shared, always-in-sync definition of harm, and even assuming oaths worked all the time, there are plenty of competent people, whatever label we give them, who will design and implement systems without having sworn any such oath, and countries and organizations (including DAOs) that will employ those people. And the first two assumptions are already a bridge too far.
A Hippocratic Oath for only Software Engineers is addressing the symptom, not the cause.
Most of us don't wake up and think, "How can I make user's life worse today?".
We need to fix the incentives if we want to fix the behavior.
We need to give Software Engineers writing spyware the option to say no and still feed their families. We need saying," This is unethical and I can't help," to be a valid second option.
Sadly, just about everyone in ML/AI is contributing to an easily weaponized technology. Every time you make it easier to train a network, every time you make it faster to train a network, every time you improve an image recognition algorithm, every time you improve the latency/jitter of inference... you're contributing to the pile of knowledge which will be leveraged to control populations or enable military action. Most people stay unaware of it and just focus on the benefits. Some of us just grow to accept it.
I would never hire someone that treats software engineering as something easier that medicine or other branches of engineering.
The parent is right. Software engineering is advantaged by a much more simplistic feedback loop. Software's "Hello, World" is validated in milliseconds. Medicine's "Hello, World" could take 30 years of clinical trials to ensure that you haven't killed anyone. It is easy because it is quick, allowing a greater understanding within an equal amount of time.
In fact, the adage of years of experience comes from learning that actually take years to encounter different circumstances and see the results play out in order to fully understand what you're dealing with. This is almost never a problem for software engineers, especially in the learning phase. Software engineers can gain "years of experience" each day by seeing the results of what they are doing in practically real-time.
I also think a lot of people here are overestimating how broad the agreement is on what is ethical and what is not (and thus what would be prevented from happening under such a "Hippocratic Oath").
Examples being: Microsoft/GitHub's work with ICE, Google's China search engine, use of AI in military applications
I personally have a strong opinion on some of those, but I'm not delusional enough to think that there are no competent software engineers that disagree with me.
- Microsoft/GitHub's work with ICE [prevents illegal migration, saving those workers from a life of wage slavery]
- Google's China search engine [gives more information than otherwise would've been available]
- use of AI in military applications [can save lives by no longer bombing civilians where it can be avoided]
A doctor's responsibility is solely to the patient. When the treatment ends, the doctor has done their job. They may be responsible for long-term damage from the treatment, but that's still about something the doctor did themselves. It's like being responsible for your software randomly running amok and deleting random files on a user's computer.
When a software developer has developed some software, that product continues to exist and can be used by anyone who can run it, to do anything the software is capable of.
Turning this around, given that the doctor's "product" is a healthier patient, it would be like making doctors responsible for the evils committed by patients that they have saved.
It's easy to come up with obvious unethical scenarios, but a lot of harm can come from less predictable unethical uses. If you make your software easy to use, the result is that people outside of any ethically-aware, license-controlled bubble can use it. Where does the responsibility end?
The developers of Excel and Access have almost certainly indirectly contributed to evil software.
Also, whose ethics would such a Hippocratic Oath advance? For every privacy-conscious person saying that encryption-everywhere is good, there is a law enforcement officer speaking of reduced abilities to solve crimes.
Imagine a software engineer who has been asked to place a backdoor in some software. Is there any piece of uncontroversial advice which you can give them?
And like several previous commenters have asked, could this be a way to shift responsibility from institutions and their management cadre to individual developers?
I imagine such an engineer just gets a development plan and doesn't get to see the bigger picture, implying the backdoor. It might only get enabled on integration into a larger codebase, and nobody out of the loop will be able to extrapolate its existence from what they get to know for sure.
Hence I completely agree with the argument of shifting responsibility to the developers. Seems like MS is selling more of that eyewash again.
I can't say for sure, but I have a feeling that the hippocratic oath is not what's stopping an unethical doctor from behaving unethically. Instead, it's the risk of lawsuits and loss of their license to practice medicine that keeps them in line.
I just don't think software could match the very real consequences to unethical behaviour that doctors face. We don't have a concept of a "licensed software engineer", and even if we did, there's nothing stopping an unethical engineer from working on their own. Doctors can be blackballed from hospitals, clinical trials, etc. Are we going to blackball software engineers from touching computers?
The above being said, I do believe that software engineers should try to be ethical. However, I do not think there's any mechanism that could be implemented that could force this behaviour, and any attempt to do so is simply a way for its promoters to say "look how good I am", and less so an honest attempt at making the world a better place.
Hence, I am ready to move to another position at any time if the conditions are right (pay, benefits etc). If they want me to take an oath I would probably not take the job today but if they would pay me a ridiculous amount for example, I would most likely take the job and just say the oath or whatever you have to do and just never care about it again.
Just because I am forced to say something doesn't mean I believe it. Maybe it's because I am a natural rebel that hates to be told things, despises "mission statements" or "company values" but I feel like there is a lot of people like me.
Here, it's done.
The purpose of computers is human freedom.
I am going to help make people free through computers.
I will not help the computer priesthood confuse and bully the public.
I will endeavor to explain patiently what computer systems really do.
I will not give misleading answers to get people off my back, like “Because that’s the way computers work” instead of “Because that’s the way I designed it.”
I will stand firm against the forces of evil.
I will speak up against computer systems that are oppressive, insulting, or unkind, and do the best I can to improve or replace them, if I cannot prevent them from being bought or created in the first place.
I will fight injustice, complication, and any company that makes things difficult on purpose.
I will do all I can to further human understanding, especially through the new visualizing tools of interactive computer graphics.
I will do what I can to make systems easy to understand, interactive wherever possible, and fun for the user.
I will try not to make fun of another user’s favorite computer language, even if it is COBOL or BASIC.
They're the one's who's "interesting" ethics are responsible for the things mentioned in the article.
The engineers wouldn't generally even attempt to implement the concerns in the article unless directed by their organisational higher-ups.
Let's consider Boeing. Aviation is complicated and takes a long time to learn. Good software engineers are constantly moving jobs to boost compensation, so Boeing would retain them for 1-2 years at most unless it were willing to radically increase its compensation year over year.
I have spent 1 year in my job at a parking software organization. I have learned very little about how the business works and couldn't tell you much about the software I work on beyond the features built while I have been there. Most of my team has been there just a year or two and they don't know either. Give me all the training you want, but I know very little about how any of this fits into the business model or how it is going to be used and I will probably never know before I leave. And if we use the median tenure of engineers for my organization, I am arguably scheduled to leave.
A friend is at a medium sized tech company. He spent 6 months on one project, spent 6 months on another, and now is interested in moving to a third. He works on "some cloud service, not quite sure which."
Another friend is just fed tickets, completes the tickets, and otherwise isn't even sure what his software is used for and what kind of companies buy it. He has been there a year.
So, sure, you can create an oath, but how many software engineers are going to ever know enough to object to something? It is maybe 50%?
To understand the bigger picture in many cases, engineers would need to stay on their projects to for 2-4 years. They would need to be there from the design phase to the implementation and deployment phase, as well as the day to day use phase. They would need to know a lot more about what the business actually is in many cases. In aviation, they would need to have engineers stay for 6-8 years.
How does that happen when 3 years at a company is considered a long time and plenty stay 1-2 years?
--- Nathaniel S. Borenstein, https://en.wikipedia.org/wiki/Nathaniel_Borenstein
Smells like diesel gate where executives claimed that they did not know about the cheating device. It is entirely possible to force subordinates to do things in secret in a way that cannot be traced back to the original decision maker. If there is money to be made it's going to happen.
This is especially rich coming from a faboulsy wealthy executive of a company engaged in many unethical activities
If he means we need a professional code of ethics, and the organized professional discipline that makes such a code meaningful, sure, maybe.
But it's odd for a software executive to make that case, since the only reason such a code would be necessary in software would be to provide a countervailing force so that developers resist unethical demands from their employers. IOW, if it weren't for the lack of ethics by Smith’s peer group, their subordinates wouldn’t need new constraints.
What we need is ethics in management.
https://en.m.wikipedia.org/wiki/Professional_Engineers_Ontar... https://www.peo.on.ca/licence-applications/become-profession...
I'm quite certain that many HN readers knowingly contribute to companies that do plenty of evil. Heck, some of the HN crowd have created startups that specifically do gray things just for money.
Practically speaking, there's no way to know that your efforts will not somehow result in doing harm to others. Rather than taking some oath which frankly has little meaning without full knowledge of the outcomes, I think it would be more beneficial for everyone to make a concerted effort to get out in the world - see places that are in severe poverty or political turmoil. Meet other people who are very different from our bubble. Then we start to have an idea how even some of our good intentions can result in worse situations for others (such as giving lots of clothes away to have shipped to Africa).
They're just going to treat it like munitions, because that's all the lawmakers know.
There are professional ethics, but they are much more tightly bound - more comparable to an accountant's ethics than a doctor. For me that will be along the lines of
- I will not use trickery in a demo to decieve others about the progress of a project
- I will report all bugs and mistakes I find and make, regardless of any personal cost this may bring on me.
- I will be honest in my dealings with non technical people, and do my upmost to accurately represent the work I have done and the work I plan to do.
- I will make decisions and advocate for changes for the benefit of the codebase and organisation, not for the benefit of my own CV
... And so on - you can pretty quickly get into controversial territory I'm sure.
This is a good starting point. All those who've used dodgy marketing to convince investors that their machine learning is AI and so we're 'first generation of humans to endow computers with the ability to make decisions' have lent their words to a culture war with no basis in fact.
How can we enforce this? Strong individual ethics without ability to do anything about it will simply make engineers feel helpless. And that’s just going to hurt happiness at the workplace.
The oath states:
"I am an Engineer. In my profession, I take deep pride. To it, I owe solemn obligations.
As an engineer, I pledge to practice integrity and fair dealing, tolerance and respect, and to uphold devotion to the standards and dignity of my profession. I will always be conscious that my skill carries with it the obligation to serve humanity by making the best use of the Earth's precious wealth.
As an engineer, I shall participate in none but honest enterprises. When needed, my skill and knowledge shall be given, without reservation, for the public good. In the performance of duty, and in fidelity to my profession, I shall give my utmost."
The Order of the Engineer started in Canada in 1925 as the Iron Ring. It was imported into the US in 1970, with many changes.
Worn on the pinky finger of the working hand (depending on dis/ability), the ring is meant to drag on drawings and leave subtle marks as it ages and rusts. It's facets are meant to be noticeable, to remind engineers of their duties and obligations. Though I cannot find the source, I have always heard that the original Canadian Iron Rings were made out of steel from a collapsed bridge that was lethally designed.
The US version is significantly different, as engineers are licensed very differently in Canada and the US.
The US based Order of the Engineer is fairly accommodating to change and updates. Bioengineers are welcome to take the oath, for instance.
Perhaps the US based rings for software engineers should be made of the Uber car that killed that poor woman in Arizona, Therac-25, or Theranos machines.
* Medicine is thousands of years old. Software is decades old. There's still a lot we don't know.
* Medicine involves a lot of repeat situations, do you can set precedent and learn over time. Software isn't always new, but there are more novel applications than in medicine.
* The patient is, generally speaking, at the center of a doctor's ethical universe. It's not clear who the "patient" is for engineers. Users are often malicious. You can just say engineers must consider "society as a whole" but that's kind of a cop out, and not useful for trickier situations.
So the oath would be to do no harm to the company.
Engineers mostly do what they are told, it's too easy to let them carry the moral burden while higher ups can ask whatever they want out of them. It's too easy to shift the blame on developers.
Considering the life of Archimedes of Syracuse who illustrated the ambiguous potential of technology since the Antiquity, Considering the growing responsibility of engineers and scientists towards societies and nature, Considering the importance of the ethical problems stemming from technology and its applications, Today, I commit to the following statements and shall endeavor to reach towards the ideal that they represent:
I shall practice for the good of humankind, respecting human rights1 and the environment.
I shall recognize the responsibility for my actions, after informing myself to the best of my abilities, and shall in no case discharge my responsibilities on another person.
I shall endeavor to perfect my professional abilities When choosing and implementing projects, I shall remain wary of their context and their consequences, notably in their technical, economic, social and ecological aspects. I shall give particular attention to projects with military applications.
I shall contribute, to the extent of my abilities, to promote equitable relationships between people and to support the development of economically weaker countries.
I shall transmit, with rigor and honesty, to discerningly chosen interlocutors, any important information, if it constitutes a gain for society or if its retention constitutes a danger for others. In the latter case, I shall ensure that the communication yields concrete action.
I shall not let myself be governed by the defense of my own interests or those of my corporation.
I shall endeavor, to the best of my abilities, to lead my company to take into account the preoccupations of the present oath.
I shall practice my profession in complete intellectual honesty, with conscience and dignity.
I solemnly take this oath, freely and on my honor."
1. According to the Universal Declaration of Human Rights of the United Nations (10 December 1948)
I attended an event about AI ethics a couple of years ago, many of the panelists were getting very excited about the upcoming AI eithics guidlines from the IEEE and how it would set a gold standard for state level AI development. They were completely unaware that the IEEE already has a general code of ethics[0], which many governments implicitly require engineers in certain roles to ignore.
[0] https://www.ieee.org/about/corporate/governance/p7-8.html
You can't become surgeon by Googling and slicing things
While you can become /Software Engineer/ entirely by yourself (self taught)
"The inflating red bag stopped inflating. What do I do?"
> Tick here to certify that you have read and understood the Techno Oath
"Yeah yeah, whatever"
It isn't taking a five why's approach to the problem of ethical failures in the industry. It won't fix root cause problems.
We don't need a Hippocratic oath for Software Engineers. We need laws and regulations around some things. And then the government needs to penalize C-level execs, the board, and major shareholders for violating them.
The problem is that laws apply to individual without significant financial resources, sporadically to the truly wealthy, almost never to corporations, and never to the shareholders.
Trying to fix a broken system with a "Hippocratic oath" for software developers is like trying to fix school shootings by making sure the doctors will save the school shooters as much as the victims.
Microsoft could still be profitable and fix way more bugs in the software than they do - they choose the higher profit margin over "the good of humanity" all day every day, and it is the executives and major shareholders that make that own that decision.
IEEE and almost all of the serious engineering organizations have a code of ethics.
Accountability of the management is what is missing. If a hospital management instructs their doctors to skip costly steps in a procedure, they are going to jail. If any engineering company screws up and lives are hurt, lost or damaged somehow (e.g. privacy), they issue a public apology; rinse and repeat.
Which is ironic, as "a doctor might screw up and kill a few patients, but if you as an engineer screw up you can kill thousands." Because you are not there holding their hands when people get hurt, get their identity stolen, their money or lose their lives due to a bug or critical error, does not make you less responsible.
The problem with our field is that i) it has immaterial direct results, ii) big failures have big money behind them. The first makes it hard to impossible for the public to understand the implications and does not excite terror if you are not directly involved. How many people run in fear hearing about "x credit company got hacked"? The latter reason implies that there is an incentive for no punishment -- similarly to banking institutions in 2007.
I think we are unfortunately a lot of the times swaying between being a statistic (https://quoteinvestigator.com/2010/05/21/death-statistic/) or having insignificant or incomprehensible impact for us to gain legal power to push back. Recall that deep learning, distributed ledgers, etc are magic or demonized by the public.
P.S. One could argue that not all software developers are trained engineers etc, but I am not buying that. As a lot have noted even if a person is not academically trained they can take an oath, but nothing is going to change in the responsibility realm.
Software engineering is one of the few where word “engineering” is not really engineering. Software engineers neither know the law, not are trained for that, work globally and have no idea what laws apply. This has its own benefits and drawbacks (such as, less protection - anyone can take your job, no matter the degree, but you can also work anywhere).
For “software engineers” to be real engineers in order to take any oath, the education system and degrees for software need to change and match those of other local protected engineering degrees, with all benefits and drawbacks of a state recognized profession that can by carried only locally. None of current developers matches any of that.
It could be a possible future, but it may also kill talent as know it in software. The software market dynamics of today exist because everyone that can write code can claim they are “software engineers” and they only need to write code to prove that. You cannot claim you are a doctor, just because you think you have skills to heal people and prove that by healing out a few.
But it's not the decision makers who should be targeted. The workers told selling the product aren't a problem. The workers marketing the product aren't the problem. No. The workers told to make the product.
These workers have such easy, stable, and wealthy lives that they cannot be pressured in any way to make something harmful. They are so intelligent, so smart, so perceptive that the truth can't ever be hidden from them. In a world governed by relative and fuzzy morals and ethics, with few absolutes and a plethora of grays, these guardians of the righteous should take an oath to be better; to take responsibility; to uphold these virtues; to lead by example; to let no more evil pass under their gold hands as they type in the language of the creators (perl).
All you have to do to become one of these divine creatures? Take an online programming course and tick a box stating you read and understood the ter... uh... oath. Yeah. Oath. Because that's what's gonna solve the problem.
As a male, white, urban, and affluent, perhaps he could demand to get replaced by a POC like some corporate board members have done, instead of endlessly yapping and proudly showing off his self-hating racism.
It sounds more like a mandatory code of conduct for all software projects.
Why? If four weeks is enough for somebody to learn enough that they can build Joe the Plumber a slightly nicer-looking website that he's happy with, what's wrong with that? A lot of excellent software engineers have zero days of formal software education. Even Donald Knuth doesn't have a software engineering degree; he has physics and maths degrees. Should Donald Knuth be forbidden from developing software because he doesn't have a piece of paper saying "software engineering degree"?
How about we start with something we need more immediately - updated antitrust laws that are enforced against giant companies like Microsoft? Brad Smith is talking big about requiring an oath for engineers but neglects his own role. Before he was President, he was General Counsel for MS. He is still today the chief legal officer and chief compliance officer alongside his president title (https://en.m.wikipedia.org/wiki/Brad_Smith_(American_lawyer)). Let see him and Microsoft come clean on antitrust, privacy, theft of others’ innovation (like Slack) while holding a giant patent war chest, etc.
People very much disagree on what is ethical, and the people who know they are being unethical rarely let simple oaths stop them. Am oath like this simply becomes a nuisance to the ethical free thinkers, while becoming one more useless social more for the people who don't care about ethics.
1. Care of Earth
2. Care of People
3. Fair Share
They can be applied to technology as part of the whole ecology, not just human-centric. It puts life and death in the proper context of the whole, rather than the way modernity had twisted it up.
Neither the planet nor its biological inhabitants "care" about anything. They just are (as in they exist).
The main driver for protecting the ecosystem is the fact that we - as a species - can't survive (for long) without it.
Granted, there are some who strongly believe that becoming a race of cave dwelling mole people or inhabitants of glorified snow globes in an inhospitable barren wasteland is a desirable prospect somehow. But overall humans prefer blue skies, green meadows, birdsong, trees, and diverse natural landscapes.
Until the people pointing the software engineering departments are held accountable for their immoral behavior and goals, I don't think the engineers beneath them can really be assessed fairly.
Funny how same topic comes up every time some new issue pops up but we've had the answer nailed in the 90s.
If you want software you can trust no amount of oaths, certificates and overseer bodies will ever be enough. It's only possible to trust fully transparent software.
This isn't about trust. Is using Libre Office Calc for operating a death camp more ethical than using MS Excel?
Is the creation and use of an armed autonomous drone justifiable and ethical as long as its firmware is open source?
Are misinformation bots and fake news generators OK, as long their source code is freely available for use and further modification?
Software, all software, needs an ethical standard and it currently has nothing remotely close in common practice. This will be a hard sell though, because most developers have no idea what ethics are, why they would be needed, or how they apply to practice. Frequently when the idea of ethics are raised the response is hostility and often framed in terms that are purely imaginative.
When people in other industries ask me what’s required to be a software developer I always tell them it’s just a matter of charming an interviewer. There is no education requirement, no licensing/certification, no standard skill definition, no internship or agency, and certainly no ethical standard. It’s always amusing to watch their response.
Does the same set if ethics/regulations even make sense for one person developing a guided missile versus another making a video game, that happens to shoot missiles at virtual targets?
>The Iron Ring is a ring worn by many Canadian-trained engineers, as a symbol and reminder of the obligations and ethics associated with their profession. The ring is presented to engineering graduates in a private ceremony known as the Ritual of the Calling of an Engineer.[1][2] The concept of the ritual and its Iron Rings originated from H. E. T. Haultain in 1922, with assistance from Rudyard Kipling, who crafted the ritual at Haultain's request.[1][3]
Engineers are hired to make weapons systems that kill civilians. Engineers make software that enable ubiquitous surveillance by whomever wants to use it for whatever purpose. No a Hippocratic oath will have a hard time influencing individuals and their organizations that’s what laws and the enforcement thereof are aimed at. I have little hope for oaths or laws however.
Most Software Engineering Degrees require an ethics course. Yet - we still have the issues we have today.
Asking for good intentions isn't going to change outcomes, because Software Engineers already have good intentions. Nothing will change.
How about Software Engineering Licensure? You have been found to contribute glaring security issues in widespread code? Lose your license and employment. Use dark patterns to defraud people? Lose your license and employment. Leave "debugging" endpoints open or collect unminimized telemetry? Lose your license and employment.
This seems stronger, and much more likely that outcomes will change.
But good luck getting the industry to move.
Also it's not that the underbelly asks would ask their hacker candidates for a license, or that the military/cops would care about a licensing body.
And markets/businesses with a high risk of negative effects for society (and software already belongs in this category) should be strickly and proactevly regulated.
Maybe the oath is yet another hollow ritual and what Software Engineers need is what everyone else needs, a sense of morals and a spine.
What would be good if we took a leaf out of the book of the real engineers and started caring out what we put out there as though lives depend on it. But that starts with getting rid of all of those disclaimers. No other industry gets away with accepting such little liability as the software profession.
I am honestly much more worried about terrible directives coming from c-suite persons, that result in ethically questionable apps and software.
In addition to ethics training and commitments from the persons doing the actual engineering work.
Still, I'll always try to fight for the users in any code I write and, specially, in any code I control.
It would only work for software engineers if we moved to the same model: Management run the facility but cannot tell software developers what to build.
I don't think that's a viable model for managing software development.
In all seriousness, ultimate decision making, hiring, firing, promoting, rewarding, punishing and culture building are all on management and leadership. Microsoft President happen to be on top of it. If they think companies use engineers in an unethical way, what about building management culture that ensures this wont happen.
A tale as old as time itself.
In fact, I am starting to think certifications like PCI DSS are similar to oath: "I will not leak customer data, I will not leak card information". If the oath is broken, the PCI retaliation is swift: the whole company is totally cut off from processing payments. The company compliance is ensured on threat of bankruptcy.
So you push back: pushing this through means the company will fail PCI DSS audit once that comes up, and then we're all out of jobs. No salary for anyone. And the boss thinks again.
Software, as an object, is harmless. It can be used for good or evil, just like anything else. So if anybody needs an oath is the people using the software, not the people building it.
Engineers want time & resources to do a good job, to make sure things are well considered. It is capital that drives us to push for expediency & convenience.
Mechanical Engineers have a similar one (ASM).
This is not novel.
This has lots of implications but the big one is really very few railway deaths.
A thief knows stealing is wrong. A murderer knows killing is wrong. You don't stop thieves and murderers by making them swear hippocratic oaths.
No.
No again.
A thousand times, no.
The reason that doctors take an oath is because their practice directly affects people. A patient puts their leg, their heart, their eyes right into the doctors hands. Any mistake, any "experiments"... this directly harms the patient in an objective way.
A "do no harm" oath doesn't work with software. A software engineer writes a piece of code. This can be used for good, or evil, but by whose standard? Once the code is out there, the software developer cannot be held liable for its abuse, except perhaps in the limited circumstance that the software was designed FOR abuse. (ie: a zero day released without responsible disclosure).
I would never take an oath that I wouldn't release "harmful software". First, because "harmful" today has become incredibly subjective. (Someone might say I'm being harmful in writing this!) Second, because it's difficult to tell how something will be used, and I don't have the time or energy to follow up with everyone using it to make sure they're using it to my liking.
Last year, someone got a lot of attention for taking down their insignificant NodeJS is_even implementation when they discovered that ICE was using it. Can you imagine what would happen if Redis disappeared, because the authors discovered it was being used by some disagreeable activists?
I'll tell you, very simply. People would continue to use it, having archived the source. The activists, ICE, you, me, everyone. Any such oath is useless posturing, and subject to the whims and pressures of what's in vogue today. Gives the appearance of doing good, without having to do any good.
> This can be used for good, or evil, but by whose standard?
Your own. The oath is taken by humans, and humans differ in opinion - no one can impose an absolute standard on you, and you follow your oath as you understand it. There's no board to deprive you of your license in IT.
> Any such oath is useless posturing, and subject to the whims and pressures of what's in vogue today.
It's only as good as the will to uphold it. I think that's what's in short supply among the stereotypical "bro" coders.
The problem is the lack of value of the word "engineering" in several countries, where everyone fells like calling themselves engineer after a six weeks bootcamp without suffering any issue with it.
Or the fact that "enginnering" has be co-opted and made de-facto illegal for all but the small minority who agree to pay the local mafia^Wengineering association. This is especially a problem in North America. In Europe, the problem is different as you can only declare yourself an "engineer" if you've been in the right university.
Anyway, I could be a P.Eng, but paying the yearly racket money... I'll pass.
"We need a Hippocratic oath for software engineers"
What it would be useful for the Microsoft President to say:
"We need a Hippocratic oath for software engineers, marketing, sales, accounting, customer support, management, senior executive staff, board of directors, ..."
I'm not sure how well this meshes with the "you don't need a computer science degree to be a programmer" trend.
And hey, now that remote work is the new hotness, we’re more replaceable than ever!
Before we can be expected to take an oath of any kind, software engineers need to have agency.
The surveillance apparatus wouldn't exist without these types of engineers.
Managers handle the context of application of a software.
A senior executive who wants to push this onto their employees can be safely optimized out of the discussion.
Anything else will just be buck passing!
I could get behind that, dont think you will though.
This may seem ridiculous considering the responsibilities of medical professionals but it works as a baseline independent of the legal and moral standards of the particular society they live in.
So to answer your question:
Yes, the software engineer could be held accountable - yet, that would arguably be up to him- or herself in many cases [1]. There were physicians in history who have done horrible things [2] and - if not by the legal system - they were at least judged by their peers for their actions.
From my perspective, the most valuable aspect of having an oath (and that I regard for professions in general) seems to be the opportunity it offers for reflection, identity and future aspirations.
[1] https://en.wikipedia.org/wiki/Hippocratic_Oath#Violation [2] https://en.wikipedia.org/wiki/Josef_Mengele
the medical profession hierarchy is completely different to accommodate for medics own agency.
responsibility without freedom (and without just compensation, dare I say) is just bullshit.
Also, it makes sure we have a ethical framework to work with. Right now, it's every engineer for themselves. You won't do something that sits in a legal gray area? Companies will find someone hungry enough to do it. If we're all staking our professional reputation on it, there will be fewer competent takers.
Personally, I think it will also be a step towards better legitimizing the profession and being more inclusive. Many folks have started equating Software Engineer with hyper-rationalist orthodox libertarianism and... the stereotype isn't far off from what's typically upvoted in these sorts of communities.
Anyway, them's my thoughts on this. I definitely support this sort of notion.
Certifications is meaningless corporate propaganda and we should eliminate it completely unless it's free and public.
I remember the professor starting the class as,
>"If a Structural/Civil Engineer builds a bridge and it goes down, he/she will go to jail; lucky for you guys there are no ethics for computer science".
Last time when I said this, I was told there's no way a Structural/Civil Engineer goes to jail if a bridge goes down. May be its more common in India, could be very well be the norm because every time a bridge goes down a related Engineer gets arrested the very same day or soon under 'Causing death by negligence'; perhaps a practice from colonial era still being practiced to pacify public.
Read:
Another BMC engineer arrested in Mumbai bridge collapse(2019)[1]
Bhubaneswar flyover collapse: Engineer, director of construction firm arrested(2017)[2]
4 Engineers Arrested In Kolkata Flyover Collapse Case(2016)[3]
IIT Roorkee: Two professors arrested in bridge collapse case(2015)[4]
SMC engineer held in bridge collapse case suspended(2014)[5]
Gammon, Hyundai officials arrested, probe ordered(2009)[6]
I'm sure you can pull up such cases going back at least 200 years.
[1]https://www.deccanchronicle.com/nation/current-affairs/02041...
[2]https://www.hindustantimes.com/india-news/bhubaneswar-flyove...
[3]https://www.ndtv.com/kolkata-news/4-engineers-arrested-in-ko...
[4]https://www.indiatoday.in/india/story/iit-roorkee-two-profes...
[5]http://timesofindia.indiatimes.com/articleshow/38044181.cms
[6]https://www.ndtv.com/india-news/kota-bridge-collapse-30-dead...
The people in charge who run the business need a Hippocratic oath.
This is the most hypocritical, low-minded dirty blow, pass-the-buck mentality I have seen in a long time. That's even counting most of what Trump has said. You should be ashamed of yourself, Brad.
How dare you make that remark when your WHOLE PLATFORM is built on shady business practices and monopoly leveraging? How dare you try to shunt the blame on software engineers when YOU YOURSELF profit from lack of ethics and participate in systematic blacklisting of engineers? You earn seven figures a year AT LEAST and are trying to blame people just doing their jobs, and telling them they should look at the big picture and they should "have principles" to basically refuse to work or quit? While at the same time actively preventing these kinds of people from finding ethical work elsewhere?
Put your money where your mouth is or stop spewing Trumpian bullshit, Brad. Anybody who buys this needs to have their head examined. With a cactus.
Brad Smith in 2020: We Need a Hippocratic Oath for Software Engineers.