And then, if their code results in a death, they are liable and can have their license completely revoked, and they would be unable to work on self driving cars again.
- If bad code makes it into production, that is a systemic failure not an individual one (Why didn't the bug get caught in code review, QA, etc.)
- No one is going to want to work on a project where a single failure can taint their career.
- What if I use a 3rd party lib and that is where the bug is. Who is at fault then? What if the code isn't buggy, but I'm using it in an unexpected way because of a miscommunication? If I am only allowed use code that I (or someone certified has written) development is going to move at a snails pace.
- What if I consult with an engineer who doesn't have a certification on a design decision and the failure is there, who is at fault?
- What if the best engineer on the project makes a mistake and ends up banned? Does he/she leave the project and take all their tribal knowledge with them, or are they still allowed to consult? If they can consult, what stops them from developing by proxy by telling other engineers what to write?
Not to be a dick, but this is an awful idea that would basically kill the self driving car.
Yes, there are reviews, QA, and all of that. So, yes, there is no single person responsible (exceptions apply).
But there is no excuse for using 3rd party libs. Just don't use it. If you not know: do not use it.
That is the reason, why certifications are for. The same rules apply for medical and other areas.
Wait, what? That goes against one of the core benefits of open source software--that having many eyes on a problem decreases the risk of bugs. I'm willing to bet that if Uber had to implement their own machine learning/vision libraries from the ground up, there would be significantly more issues.
Certification etc is about process. Open source code can be used in a safety critical product, but it must audited and confirmed against the system requirements.
The problem with that is, nobody audits code, if it is working just for fun. And even if it is buggy, then most people look for bugs in their own software and then they work around, so that the original piece is not modified.
We have this seen in many open source projects. Remember all the obvious, mostly security related, bugs that weren't uncovered for years. They weren't uncovered because everybody thought: "huh, that is hard. I assume that other more experienced than I will have reviewed it, so I will trust it."
The thing with certification is, that it is required that it is really reviewed. That there is a guarantee that it is reviewed. That there were people with a different mind set, with different background have reviewed it and as such have brought in their own view.
Certification does not guarantee that something bug free. It guarantees only, it is reviewed. Open source has no guarantee that it is reviewed. There is only hope, that someone has reviewed it.
Pretty much everything in development relies on the work of other people. I used 3rd party lib just as an example, but what if it's in the framework or even the language that an engineer uses, who would be at fault then? You can't expect every developer to have gone through the entire source code for whatever language they are writing in.
Sciences build on each other and and after a certain period of time you have to take things for granted in order to keep moving forward.
> The same rules apply for medical and other areas.
No, they don't. Doctors kill patients all the time and they aren't banned from medicine for it. There is an investigation, they make sure it wasn't intentional and there wasn't any gross negligence and that this isn't a repeating pattern, if none of those are the case they see what they can learn from it and move forward in hopes that what they learn can help other doctors.
I agree dith parent: never use copy paste from internet in safety critical SW, anyway it most probably isn't designed for your use cade. Personally I always have been disapointed by copy pasting stuff, it was always buggy somehow. In the end I always reimplemented it from scratch by reading the theory
From what we seen so far the Uber car failed to detect an obstacle, we also had Tesla crash where the car did not seen a truck so it is obvious that there are some major issues that are not tested for. They need to have better tests and maybe get better security drivers inside the cars so they don't text on the phone on the job.
Does that demonstrate that the cars are safe? Even a little bit?
To me it just demonstrates that a grunt is on the chopping block for what amounts to systemic failure.
In safety-critical fields, setting a much higher quality bar than the regular 'it seems to works, the tests pass' seems perfectly rational to me. We can now write provably-correct C compilers (CompCert) and OS kernels (SeL4). There's no excuse for not putting similar levels of effort[0] into something as safety-critical as self-driving cars.
[0]: Note that I'm not advocating for "provably-correct self-driving car software" (that may not be the right approach, as a formal spec is likely unrealisable), but find the argument that "it's ok to write buggy spreadsheets, so it's ok to write buggy self-driving cars" to be morally unacceptable.
"The engineers employed by Jack D. Gillum and Associates who had "approved" the final drawings were found culpable of gross negligence, misconduct and unprofessional conduct in the practice of engineering by the Missouri Board of Architects, Professional Engineers, and Land Surveyors. Even though they were acquitted of all crimes that they were initially charged with, they all lost their respective engineering licenses in the states of Missouri, Kansas and Texas and their membership with ASCE.[22] Although the company of Jack D. Gillum and Associates was discharged of criminal negligence, it lost its license to be an engineering firm." - https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...
https://insight.ieeeusa.org/articles/professional-licensure-...
That's the whole process. Repeat that over a long enough period of time and you tend to select for a more competent, safety-conscious group of engineers.
Over the long term, its survival of the fittest. Reckless engineers have safety incidents and get barred from both working and from supervising new engineers. So, it tends to be the safer engineers who get to pass on their work culture to the next generation of engineers.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
Likewise if you knowingly observe anyone else in your company breaching safety/regulatory guidelines then as a professionally certified engineer you have a legal responsibility around ethical disclosure.
See: http://www.professionalengineers.org.au/rpeng/ethical-commit...
I do not know how things work in the US but in Australia these rights are protected by law. The company legally can not fire an engineer in this situation.
Professional engineers (note the lowercase 'e') are usually not protected in the U.S. -- they're regular employees whose profession happens to be engineering.
If you want error-free software you need a blameless culture based around process, not individual ownership of code. It should not even be possible for an error to be one individual's mistake, because by the time it hits the road it should have gone through endless code review and testing cycles.
Just like it relocated its testing to get a more "business friendly regulatory environment".
For a mostly topical example of such rules: https://www.dmv.ca.gov/portal/dmv/detail/vr/checklists/outof...