“I Want to Know What Code Is Running Inside My Body”
backchannel.com
backchannel.com
Open sourcing this code would do a lot to mitigate these issues.
Obscurity is good practice as one layer of a layered defence system.
See "Defence in Depth" https://en.wikipedia.org/wiki/Defense_in_depth_(computing)
"Defense in depth is originally a military strategy that seeks to delay rather than prevent the advance of an attacker by yielding space to buy time".
We have to acknowledge that no system is perfect, there will always be holes, therefore a good approach is to layer up the imperfect systems which delays the attacker.
Obscurity is one of those layers, a system will always be more secure if you have to find it first.
its one part of a whole system, and in that, if it does delay or deter (even slightly) then it has worked.
The trust question is the problem with obscurity. Do you trust the people making it obscure?
In this particular case, where safety-critical standards are relatively well known (within the industry) and not themselves obscured, they deserve to be trusted.
Do we trust them or not?
The problem with VW was not a 'bug'. it was a malicious design.
After all, if you're a system maker, why would you hire hardasses who have rejected your products in the past? And if you're a certification house, why would you $$$ on many hours from experienced engineers when you could use fewer hours and junior employees giving you happier customers and higher profit margins at the same time?
You can hire "independent" people to tell you what you want in a lot of industries. You want an "independent salary survey" to tell you that $50,000 is the market rate for an experienced programmer, but that your CEO needs a $5 million raise? Or an "independent credit rating agency" to tell you your subprime mortgage backed security is triple-A rated? The free market will happily provide such "independent" reports at the right price.
But the government(s) also employs similar agencies to perform the enforcement of the certifications.
Its not a perfect system, but its a lot better than the developer-on-the-street realises.
It's like when you leave a key for someone under a door mat. You don't often consider that the door might be easily kicked in by an intruder.
Thats the problem right there.... not obscurity.
This applies to security and safety. There is no reason not to implement obscurity also.
One thing I can guarantee is that your implementation of one security-type will not be unbreakable.... and neither will mine.
Better to add obscurity than not.
For example, hiding the algorithm from whitehats may prevent/discourage them from hunting/reporting bugs.
Yea, this is a serious concern, I guess it depends on use cases. For sure "security by obscurity considered harmful" could be true, thats the thing people overgeneralize and fight over it when this should be weighted depending on the circumstances.
I think the big problem with obscurity is that its impact is asymmetric in the wrong direction: it inconveniences white hats a lot more than black hats.
Even if mathematically unbreakable, the implementation won't be.
This is the whole premise of defence-in-depth... delay rather than prevent.
An acceptably low break in rare using mathematically valid encryption.... Yes, fine... Given a perfect implementation.
You haven't got one of those.
I don't think it's too much to ask before adopting a given security policy that it provide some evidence that it increases security. Or should I also be gathering a collection of rocks that keep hackers away?
1# security by obscurity gives a false sense of security. Under no circumstance should obscurity be used as a deciding factor behind a management decision.
2# security by obscurity cost money and time, and should only be used when all real form of security measures has been implemented. Even the military are currently not always implementing multi-token authentication, ipsec and selinux. Instead of trusting that the medical deceive is safe behind two layers, a static password and a secret port, add a certificate and implement challenge and response.
3# the priority to implement security by obscurity should be far lower than all the real security technologies. When reading security reports by pen testers, its important to understand the difference between a verified code injection vulnerability and a system information disclosure. Fixing a remotely code injection bug is much more important than hiding the fact that a system is running a up-to-date stable version of Debian, yet many security guides and reports for pen-testing tools rarely priorities.
4# security by obscurity often has real cost in support, brittleness of the system, and debugging. There is still in 2016 firewalls that will permanent block any ip address that has sent a icmp package to them. The amount of work employees are spending to unblock customers that accidentally end up in the block list could be spent on making sure that the system is just that more perfect against the serious attackers who can afford to spend $50 to access a botnet for a few hours.
Its common sense that I can't pick a lock if I cant find the lock.
This says nothing about the quality of the lock or what is behind the lock.
Spending time on security by obscurity should be a job for the small minority of people who already done everything else, and then only if there is a cost-benefit analyze that show cost of the obscurity to be less than the calculated gains.
These types of products are entirely designed up front and analysed before any code is written so the implementation order is irrelevant.
Obscurity don't scale. Things that are commonly used should not use obscurity.
Somebody who mass produces computing equipment or software that many use can't use obscurity because it's economically efficient for attackers to look past obscurity. It's also unproductive to advice others to use some obscuring methods, because as soon as something becomes even slightly common, it can be detected and security of obscurity vanishes.
Obscurity must be obscure. Great minds think alike and it's very easy to build obscurity that is similar to what everyone else thinks is nice trick.
Genuine obscurity can provides additional security layer (in probabilistic expected value sense) against automatic or routine attacks. If obscurity requires even small time to figure it out, it's likely that attacker moves to next target. But it's hard to know how well the obscurity is working.
Practically speaking, obscurity is a "platform" that lets you bypass everything, whereas knowing a password is more limited since it grants access to a single user. But practically speaking, obscurity and root passwords are similar..
I wonder if there are formal definitions here that makes the separation clear.
Locks are rated in how many seconds/minutes they can withstand from a dedicated attacker. Perhaps that would be a way to determine a similar safety rating for passwords/crypto based systems.
In passwords: how many passwords can you try per second before the server refuses? Then Password space/# per second = total seconds for guaranteed entry.
Crypto: how many keys do you have to calculate before you succeed in finding the correct key? Keys/second * how many machines / keyspace.
But in the end, I don't believe we really have a formalized difference between types of obscurities, aside the "go to /root and get root" obvious badness. It would be a rather nice way to provide security in "seconds to millenia depending on techniques used".
Most individuals defending algorithmic security through obscurity believe that hiding the algorithm improves security. That may be true in an extremely technical sense (the attacker must recover the algorithm first), but it is very misleading and unprofessional commentary. Algorithmic security through obscurity is at best calculated in difficulty-to-reverse-engineer (or difficulty-to-steal), which doesn't provide per-use(r) specificity (per-user password) nor scale in complexity (a 256-bit key is generally 2^128 times stronger than a 128-bit key, but doubling the algorithm length increases reversing time by slightly less than a factor of 2).
Algorithmic security through obscurity provides negligible security, but what's the harm? Why should we care? Attempting to hide the algorithm provides a false sense of security, limits review to "approved" parties, and induces legal/social efforts to "protect" the secret. The limited review is particularly noteworthy since it promotes bugs in both the algorithm and the implementation. The end result is a facade of security, some very unhappy whitehats, some very happy blackhats, and more users betrayed through poor security practices.
[1] http://csrc.nist.gov/publications/nistpubs/800-57/sp800-57_p... [2] http://tjscott.net/crypto/64bitcrack.htm#INTELG
The major difference here is this: with security through obscurity, someone can reverse engineer one product and then they've broken all products. This is why someone upstream said "security through obscurity doesn't scale". Security through obscurity is often okay if you're protecting one thing, but if you're using it to protect a system (like a pacemaker) that is going to be used by a lot of people, the more people who use it, the more valuable a reverse engineering hack becomes. Security through obscurity can't be individualized to provide security to each individual--if one system is broken all systems are broken.
Compare this with key-based security--if each instance of the system has an individual, randomized key with a large enough keyspace, breaking a key will only get you into a single instance of the system. It scales because the reward for breaking the security doesn't grow as the number of system instances grows.
Note that the problem with security through obscurity is basically the same problem with master keys, i.e. those used for backdoors or DRM. If someone can obtain the master key for the system, they can break all the instances of the system.
It is much harder to formalise how hard it is for an attacker to find out what algorithm you use, so it is risky relying too much on him not being able to do so.
It's qualitative versus quantitative. Security is a quality, which means it can become absolute: the complete absence of security holes. Obscurity, on the other hand, is quantitative, because you can always add more obscurity. There is no "absolute" obscurity.
"Buying time" doesn't make sense in this context. It's pacemaker software. Who's buying time by making proprietary pacemaker software and for what reason?
Buying time is only useful if you can see when an attacker begins trying to subvert your system. If he can sit at home working on it for a year, "buying time" to improve security makes no sense, as you can't spend the time to improve security,p when you don't know you're about to be attacked.
No layer of security is ever perfectly implemented, mathematically perfect tho the algorithms may be.... this is the key point that defence-in-depth acknowledges... and hence is the key point that obscurity addresses.
Buying time certainly does gain you a lot in this context. If someone was attempting to bypass security to get into a pacemaker in side me.... I would damn-sure prefer the apparent "lock" to be hidden rather than in plain sight! (given everything else is equal).
I would prefer the lock be visible to me over either situation. Otherwise, how would I know how easy is it to bypass?
What most developers don't realise is the level of engineering strictness that goes into anything safety-related. The rules and regulations related to anything that affects the human body is in a different league than what most developers are familiar with.
What is a problem here, is that the design (not the code) apparently did not take into account any messaging security, relying on obscurity as its only defence.
If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors, or simple things like that. Its the design of the system as a whole at fault, and that is already open.
Medical devices such as these are not black boxes to the people that certify them, everything is open to them, source included. Having worked in that sort of area, I trust the systems that are in place.
I'd argue that over time safety critical systems have worse code than normal software because refactoring is almost impossibly expensive due to all the paperwork that each software change ensues.
In fact, I'd say the restrict them. Adding a tonne of paperwork is a great way to prevent people from "noticing" problems.
The security/safety processes (as applicable) are examined at every level from system-design all the way down to implementation.
Its not perfect, but its best practice we have for secure/safe product development.
"industrial control" covers both certified and non-certified code, you will have to be more specific than that for the purposes of this conversation.
Quite often, cases such as the pacemaker are flaws in system design, not code. (e.g. no secure messaging, probably due to the lack of awareness when it was designed).
That is not to say that safety-critical code is perfect... just that it has a lot more rigour and inspection involved than run-of-the-mill website code.
True, the expense and overhead does indeed affect the level of change that is acceptable to a business, but given that change is allowed, the quality of that change is what we're discussing here and all I'm saying is that there is a lot more rigour involved than most developers here are aware of.
I had assumed that as well until all of the horror stories around Toyota's firmware came to light.
https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...
Yes, the toyota case is a well publicised case. Consider though, the number of safety critical systems that are out there performing perfectly everyday. Of course, that is not proof of much, but the fact that you can name the Toyota case (and probably the Therac 25 case) means that the process generally works.
Smart electricity meters are safety critical and I know students writing on it in their summer jobs. Airplanes are safety critical and they accidentally bridged the on plane wifi into the fly by wire system.
Unless this stuff is written in COQ and formally verified, I won't trust it a bit.
It involves a hell of a lot of work, enough to double or triple the software portion of a project.
Whatever certification you have been involved with may have been a joke.
Safety-critical certification most certainly is not.
BTW: I'm s/w architect on a set of smart meters. And they most certainly are not safety critical. (IMO They should be, but thats a separate issue).
...and no Wifi was "accidently" bridged. evidence?
Similarly, witness recent revelations about car hackability - I forget if it was Blackhat or DEFCON where a couple of guys demoed a fully remote attack through the entertainment system's cell network access that culminated in the ability to override the steering, throttle, and brake controls.
Maybe these are not what is meant by "safety critical" in your use of the phrase. If that's so, I think it would clarify matters greatly for you to define what you mean by it that's more specific than "can kill people if it goes wrong".
- Given physical access, all bets are off.
- "safety critical" refers to software that has a SIL level(https://en.wikipedia.org/wiki/Safety_integrity_level). Note the list of IEC/ANSI/EN standards.
- Yes, the hack of the braking and steering system via the entertainment system is a breakdown of the certification system. That should have been found earlier. This is why critical systems are air-gapped. I personally have never worked on automotive systems so am not familiar with the regulations there, but I was/am extremely surprised this was possible.
Various attacks on car ECUs cleanly show what the problem with current relevant certification processes is: each component is certified separately and nobody cares about the complete resulting system (head unit is not safety critical, engine ECU is, but has no untrusted inputs and there is bunch of things in between that are classified as one of these two categories). What is ironic in the automotive case is that the whole reason why there is immense number of separate ECU's in typical car[1] is safety (ie. limiting impact of one ECU completely failing) and safety certification.
[1] my car has separate ECU for each door even though rear doors does not have power windows and central locking uses dedicated wires. I assume that only purpose of said ECU is that diagnostics system can detect when the door is missing.
I've never heard the 2xN = 1x(N+1) argument, but will that improve for >2 ? e.g. triply-redundant systems?
I think the root problem there is the "wishful thinking" isn't it?
As for dual-redundant vs. triply-redundant it highly depends on whether shutting the system down on failure of one redundant component is desirable outcome, both railway signaling and most industrial systems can and are designed that way, but for systems where that is not possible (either because they need some non-trivial actions to get into safe state or because the whole SIL dance is about keeping the thing running at all costs for business/legal reasons) the dual-redundancy actually decreases the reliability, because additional components handling the redundancy also have their own failure probability (and in many systems does fail more often than whatever it is supposed to protect from failure, especially in master-slave systems that attempt to detect which of the halves had failed and respond to that by failover to the other one).
Interesting approach that is often used for road traffic signalling and general industrial control is that there is second control system, that only checks that outputs of the primary control system are consistent (for traffic signals, it is trivial boolean function of the outputs, usually implemented in hardwired circuitry) and shuts the whole thing down when they are not.
"The described technique cannot engage or control the aircraft's autopilot system using the FMS or prevent a pilot from overriding the autopilot," the FAA's statement explained. "Therefore, a hacker cannot obtain 'full control of an aircraft' as the technology consultant has claimed."
"The statement went on to explain that although Teso may have been able to exploit aviation software running on a simulator, as he described in his presentation, the same approach wouldn't work on software running on certified flight hardware."
http://www.theregister.co.uk/2013/04/13/faa_debunks_android_...
https://www.softwarefreedom.org/resources/2010/transparent-m...
https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFR...
rules and regulations [...] in a different league than what
most developers are familiar with. [...] the design (not the
code) apparently did not take into account any messaging
security, relying on obscurity as its only defence.
You would be achieving a very high and a very low standard of inspection at the same time.
Some people would say standards of inspection are only as strong as their weakest link; that the whole claim of such inspection is to prove the absence of such weak links; and that a standard that has approved such a bridge has, by so doing, shown the claimed proof of safety is a useless joke.
Other people would say the inspection is a checklist of common errors, rather than a proof of the safety of every aspect of the system; that proof of perfect safety was never its claim or goal; and that this merely shows there should be some extra boxes on the checklist, which was in any case a constantly evolving document.
So in your example, it would show that the loading and stresses on your washing lines were sufficiently low to meet the safety margin of the bridge.
Certification of safety-related and medical devices is not a check-box exercise, it looks at the dynamic and static behaviour of your system as a whole (not just the software).
I mean the washing lines weren't fit for purpose (just like the insecure comms wasn't fit for purpose) but the inspection didn't identify them as unfit for purpose (just like the inspection didn't identify the insecure comms as unfit for purpose)
Given that something unfit for purpose was certified, certification doesn't prove something is fit for purpose.
Of course, certification may indicate something is probably more fit for purpose than something that failed the same certification or didn't attempt it.
So your point is:
"we don't know what we don't know"?
True. I see no way around this.
Certifiers typically don't review source code. They are operating at least two levels of abstraction away: they review documents that describe procedures that are supposed to ensure quality.
> If the code was open-sourced, don't expect to find lots of buffer overflow attack vectors
That is a big fat citation needed. Researchers working without access to source code have demonstrated multiple vulnerabilities in safety-critical medical devices. We have no way of knowing how many more they would find if they also had access to the source.
My experience is that, no, this is not an acceptable solution. I have no reason to believe that implanted medical devices will be any better.
Is this problem being addressed in any real way? 50 years in the future some of today's devices may still be operating in peoples' bodies, and it seems hard to believe that anyone would still have the knowledge and/or tools to upgrade them. And surely it's quite a big deal to open someone up to replace the hardware every 5 years?
I've never thought of it this way, and you are right from both a technological and biological perspective.
Biologically we are wonderful containers of nutrients, but we have an army only the very sneaky or militant can overcome. Once that army stands down we are rapidly colonized - which is why we must be so careful with food/meat storage.
Realizing that, it's easy to see why autoimmune diseases are so devastating.
Presumably they sheath these devices in inert plastics or something in an attempt to counteract all this?
Fun fact: Gold teeth caps are really good for you, though maybe not your love life. Gold has very very similar mechanical parameters to your dentin.
Overall, bioengineers are making great strides into these compounds. We are developing new heart stents that coil and expand in response to heat or can just be injected and will solidify and expand wherever they are needed. A cool development sector is in 'smart' hip replacements. They have nano-pores in the titanium to induce bone growth into the implant and create a better and more healthy bond.
EDIT: Not sure what's up w/ the downvotes. There are well-established ways for regulatory agencies (whether FDA, FCC, etc) to obtain firmware for devices -- and it almost always involves a warrant under extraneous circumstances -vs- proactively receiving proprietary code.
Similarly, what about testing for drug quality? You could even extrapolate it out to the SEC's right to examine a private financial transaction in order to determine legality. Or the IRS's right to inspect one's taxes to determine compliance.
Point is (IMO) there is a need put on government by society to bypass some of our "Inalienable" rights. In most cases this societal decision is necessary and makes sense, and I'm arguing that code inspection of life-critical systems is a reasonable example of such a case.
But I also know that the FDA is a massive organization, and there's no reason they couldn't hire for this specific purpose. But then the cynic says that government pay grades may not be up to snuff.
See the HCA rollout and subsequent rewrite.
Sorry, that was a long way of writing, "I don't know".
Think about that for a minute. There has been an official government review process for safety of computer controlled airplanes since the 70s, and a good decade for medical devices (likely earlier) 10 years earlier than the WWW existed.
I think it's kind of funny that you'd think that you were doing it better than production life critical systems THAT CAN ACTUALLY KILL PEOPLE that have been around at least ten if not 30 years longer than the tech you likely (sorry assumption) base your career around.
Your overall point is well taken, but maybe put a little more thought into the examples you pick to support it...
If they really wanted to, they could sidestep the pay issue as every other government agency does, by hiring contractors.
My understanding is that they don't review the code, but they do review all of the validation that goes into making sure the code does what it should.
[1]http://www.fda.gov/RegulatoryInformation/Guidances/ucm085281...
We never got that. For the most part, checking that we followed what we wrote down involved reading more documents that said that we did what we wrote down. The auditors weren't even allowed to walk around freely in our office, nor even be left alone without supervision, and we were all carefully coached on how to answer interviews with non-compromising responses. "I do what the SOP says. I cannot quite recall at the moment, but if you let me refresh my memory by reading the SOP, I will gladly clear it up for you."
It's just smoke and mirrors to pretend like important testing took place. Probably also to make sure whom to blame when things go bad. Some testing does frequently happen, between the document writing about the testing.
I guess that they function as regular police do - the actual experience is less than ideal and leaves a lot to be desired, but without any at all, the world would be horrible.
In normal auditing, they will not inspect your source code - they may inspect everything around your source code (what you procedures for changes are, how you do your testing, etc etc).
However, I believe there's a general understanding that if you fuck up, your source code will be open to inspection - along with everything else. Cause you'll either voluntarily surrender it in hopes of getting on the FDA's good side, or cause they'll subpoena cause you killed someone.
Hold it right there, Karl Marx. We can't just be giving the proletariat access to the means of software production because of one little boo-boo. That could totally bankrupt a company and would be a theft of IP. Rest assured that the proper procedures will be followed and the flaws corrected, but under no circumstances can we take the lawful property of entrepreneurs and daring businessmen.
The FDA probably doesn't even know what source code is. They have vague regulations on how medical devices should be tested, which by tradition has been interpreted in a particular way to mean certain kinds of documents have to be prepared. There are auditors that check that those documents are written. Nobody checks that what the documents say about the software is in fact true because neither those writing the regulations nor those auditing the documents really know anything about computers.
Did the FDA actually check your software or just your documentation? Did the auditors very carefully grill you or just languidly ticked off boxes on a form? Did you find the process of writing documentation instrumental in ensuring that your software was carefully tested? Did you ensure that your tests were reproducible and comprehensive?
None of these things were done very carefully in our case. My superiors were very insistent on the documentation and were quite proud of the quality of our software but they mostly never had any interaction with it and had no real idea of what we did for software validation. They were more concerned with making sure signatures and dates were correct and that our documentation didn't make us look bad.
I think the audits must look more impressive when you're in charge, but as the one actually writing the software I was thinking... that's it? The FDA doesn't really give a damn about what I really worked on, do they?
Class III devices (pacemakers, deep brain stimulators, insulin pumps)
> Did the FDA actually check your software or just your documentation?
In a submission, occasionally. In an audit, it depended on the nature of an audit. I've been part of one audit that was due to persistent SW failure and yes, they checked code.
> Did you find the process of writing documentation instrumental in ensuring that your software was carefully tested?
No, but that's not really the point of documentation.
> Did you ensure that your tests were reproducible and comprehensive?
Yes, we could kill people if we screwed something up.
> They were more concerned with making sure signatures and dates were correct and that our documentation didn't make us look bad.
This practice is common, I grant. Doesn't mean it's universal or right.
> The FDA doesn't really give a damn about what I really worked on, do they?
FDA heavily triages activity based on risk. If your product is life-sustaining they will definitely be all up in your business.
https://www.youtube.com/watch?v=5XDTQLa3NjE
It's mentioned in the article.
Ever since then, I've been very interested in all the paperwork sent to my grand father about his pacemaker + defib implant. Recently they mentioned to him that they will send out a remote monitoring device and Karens talk immediately jumped to mind.
Initially the documentation talked about how the remote monitoring device needed to be held close to the heart for a few seconds in order to download the relevant data. This gave me a modicum of hope that at the very least it was some sort of NFC type communication. This would at least make it harder to physically exploit.
However, they continued to say that if you want it to monitor you every night, just put it within three metres of your bed while you sleep. I had a search online and there does not seem to be a lot of people particularly interested in this area, although repos such as https://github.com/openaps/decocare give me hope in the ability of the community to reverse the relevant protocols and investigate these devices.
"Unpatchable - Living with a vulnerable implanted device"
There is a lot of uninformed discussion in here currently.
(I realize this sounds like a low-effort joke, but think about it for a second.)
[0] - http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
On the good side, bugs are fixed and vulnerabilities closed eventually. But the process is so slow... It spans numerous minor releases.
Not a rhetorical question.
If you're OK with someone else (who probably doesn't have your well-being anywhere in their list of priorities) controlling all of your devices, then the answer is no, you are not required to have access to the source code and output data.
EDIT: I didn't realize this was such a controversial statement. I stand by it, though; even as I knowingly own many such devices myself. I can certainly see, however, how I may want more control over a pacemaker than, say, my phone.
I'm sure I'd lack a lot of the heart mechanics side of the equation, but I would be very highly incentivised to hunt out programming bugs.
The majority of the code, especially if as said in the article it has wifi and listens to external requests, is code that requires only expertise in more general software, of which there are many many people who are qualified to review such software (or write it).
In the case of a pacemaker or other medical devices, this liability could even include manslaughter.
(As an aside, the suggestion that a software developer writing firmware for pacemakers "probably doesn't have your well-being anywhere in their list of priorities" seems unfair. You wouldn't believe the amount of effort that goes into ensuring that critical systems behave properly.)
I wouldn't either. I have seen some of it, and not that much effort goes into it. A lot of paperwork goes into saying that the effort was done, but the actual effort carried out pales in comparison.
It's more about finding someone else to blame than to make sure nothing bad happens. Organisations like the frickin' FDA who have no idea how software is developed are in charge of worldwide standards of medical software bureaucracy.
Now we find people who like yourself have to specify that the radical position that all software should be free is something worthy of serious consideration. That they're not joking or trying to be deliberately provocative.
What happened to us? Why did we go from boasting about installing Linux on a dead badger and talking about how all software will some day be free to being afraid to seriously consider the proposition?
Basically, it's easier to tell where there is no Linux than the opposite. That surely doesn't sound like such a failure. It's as if all the other OSes and systems are front-ends for Linux subsystems.
It's an interesting proposition.
HN is clearly not attracting much of the FLOSS crowd.
What happened to us?
Because HN started as part of YC its culture really likes VC-backed startups. And we think VCs want the kind of huge returns that are seen more often by closed source companies - they want to back the next Microsoft or Apple or Google or Facebook or Paypal or Amazon, not the next Red Hat or Canonical or MySQL.
It wouldn't make much sense for people to believe free software is a moral imperative, while aspiring to launch huge closed-source companies.
It's good enough for most people, we have some Unix under the hood if we go looking for it, free software has mostly won. Or something. I really don't know. I miss the rebels of yesteryear.
http://www.sylvialiuland.com/2012/01/mafalda-classic-cartoon...
The impression I get is that it's mostly the opposite: people have mostly given up on the notion of having all (or majority of) software being Free, and see that more and more as a pipe dream. Hence they don't want to "waste" their enthusiasm on something that feels like a (literal) utopia now.
There's a lot of reasons combined to get people there:
* seeing corporate behemoths everywhere that want to keep critical software proprietary
* many more people wanting to monetise their own software, and seeing that GPL-ing software offers only a few (and not-universally-applicable) monetisation strategies
* seeing that the "all bugs are shallow" and "everybody pitching in will make Free software the best there is" turn out not really true in practice
* seeing that in practical terms, having access to read and modify source code means nothing in 99% of cases because of software overload and others' code being so hard to get into that it's often practically impenetrable.
Not to say that I agree with all or any of these completely, but it's easy to see such a multi-pronged attack pushing back hard against the FSF's message. Further reducing FSF's ability to spread the message is the widespread character assassination of RMS as an "insane, overly idealistic, and rude slob" that has been going on for at least a decade.
As someone that read the Halloween documents non-stop overnight, it makes me a bit sad too.
I've been reading up on aviation regs due to a recent interest in getting a pilot's license. By my understanding, airplanes certified by the FAA as airworthy undergo some fairly heavy testing to exactly determine and prove what their capabilities and limits are. Given that getting your prototype wrong can cause the plane to crash and kill your test pilot, there's incentive to get this stuff right. Furthermore, once a plane is type certified, it can't be modified from that configuration without further testing to prove the modified configuration. Thus you'd better be damned certain that your engine control computers are correct, lest the FAA revoke the plane's airworthiness certificate, making that model unsellable, nevermind the lawsuits of the survivors of deceased passengers or insurance companies recouping their losses.
Additionally, from what I've seen, most general aviation airplanes are stuck in the 60s as far as engine tech goes, in part because of the strict FAA regs. We're talking air-cooled engines with carburettors here, no engine computer to speak of, or if you're running a fancy modern engine, mechanical fuel injection. Unless you're flying a brand new Cirrus SR22 or Diamond DA20, there's no code to inspect. Even if there was, it'd be in the avionics. You don't need a nav beaon, radio, or transponder to land an airplane safely -- though they can make it much easier, and the FAA is going to want to know what happened once you're back on the ground.
As far is large passenger jets? I'd say as a passenger, no, you don't have access to the code. It's not your airplane; it belongs to the carrier.
Personally, if I were to own a plane and fly it, I very much would want access to any and all code that makes the plane go. Though if I'm going to be hacking on said code, I suppose that's what the FAA's Experimental category is for.
Would this work similarly with autonomous cars? For example, what if google wants to change one line of code in their car?
A pacemaker is something you bought and owned, when flying in a plane you are buying a service, this is similar to earlier discussions about being able to change your car's software under DMCA etc...
A pacemaker is also personal, in that it's something that only you have and for your specific pacemaker the only affected party is you. A specific flight has hundreds of affected people which makes the burden of responsibility (for lack of a better word) shared between many people.
Sure about that? How much did you pay for, and how much did your insurance cover? What's the proportion look like?
Before you downvote, note: I don't like this argument. I find it downright horrifying. But I can't imagine no one will ever make it in a serious way, so it bears considering how to respond.
If no one can take it from you (or you need to return it) without your consent then you probably own it.
It might be a little vague legally, but we aren't talking about legal issues here (since it's perfectly legal to not have access to the sourcecode of something you own), but rather a perception issue.
Yes, and it has been. See "DO-178C, Software Considerations in Airborne Systems and Equipment Certification" [1] for example.
Also:
Yes, but you don't have to. You're not gonna die if you don't put your life in their hands. But with pacemakers and such, you have to get one or you die. Then, you depend on the manufacturer.
Ask any question and you will find an answer. Any question you have, no matter how banal or left-field. What is the weather? Is my grandson a lesbian? How do I eat pizza in Italy?
Where is the biography section? How do I understand the Dewey Decimal System? What is in the Special Collections, and what are the hours -- and do I need an appointment? The computers are down... is there a way I can search for books offline without randomly roaming the stacks?
Now, in this case, FDA approval should include software standards that prevent this scenario. Done*.
For example if someone invents an artificial device that helps me get rid of diabetes I would be super happy. It is only a generation later we would demand that device meet certain quality floor. That is the case with all innovations.
Government however can put a very hard nail into the head of an innovation.
She doesn't know the code that's running on machines inside her body.
I don't even know the code that's running my heart.
And yet, I trust it.
She found out what happens when one of them gets triggered.
I'm sure she must have signed a user license agreement of some kind upon buying the device. So she shouldn't have to complain.
There's a competitive advantage in keeping pacemakers' source code proprietary?
Open source is not necessarily any safer (heartbleed bug ... )
It's a hell lot more amendable to inspection than proprietary software.
There definitely is. If your company takes ~2 years to develop a pacemaker's software, it's not to your advantage to let your competitors catch up. I'm not saying it's a good thing that it's closed source, but there is definitely incentive to keep your research to yourself.
Why should the patient who has the pacemaker implanted care? This seems like a clear situation in which the patient's interests trump everybody else's. Pacemaker manufacturers should be competing in how well their devices meet patient needs. Closed source doesn't meet a key patient need.
I mean, I get the business incentives involved, but I also think it's high time to realize they're getting insane and should be altered.
Exactly what patient need do you believe open source would meet that is not being met by the current closed-source development process?
How do you know this?
For starters, without access to the hardware and understanding what it's supposed to be doing, how do you know if the code is right or not?
Let me clarify that I'm not opposed to open sourcing the code, once issues around Trade Secrets are handle. But I don't think it would have even remotely the impact this group believes. The vast majority of device problems I've seen were system issues, not just a coding error.
I also don't need to know much about the intended function of the device to find coding styles that cause problems in an embedded environments.
The need to know, and be able to control if necessary, what a device that affects your life and health is doing.
I'll give an example from my own experience. I have sleep apnea and have to use a CPAP machine. There is a nice little SD card in the machine that records detailed data from every use--which means it records how long I sleep, how soundly I sleep, how many (if any) apnea episodes I have, etc. This would be very useful information to me, but I have no way of getting it, because the software and data inside the device is proprietary. The only way for me to have anyone look at this data is to pull the SD card out and take it to a sleep therapy doctor, and even then I won't see the actual raw data; the best I'll get is some proprietary analysis report designed by the device manufacturer.
Furthermore, the ostensible use for the data is to be able to adjust the pressure the CPAP machine is set at in order to improve the therapy. If I had access to the data and the settings inside the machine, I could easily do this myself. Instead, if I want any adjustment made, again, I have to go see a sleep therapy doctor--which means I need to make an appointment, which usually means waiting weeks or months, and I need to take off from work to go to the appointment, etc., etc.
It should be obvious how open source would greatly improve this situation.
Security by obscurity obviously doesn't stop a determined attacker, but it does raise the barrier to entry for script kiddies.
Tell me, how many vulnerabilities are running wild on Linux, the software that powers... well, pretty much anything (including the servers through which you read this content)? Even if you find a vulnerability, it gets patched within hours and it may take a day or two for it to be distributed to everyone.
> Security by obscurity [...] does raise the barrier to entry for script kiddies.
Which script can help you find a vulnerability inside of the source code? You'll need a script that can understand the code it's looking at. I'm not aware of any such script.
I'm not advocating for security by obscurity in the slightest because on balance I think it's bad, but we should acknowledge that publishing your source does change the potential cost of mounting an attack in various scenarios, and some of them might actually favor obscurity.
Nobody except the most determined attacker will attack a device with some custom, unpublished code. On the other hand, popular software have a variety of exploits in the wild because their popularity makes them more attractive targets, one consequence of which is enabling script kiddies (since the hard work can be outsourced).
> it gets patched within hours and it may take a day or two for it to be distributed to everyone
This doesn't work for embedded devices. Hence why it might not be a great idea to publish their vulnerabilities, or tell the whole world that you're running on old vulnerable source.
This is patently incorrect. Even the underpowered Z80-clone micros with 18kB RAM I was writing firmware for 15 years ago had trivially updated firmware; modern devices are even easier.
The article even mentions that wireless firmware updates is a feature:
Then she bought a pacemaker programmer online,
and she and other hackers figured out that it
could be used to update the code on her implant.
Relying on the laziness and ignorance of the attacker is a terrible idea.
> Hence why it might not be a great idea to publish their vulnerabilities
This is why any it's important to practice responsible disclosure. The manufacturer should have a reasonable time to make their patch before telling the internet.
We live in a world where even phones don't get patched as frequently as they should; you expect end users to patch their pacemakers?
Putting them online and allowing auto-patching would probably be worse since it also increases their attack surface drastically.
> This is why any it's important to practice responsible disclosure.
Right, and fact is that 'responsible disclosure' ends up looking a lot like obscurity in a world where you can't guarantee that devices will be patched before an attacker would be interested in exploitation.
I have no idea what the current patch rate is for pacemakers, but they do happen. The use of radio was a feature specifically to allow updating and management of the pacemakers while avoiding the serious risks of surgery. The pacemakers would be patched when the patient shows up for their next checkup appointment, which is probably every 1-2 months. They already connect to the devices for regular diagnostic purposes at those times.
If the problem was severe enough, calls would be made to the patients to come in right away. Medical services already handle problems on a priority basis ("triage"). This already happens for other types of problems.
> 'responsible disclosure' ends up looking a lot like obscurity
That's a circular argument. You're implying that the manufacturer wouldn't want to fix their product, which is highly unlikely. The only reason they are resistant to the idea at present is because the source is closed. The entire point is that by opening up the source the community can work with the manufacturers to fix these problems.
You're arguing that because the current system currently doesn't patch bugs that often, we shouldn't allow more debugging. Pretending that either bugs don't exist or that malicious actors won't find them without the source code is dangerous. "Pride goes before the fall"; do you really want to bet - potentially with your life - that all malicious actors are too stupid to find security problems? Hint: many medical devices have already been hacked (without the source). Or do you want to let the community at least attempt to find the bugs first?
In other words, it's a ready-made vector for potential attacks, as long as an attacker can get close to a pacemaker with a radio. You could probably even put a powerful device near a hospital and pick up random people coming and going.
> Or do you want to let the community at least _attempt_ to find the bugs first?
The promise of open-source security has always been that you let many eyes look for problems, then you patch ahead of potential attackers.
Logistically, that's extremely problematic for embedded devices. Every time a vulnerability is found, you ask the patient to go back to the doctor to get it updated? That doesn't scale at all.
The idea that device manufacturers will change their entire engineering and security philosophy is somewhere between idealistic and naive. I sincerely hope it's the former and not the latter.
Now there is an argument that there is a trivial way to find vulnerabilities in Open Source code - just diff the commits to look for fixed bugs, and attack those who didn't manage to update their software yet. That's part of the reason why e.g. Wordpress blogs and PHPBB forums get spammed so heavily.
Whether or not the benefits of Open Source are greater than those problems is another topic, but let's not pretend opensourcing doesn't lower the entry bar for attackers.
Security by obscurity is never a good idea, but especially not when it might prevent a white hat from finding a bug that would allow a malicious actor to remotely STOP MY HEART.
1) White hat finds a vulnerability in the source code which applies to a large number of devices. 2) Source is patched but vulnerable devices exist in wild
Now all an attacker needs to do is find a vulnerable device; because the source code is public like OP suggests, figuring out which devices are vulnerable is trivial.
Unless I'm missing something drastic, this is actually a problem in the embedded space where obscurity seems to help.
Put another way: If you had to explain Marie Moe's death to one of her relatives, which of your three points would make the relative understand your position?
We do not allow food companies to hide the ingredients of their products, because we know that companies (in the interest of profit) will fail to inform consumers about the dangers of eating unhealthily. Similarly, we do not allow meat producers to leave their meat ungraded because it endangers their "competitive advantage" - we understand (empirically) that doing so leads to food contamination and otherwise preventable disease.
Your choice of proprietary medicine as a counterexample is an interesting one. The pharmaceutical industry in the United States has a long history of "protecting its competitive advantage" at the cost of individual welfare - consider how frequently "reformulations" of the same base chemical are patented to continue milking a lucrative product that could improve the lives of thousands if genericized.
Perhaps even more pointedly, consider the fact that we deem it acceptable (and necessary) that the FDA step in and regulate the release of new drugs. Is there a valuable distinction to be drawn between the sort of regulation and evaluation that the FDA does and the sort that would be possible if programmers could openly evaluate medical devices?
You're attempting to justify the release of a (potentially fatally) flawed product on the grounds of lost "competitive advantage".
If you lost a family member to a faulty (yet certified!) medical device, would you feel that their death is justified by a marginal improvement to some company's bottom line? What if it wasn't a medical device with closed code, but a miracle pill whose manufacturer refused to release the formula to the government for analysis?
The argument I made in my original comment does not rely on the (un)soundness of your original claims - it is a moral argument about the value we place on human life. I'd be willing to bet that, even if you think that closed source is superior in every fashion, you would never be satisfied by your own justifications in the case of a personal tragedy. This contradiction suggests that, at the very least, we must draw a distinction between justifiably and unjustifiably closed source code.
2) Open and closed are equally vulnerable. Closed may be even more vulnerable, since security researches have to attack a black box and can't run ordinary code linting tools.
3) Liability after update is a good question, but one which can be (and is) solved with traditional contract agreements. See cpap machines.
> 1> Loss of competitive advantage
Patient safety trumps business considerations. Skipping clinical trials would be a major competitive advantage (lower costs, quicker to market etc.), but we don't allow that for the same reason
> 2> Open source is not necessarily any safer (heartbleed bug ... )
I'll agree, in so far as saying making it open source doesn't necessarily make it safer. However, it should, at worst, be no less safe - the requirements on the manufacturer (testing etc.) should be the same regardless
> 3> If software for the pacemaker is allowed to be updated like that on a computer, someone will update it with buggy software that can cause adverse side effects.
You can open the code but, for example, required a signed binary before a device can be updated. The openness of the code and the openness of the device itself are linked but separate issues.
About that ...
Have you seen http://www.fdareview.org/05_harm.php and http://www.fdareview.org/07_market_failure.php?