Book Review: "This Is How They Tell Me the World Ends"
addxorrol.blogspot.com
addxorrol.blogspot.com
What causes that vulnerability? Lack of liability for software vendors. If Microsoft had to pay the costs of vulnerabilities, we'd have far more secure systems.
There's one area of the industry where companies are held financially responsible for their mistakes - gambling systems. A few percent of revenue goes to paying for mistakes. For GTech, before they were acquired by a non-US company, you could see the numbers in their annual report. It's not a killer.
I don't think it makes much sense for a random one-off script written by some lone developer starting a company to be subjected to all the alphabet soup of regulation, but I don't think letting everyone get away with security breaches forever is a good idea nor is just throwing up our hands and going "oh well, we're going to keep getting break-ins" for another 50 years.
That seems unlikely to happen as long as it is in commercial software businesses' best financial interests that it not happen.
So, if a company is small and unable to afford any security, they can just set the number at $0 which means it is the customer's problem. However, if the company is big and people have security expectations, they would need to specify a reasonable number that would alleviate those concerns. As long as the number is clearly communicated and you are not allowed to fraudulently or misleadingly advertise a different number, then customer's would be able to make an informed decision with respect to the security level and liability they are accepting from their vendors.
Obviously there are some complexities with respect to properly communicating this information. For instance, for a consumer-facing company you would probably want a per-consumer number instead of an aggregate number. As an example, if Apple were to claim $300M for the iPhone, that might seem like a large number to an average person, but that would only amount to ~$1.50 per iPhone sold in a year. You also need to prevent misleading advertising that might attempt to divert from the quantitative liability they are actually accepting. However, the base scheme of mandatory labelling requirements along with opt-in liability should allow for a solution that is hard to bypass while being flexible enough to support both small and large businesses.
And if a company is large and low margin, they can do this too so nothing changes. A few companies will be out there promising things, like today, and maybe they can deliver and maybe you can sue them if they don't, like today.
Being able to actually get your money back... is another problem. But obviously a toothless regulation is toothless, and changes nothing, so its not a useful assumption to make
This is would be different than the current state of affairs because lying or overstating your security story would have direct, pre-committed, quantitative downside. This is contrast to the outcomes of a lawsuit which are expensive, highly variable, generally require proof of intent, and depend on demonstrating actual quantitative damages.
The decision to take on risk for the purpose of marketing is not at all palatable to decision makers. Why would you want to accept an increase in risk for an unknown-unknown?
Meanwhile, quantitative, money-linked claims about software security actually do happen in rare edge cases, already.
It's not about creating 'perfect software', but about holding developers accountable. If a customer accepts a contract with a faulty spec, that's on them, not the developer.
>But even if it is you still have to worry about every layer below you in the software stack on top of the hardware. You might do everything perfectly at great cost and it all gets rendered moot by spectre/meltdown/rowhammer etc.
So what? It's not your fault. There's only one party that should be held liable for spectre, Intel.
Well, if you have formally verified your system, you know that if the underlying systems work correctly, your work will correctly adhere to the spec. So if a breach occurs, there's two options - either the spec was bad or there's a hardware/OS/library bug.
Otherwise, you can't always know it, that doesn't strike me as a realistic goal in the first place. It's not about always being able to perfectly assign liability - obviously if you can't figure out how something was compromised, you simply can't assign any liability. But one could investigate and if you find deviations from the spec, you know where to correctly place blame.
>You'll look around and definitely find bugs in the software though.
If these are deviations from the spec, I think it would only be right to hold the developer liable. If a developer doesn't want this to occur, they could instead formally verify their software.
If the bugs are in the spec instead, the blame should lie with the party that accepted the spec.
Do you literally mean "nobody" here? As in if I wanted to hire you to put a team together to reliably ship secure commercial software, you couldn't do it either?
I think we have, as an industry, tacitly agreed that it's better to ship cheap, insecure software than it is to ship expensive, secure software. A lot of the innovation in software has occurred specifically because it is so inexpensive.
It seems clear to me that, so far, this tradeoff has been a net benefit, but at some point we may want to trade the inexpensiveness and innovation for some security, particularly has we start to put software into more things that can kill and maim us.
It's fundamentally impossible given that you won't control everything about the operating environment of the software, and can't see into the future.
I mean, even if you could create a perfect piece of software with no bugs present at the time, that's still not enough, because vulnerabilities in dependent components will arise which were unknown at the time.
For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
This is on top of software development being such a complex process that even the best of its practitioners write code strewn with bugs.
Why not? Given the cost of writing secure software, for all but the highest volumes, you can ship it as a dedicated app on its own hardware.
> For example, if you'd written the perfect web browser a few years back, how would you prevent vulnerability to CPU-level attacks such as Spectre and Meltdown discovered later on?
Given that we are talking about liability here. Presumably Spectre and Meltdown would be primarily liabilities for hardware vendors. Software shipped after the vendors recommend software mitigations would shift that, but I don't think
> Or consider if you write the perfect wireless networking firmware, but then end up being subject to vulnerabilities in the protocol standards (WEP was a good example of this)?
First we must put aside the fact that WEP raised red-flags for cryptographers far before an exploit was found, and the fact that those concerns were dismissed with a hand wavy "People can tap into Ethernet lines too, this is just 'Wired Equivalent Privacy'"
Now, I think that if you write software that faithfully implements a protocol, advertise it as implementing that protocol, and it is exploited not in-spite-of but because it did so, what exactly are you liable for?
Even knowing that WEP, WPA, WPA2 and WPA3 are all flawed protocols, I'm sure there will be a WPA4 some day that is also flawed. If the only thing we had to worry about in WiFi stacks were protocol flaws, attackers would have a much harder time of things. No flaw in the WEP protocol allowed gaining execution access on the WiFi adapter, much less the host computer (though given the bus designs of the time, the former inevitably led to the latter).
But rewind to last year, when I was still at Latacora. Ask me the question: can I hire you to make my development process more secure? Sure, I think I can add value there. Ask me instead: can I hire you to make my software liability-proof? Not a chance.
Take 9 strong software security people and divide them into 3 teams. Give each team a stretch of time to find vulnerabilities. There will be a ton of overlap in the findings, but each team will find different things. One team will happen to have a specialty in the low-level details of the platform it's running on; another will spend 80% of their cycles thinking about SSRF and HTTP desync vulnerabilities; still another will happen to have a reliable process for exhaustively testing authorization. Don't get me started on K8s or Electron or cryptography or iOS or browser extensions or Node.js.
And that's all for pretty standard app stack stuff. None of those problems are the subtext for Perlroth's book; the vulnerabilities she's trying to write about are in low-level C/C++ code. The biggest security teams in the world have build enormous, carefully-instrumented fuzzing farms for this stuff and they don't find everything.
I literally mean nobody.
I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
> I think you could stand a chance if you designed your product to fit the limitations of what we can deliver reliably. You'd lose most of your features. You'd ship boring CRUD stuff exclusively on your own dedicated hardware, and you'd build it with minimal library support. It would be excruciating, but people do it. Just not for any of the kinds of software we normally interact with.
I think we are in agreement then. If you are targeting K8S or Electron, you've already lost the battle against many threat-models that matter for liability. I think what you are pushing back against is people imagine bug-free software and they think "oh maybe 4x more expensive and it will look like it's from 2005" while the reality is more like table-stakes of 10x more expensive and it will look like it's from 1980.
I personally think that the tradeoffs are worth it for more industries than currently make it, but I don't think the law can be nuanced enough to both be effective and not cause massive collateral damage to many other industries. Look at the chilling effect regulation has had on software in the medical industry, and yet those regulations seem to have done little to increase the resources required for an attacker who wishes to exfiltrate patient data.
Nate Lawson used to tell me (I assume he'd still tell me) that companies trying to ship secure software need to budget 10x for verification what they do for implementation. The best companies in the world don't even budget 1x. I'm not sure it's reasonable to ask them to. I think we'll know stuff in 10 years that will change how we look at things.
In my opinion, if Jeff Bezos, Satya Nadella, Sundar Pichai, and Tim Cook all got together and wanted to put together a team that ship secure commercial software, they would probably be unable to put together a team that knew how to get to the $10M level, and there is zero chance at the $1B level. If they all worked together and put their full focus and resources on the problem, they might be able to put together a team that could invent how to get to the $10M level, but they would not be able to put together a team that already knew how and had experience doing so. The key difference between these two cases being that in the former the team would need to invent good practices and experiment to see if they worked, where as the latter only needs to apply already existing known good methodology.
If you want some 3rd party evidence for my claims, not that I am claiming this is exhaustive proof, bug bounties that get claimed are, generally speaking, less than the cost of discovery for a bug otherwise security researchers would not put in the effort to discover than claim them. Not a single one of their premier products that they ship and sell [1][2][3] offers a bug bounty over $1M for total and complete remote zero-interaction compromise. It is very unlikely that they can do an order of magnitude better than the premier products just because they want to.
[1] https://www.microsoft.com/en-us/msrc/bounty
[2] https://www.google.com/about/appsecurity/android-rewards/
I see no reason why the same can't be applied to software. It's probably true that we can't feasibly create software and computer systems that are completely free from security vulnerabilities, but we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
Just like with cars, there is always going to be some tradeoffs between cost and security/safety, and exactly where to draw the line is something reasonable people can debate.
> You don't say your car made a mistake when it refuses to start one cold morning. There's essentially universal agreement on the 'semantics' of cars and bridges.
Since you mentioned it...I don't know whether this is the case, but I would guess that there might be regulations about the temperatures at which car batteries and other components must be capable of operating at. And, as car automation software becomes more advanced and prominent, I think we'll be discovering that there is not in fact universal agreement on the "semantics" of cars. There is probably already considerable overlap in the subjects of my analogy, for recent self-driving capabilities but also for much older standard things like airbags and anti-lock braking systems.
But my broader point was that this analogy is basically what 'can god make a rock so big they can't pick it up' is to a discussion of theology.
The situation you are describing is that automobile safety is implemented primarily by establishing standards and only secondarily by the existence of liability (though it's important).
Now, if you had a large company that bought cars and modified them, that company would itself have an obligation to do it safely too. And having regulations about that would be an absolutely necessary part of liability and safety.
...we absolutely have discovered some vulnerabilities which can be protected against using certain software development techniques, and I see no reason why software vendors ought not be liable for damages caused by their failure to follow these established techniques.
I would note that the existence of zero-days is absolutely not the most pressing problem of software security. Having a reasonable system of connecting X to Y is far more pressing. You can get in as much trouble with a misconfigured system as with a vulnerable system. But broadly, your point that there should be standards, is good.
Indeed, regulatory standards is what I was hinting at, not just civil liability (although I believe both exist for car manufacturers, and should probably exist for software vendors).
Things like institutes that study this situation outside the government itself may be necessary to establish trust at this point.
And you can't make the standards primarily about software vendors.
Secure software lies on a spectrum. We know memory unsafe languages are dangerous yet critical software is still being written in them. Liability would drastically change the domain in terms of discarding practices we've known for decades are deadends, and raise the bar significantly.
Doesn't sound so bad of a tradeoff, enforced for certain classes of software. Like one expects liability for medical, banking, flight control, etc. software, it could be extended to OSes, server programs, and anything handling user accounts...
Software with less impact if it's not secure, like e.g. a media player or a text editor, could continue to ship free from liability.
I think you would agree that having critical string parsing logic written in C and shipped in an opaque binary is both negligent and more dangerous than a reasonable person would expect. Traditional products liability claims focus on those kinds of questions: was there negligent conduct? is the product more dangerous than a reasonable person would expect?
Attaching some kind of liability is a good way to encourage the industry to settle on standards of non-negligent conduct. If you provide vendors an opt-out mechanism (e.g. disclose your source and there's no liability), it will not dramatically interfere with the cadence of development.
Common law is very well suited to establishing standards in a dynamic environment. It got us through the first industrial revolution.
> This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
The author including that line is saying “despite what you think this software may do (creating implied warranties and the like), it doesn’t matter. I can’t be held liable for your usage of this code/software.” Would forced liability mean open source licenses like the GPL wouldn’t be able to have those sentences mean anything?
Numerous threads on HN lately have discussed the way that "connect everything to everything" approaches wind-up inherently insecure and a wide range of organizations have no incentive to stop them. Water systems have no need to connect to the Internet even indirectly and neither automobile remote start system but organization managing these things have no incentive to take this stuff seriously.
This is what's called an "externality" and a difficult one. Externalities are generally best dealt with be direct regulation but security is a bit different externalities because defining best practicing isn't. I could imagine a "security institute", only focused on defense and independent enough to be trustworthy to most institutions. But I'm not all that optimistic such a thing could be created.
Vendors hated it. I've seen a vendor group document asking NSA not to publicly certify systems at the highest levels because it make them look bad. They got DoD to go for a system where the testing was outsourced to private labs, and you could keep going back for a retest over and over again until you "passed".
Amusingly, computer security in those days wasn't a high priority at NSA for a bureaucratic reason. Computer security was at FANX, not NSA HQ at Fort Meade. FANX is the "Friendship Annex", a few miles from NSA HQ, near what used to be called Friendship Airport and is now BWI. FANX had all the minor functions of NSA - personnel, procurement, training, etc., and was considered a dumping ground for losers. Putting computer security out there doomed it.
What really killed security is that from 1940 to 1980, DoD was the major consumer of advanced computer and electronics technology. By 1990, the commercial sector was dominant.
Here are the early security standards books.[1]
Yeah, ok but lets trade out MSFT for FOSS, Firefox or small iOS app developers.
At some point, we should ask to see if we are inadvertently advocating for scaring people into doing the right thing. Worse yet, for something that they couldn't see or anticipate.
What about the standards makers? Are they going to be held accountable too? When is the last time you've read OWASP articles? Have you seen the state of that sort of literature? Its a mess. Have you seen how arcane and arbitrary the NIST-800 series is? What is the limit to this sort of "accountability"?
At the risk of being even more trite, ideologically motivated statements like this look cool only because its on the internet.
If it doesn't get better than that pretty quickly I'm unlikely to finish it.
No one knows who the Shadow Brokers are, unless that’s covered later in the book.
In reality I think it's kind of an embarrassment how much CNE technology goes the other way, from industry and research to the IC. Not for moral reasons (though: I would have ethical problems selling bugs to attackers of any sort and am thankful I don't produce the kinds of bugs that have this market) so much as "it's not that hard to do this work and we should be getting more for our tax dollars".
Stuxnet is close to a work of art; the degree of effort involved in hiding self-reinstalling malware inside hard-drive firmware is staggering.[0] Anyone who's ever tried to write Linux drivers for undocumented hardware can appreciate how insane it is that they managed to do it for a dozen different hard drive manufacturers.
[0] https://www.theverge.com/2015/2/16/8048243/nsa-hard-drive-fi...
But yes, making it work for a bunch of manufacturers would be a lot of additional work (though easier if you can obtain the data sheets for the embedded microcontroller, which is probably doable if you're the NSA).
Side note: don't blame me for the URL, it was just the first place I found that had a PDF of the paper.
"So what is spoken in the Bronx, Mexico City and Madrid? Castilian, which is Spanish, is what’s spoken, it’s spoken in those places. So you can erase that American prejudice from your mind if you ever had it. It is true — I mean, I had a colleague at Cornell, a very distinguished American medievalist who would ask me, “In what language do you write your scholarship, Roberto?” I said, “What do you mean? In English and Spanish.” He was worried that if I didn’t write in English being in Latin America, I didn’t have a language of culture in which I could write, because he didn’t think that in Latin America we had a language of culture. It is an American prejudice."
https://oyc.yale.edu/spanish-and-portuguese/span-300/lecture...
- Author believes (or is at least attempting to make others believe that she believes) that the NSA is benevolent and not committed to any specific domestic program
- Author frequently stated that 0-day purchasing platforms should not exist as it is a matter of opaque moral policy.
- Author frequently states that contractors like Immunity are problematic because they sell tradecraft to "bad nations" for profit.
- Worth noting that the author is also a NYT contributor and clings to a lot of those same notions.
The overall sentiment of the author after her webinar was that she was egotistically confined to believing in US supremacy and very much in the midst of an ideological war against competing beliefs. Probably not going to get far in her book either in light of that webinar.
Are you genuinely interested in a book on policy? (If not, I might have some recommendations, depending on what you're interested in).
If so, you might need to adjust your expectation bar toward the lower spectrum. I honestly can't imagine a book about policy that isn't trash, but maybe that's just me.
[1]: freedictionary.com