Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
usenix.org
usenix.org
Start with the Internet of Things example. He chalks up the abysmal security record of IoT devices to two factors: it keeps IoT devices cheap, and IoT vendors don't understand history. And there's a lot of truth in both these assertions! But they are both just expressing facets of a deeper, more fundamental reason: IoT devices aren't secure because their customers don't demand security.
This deeper problem completely explains why the two higher-level problems he observes exist. Making your product secure makes it more expensive and slower to come to market than just leaving it wide open, and the IoT vendors know their customers care about cost and availability and don't care about security. So they do the rational (in the homo economicus sense of the term) thing and optimize for things their customers are actually willing to pay for.
The same causality can be observed in the ML world. Mickens asks why people are hooking ML systems whose operation isn't fully understood to important things like financial decisionmaking and criminal justice systems. The answer is that the customers demand it. ML is trendy and buzzworthy, so if you're a vendor of (say) financial systems, and you can find some way to incorporate ML into your offerings with a straight face, now you have an attractive new checkbox on the feature list your salespeople dangle in front of potential customers. And once the effectiveness of having that box checked becomes clear, you kind of have to do it, even if you know it'll be ineffective or even worse, or risk losing business to a competitor with fewer scruples.
All of which is to say that what we see playing out in both these scenarios isn't really the vendors' fault. They are instead classic examples of market failure. People end up buying shoddy products because spotting their shoddiness requires technical expertise they don't have; responsible vendors who try not to make shoddy products lose sales to irresponsible vendors who don't; eventually all the responsible vendors are out of business and the only products available to buy are shoddy ones. There are lessons to learn from this, but they're economic rather than technological.
We trust doctors to take into account all the nuances of medicine that laymen have never even heard of, and give us good advice. Because not everyone can be an expert on everything.
Its the same with software. We can't expect everyone to be an expert.. its up to our industry to act responsibly.
Sure its "market failure" in so far as duping uninformed people is a good way to make a quick buck, but the deeper issue is moral failure / failure to take responsibility.
The benefit is that medicine is effective and measurably safe. It’s obviously necessary, and the supplement industry shows why.
They do push life-explodingly addictive and harmful painkillers on their patients, despite knowing the harm it does, because regulations don't prevent them from doing that.
What would be the consequences of an FDA for IoT? Huge price increases, sudden workability of patents as a means of protection, but more security and better products?
And who would have had some of the most input into said extensive regulation?
If drugs started exploding testicles you can bet customers would start demanding they didn't (male customers at least). Just look at the Thalidomide incident, I've seen it in the news within the last decade and it happened nearly 60 years ago at this point.
The market has spoken, cheap wins over secure time after time. The consumers know, and they don't care, because to them the stakes are just not that high. Their genitals will be fine, and who wouldn't mind an extra set of eyes on the front yard.
This idea that people understand the total consequence of what they do with their money is so simplistic that it's stupid. Markets don't "speak" from a vaccuum, they demand what their constituents are convinced is valuable, regardless of the accuracy of valuation. Lies sell garbage all the time and the consumer isn't to blame for wanting it, the professional, skilled, psychology-wielding liars who sold them on it are.
Hypothetical example: pay for a bunk study that concludes eating apples prevents hair loss, benefit for decades, with negligible repercussions to your business when the lie is uncovered. I'd be skeptical if you claim you can't identify several real examples yourself.
I really dont care if my tea kettle is part of some botnet. I cant even imagine a reason why I should care. I guess it sorta sucks for the people getting ddosed :/
For instance, my router cost me $20. It is probably full of security holes. I dont care. To buy a router that was secure I would have to pay more than $20. I would not feel any benefit for spending that extra money. So I don't do it.
On the other hand, as a developer I'm always thinking about security because it is fun and I feel a sense of responsibility for the things I make.
So consumers would of course be happy if you made plastic straws - look at how many get sold!
Now if you told people they would not have plastics, and everything would cost 5x more because we dont have a cheap packaging option, OR tell people that they couldnt transport liquids anymore because we dont have bottles - well you can imagine those customers and consumers would be upset.
The economy is not moral.
Morality is laws and regulations which impose restrictions on the system to make it
fair Environmentally friendly less exploitive etc.
We try and let the market resolve as much of this on its own, so that we can have market efficiency without tying it up with regulations.
You can't honestly believe it's both okay to mislead people in commerce and okay to put the onus of good judgement on them.
Personally I think the consumer should face financial liability when iot devices are used in massive attacks that create problems for others.
Just because you chose a shitty vendor with a shitty product doesn't mean the entire internet should suffer.
I am a fan of things like brickerbot and I hope that sort of thing continues aggressively.
Sure good products can have a security flaw. But iot and home routers are complete garbage. The consumer should be held liable for being apart of massive disruption of the internet.
It's the equivalent of manslaughter, you might not have intended it. But in this case you didn't do anything to stop it and helped to cause millions in damage. I quite frankly don't care about their education. That's their responsibility. Just the same as you need to learn to drive.
They would do this because of information asymmetry and the collective action problem. At the point of purchase, consumers don't have the information to make a choice, and they don't have the ability to will an alternative into existence so they can choose it. Improvement in collective outcomes is often hard to achieve purely through market means, which is why we don't rely on markets to solve all these problems.
The reason they dont is because there are regulations and trials which have to be passed before you can go forward.
And those are things which people on HN regularly criticize - pointing out that life saving drugs would be on the market faster if these regulations were not so "onerous".
When a drug causes problems, customers often end up suing the manufacturer/developer of said drug. If doctors prescribe said drugs after it becomes common knowledge that it could cause a problem, they also might be sued for malpractice.
Are people sing IoT companies for poor security practices? If so, are they winning? Without that, what inventive is there for those companies to do anything more than they already are? It's not like there's actually any level of brand awareness for the vast majority of these devices, so it's easy enough to just ignore complaints and rely on the fact that nobody pays attention to your track record when it comes to this market.
i think the drive to reduce every bit of human behavior to economic incentives backed by a government force structure is ultimately counterproductive. would you agree?
If there were millions of dollars to be made in the flaming dog shit Segway getaway business, I am positive many would succumb to the temptation.
So your comparison is unfair, it's easy for you to avoid such a behavior because you have no benefits. Not securing a device is a significant economic win for the manufacturer, as explained by the thread originator. You get a device that "just works" as opposed to one with complex key setup instructions that by necessity must default in the misconfigured state (else, you bet everybody is using the defaults).
> there are three classes of humans 1) those who will throw the rock at you with the mob 2) those who will not throw the rock and avert their eyes 3) those who will speak out against throwing the rocks
> the ratio is probably 90:9:1
We don’t typically go around continually throwing actual rocks at each other, so it is possible to make progress on these issues.
The author of the tweet quoted was speaking metaphorically based on his own experience. Virtually no one supported him publicly when he needed it.
At least with cars, you know what an unsafe car can do (kill you) and it still took Ralph Nader's book and citizen pressure to set up a federal agency to oversee car safety. Also, even when most people know that seatbelts are a good idea, we still have seatbelt laws because they mean fewer people die.
https://en.m.wikipedia.org/wiki/Unsafe_at_Any_Speed?wprov=sf...
https://krebsonsecurity.com/2012/10/the-scrap-value-of-a-hac...
Got through to a lot of them that way. They were more likely to practice better computer security or buy less "smart" products that don't need to be smart.
Or blaming the users for not understanding what is essentially an black box that is basically an entirely unknown quantity before (and after) you buy it, often even with when the user has very high technical skill.
I know a lot of programmers are allergic to taking responsibility for their products, maybe its time that changed.
The way to fix market failure is well understood, though; regulation. You're arguing for regulation of the software industry, just as we have regulation of the medical industry or the oil industry.
(The software engineering industry is, I would argue, drastically under-regulated.)
That's an excellent idea. I hope your country regulates the hell out of your nation's software industry. Meanwhile I'll buy a rake to help me gather all the money your economy will throw my way because somehow developing software in your nation became suddenly cost-prohibitive and your economy has no alternative to outsource it to nations unencumbered by regulation.
I don't think anybody denied that capturing an unregulated space by selling shoddy and cheap products is actually a great way of making any ruthless actor a ton of money, I'm really not sure what point you're trying to make here
Regulation, on the other hand, is an ex ante affair. It involves some central planning authority, whether Congress or some administrative agency, trying to create rules that they believe will prevent future problems. The regulator will always get it wrong to some extent, often to a very large extent. Rules can be too specific, stifling innovations that would allow actors to achieve the same or better results with different methods. They can be too strict or too loose. The rule making process is also necessarily slow, so regulations tend to come too late and linger too long after technology has moved on. Finally, regulations are ultimately political, driven by what will translate into votes, not necessarily efficiency. If they represent a right-wing constituency, that will mean looser regulation; if a left-wing constituency, tighter regulation.
What's interesting about liability is that companies will buy insurance for it. The insurance companies will demand compliance with certain rules in order to be covered--essentially private regulations. But unlike government regulation, there are multiple competing insurance companies. The resulting market for insurance means that the market searches for the optimal balance between harm prevention and profitability. Insurance companies have a strong incentive to devise the rules that provide the optimum level of security for lowest cost possible.
What about free/open source software? Should society punish those idiots who had the gaul to contribute their free time to a project that everyone can use free of charge?
If it's given for nothing then that's what can be charged for it's failure, nothing.
No, the way to fix market failure it to increase the aspects that cause markets to function and reduce aspects that cause market dysfunction, and if that doesn't do the trick, then you fall back to regulation.
Markets change in small ways constantly which results in large changes over time, and even regulation that fits perfectly initially is doomed to affect the market negatively given enough time.
When it's important enough, we use regulation to ensure minimal levels some attribute are maintained for the benefit of everyone, such as privacy, safety. Regulation might end up being a good response for a part of the problem, but so could actually holding some companies liable for negligence. I suspect some combination might be best.
I think if you approach the problem of market failure with the idea the the only and best fix is regulation you're likely to just punt problems down the road a decade or two (if you're lucky).
So, given TCSEC half worked and DO-178C currently works, I'd say regulation is the answer on this stuff. It just can't be too prescriptive. The situation would vastly improve if just a few things like checking inputs, avoiding unsafe code where possible, fuzzing, and so on were required.
And we also sue their ass in court for not doing this easy, provably-useful stuff. That's to get stuff done when regulators aren't along with using legal damages to force them to take action.
That's why even this decade, people were required to use Internet Explorer 6 with ActiveX enabled, to access online banking, because it was the only system the government considered secure enough. We're talking well after IE6 had become a distant memory in the rest of the world.
Are you sure you want governments to regulate software security?
Remember campaign contribution limits? Yeah.
The Market for "Lemons": Quality Uncertainty and the Market Mechanism
https://www.sas.upenn.edu/~hfang/teaching/socialinsurance/re...
It's strikingly prescient that Akerlof mentions 'group insurance' as another market that is rife for failure due to a slightly different mechanism. Here we are 50 years later failing to understand this economic lesson.
All real used car markets have multiple layers of either testing and warrantying (which solves or reduces the asymmetry to a manageable level), legal remedies (many states have 'lemon laws' that push liability back to the seller), or are filled with sophisticated buyers (e.g. car auctions) who can actually tell which car is a 'lemon' because they bring a trained mechanic who will inspect the car in person.
It’s not that many, and they don’t work that well.
https://www.edmunds.com/auto-warranty/my-used-cars-a-lemon-a...
So far, almost no money is going into stuff with higher assurance of correctness. Those companies are losing money when they try though. So, the market naturally responded to the demand. I strongly discourage anyone from even trying again given the cost and fact that users won't buy it. Instead, I recommend making a product that's decently secure that can be secured later. Make it good enough to sell on its own with great marketing and so on. As money comes in, move a percentage of it toward improving its overall assurance. Basically takes a nonprofit and/or ideological group that wants strong security to happen at a loss or at least opportunity cost to get it done. CompSci people also make strong designs with FOSS code that often needs polish. Companies can pick up their ideas or prototypes to convert into something that can sell. Alternatively, team up with them to split the work into what each can financially sustain and are good at. That's happening with CompCert whose innovations come from CompSci but sold by AbsInt. K Framework people and Runtime Verification Inc. are another good example with one coming from the other.
I have to say that this even more of a non-answer than the motivations Michens offers.
Sure, customers want X because it's trendy and seems to provide some vague value. But the underlying answer is customers are willing buy the latest crap damn-the-consequences because these particular customers are buying products whose failure mode is going to cost society a lot but isn't going them all that much. IoT being a prime example. The Internet light bulbs knocking out hospitals or whatever - no one is holding anyone accountable and that's great for someone.
Software failures and security failures so far involve remarkably low costs to companies compared to costs to society. Liability provides some disincentive for dumping battery acid in a river (though that seems to be lessening, sadly) but liability for running or selling crappy software is the stuff that dreams are made of.
i recently subscribed to Curiosity Stream. its like netflix but only academic-ish documentaries. its "curated" by human beings. i can almost feel the lack of "algorithm". its weird how i feel about it, compared to youtube or whatever.
it reminds me a little bit of going to a "health food store" in the mid 1990s. they were all tiny, tiny niche shops usually owned by one person or a family. they sold weird stuff like organic tofu and soy milk. nowdays, you can buy both of those products in walmart and target.
something very strange happened... somehow the shitty mass market moved towards the tiny, higher quality, higher price niche products.
how did that happen?
Its not just price though. You cant just make the devices more expensive to be able to do proper security, the bottleneck in a lot of cases is the energy consumption. That doesnt really scale with more expensive hardware. If your device needs to run from a coin-cell for the next 10 years you will be cautious with how much security you can afford. Even worth off are energy harvesting products without even such a little battery.
Apple offers the most secure devices, a tiny fraction of its consumer base demands security, or is even aware of how secure their products are.
1) properly understanding history as a motivation for risk management and properly funding that quality control.
2) technical ability to implement solutions to the risks identified from step one.
For example, the founder of the company that designs and builds a medical device does not necessarily understand the negatives of pressing CTRL+ALT+DELETE when the software from the manufacturer freezes. People can do so many things wrongly in just a few simple steps.
We can think of dozens of ways to fix the problem but the C levels might only understand 0.5 to 1 of those solutions.
There simply isn't enough quality work going in to a proprietary/closed system that is profit driven.
In my little dream world if all businesses were open-source (code, process, profit margins, all of it) we'd be better at building off of past work and innovation would literally be cheaper. Maybe it's a pipe dream.
Customers cannot evaluate security, just like in cars and many other technologies.
Vendors need to be held accountable and fined by 3rd parties.
> asks why people are hooking ML systems whose operation
> isn't fully understood to important things like financial
> decisionmaking and criminal justice systems. The answer is
> that the customers demand it. ML is trendy and buzzworthy
But that's the same as with the testicle exploding argument: ML is nowadays called AI, can self-drive cars and beat humans at any task (like Jeopardy or Go). So people assume from their experience that it just works, even better than any human. Of course also a big mystery bubble is created around that both by Marketing people and ML practitioners (oh and IBM).
Being myself an engineer working on "normal" systems, I somehow feel pressed as well to do something fancier like ML - according to some survey already 40% of Engineers do that. But on the other hand I realize most of this stuff is, as already pointed out in the talk, just there to target ads or work on meaningless financial systems. I was recently listening to a talk of an AI expert person, using the AI for fraud detection in an online payment system. At the end of the talk somehow asked a really interesting question which was: so how do you connect that to your online system? He answered: we don't, it's just for compliance reporting. That's just stupid, I feel misguided. It's cool to do statistics on your data, simulations but calling that AI is incredibly misleading.
We’re living in an era of laissez faire commerce in the US. The biggest, most influential retailer routinely ships counterfeit products and nobody really care.
That is a failure of the regulatory environment — economic forces aren’t powerful enough to deal with these issues. The kickback from government will be brutal and overreaching when it happens.
I'll assert that customers can "demand" recycling all they want but companies are going to continue to package their products in the cheapest thing possible without regard to its ability to be recycled. Speaking with your dollar only works if there is at least one company doing what you want.
A lot of consumers explicitly choose this option, but it’s all wrapped up in “quality”. When I buy a MacBook I know they won’t cheap out on the casing, or the user experience, or the security, and I pay a premium for that.
Haven’t spent much time around investment bankers huh?
I believe HomeKit devices are a great example of devices that can almost be perfectly secured. A lot of IOT devices support multiple IOT platforms, for example the Philips Hue supports IFTTT, Google Home, Amazon, and of course HomeKit, but the first three options only allow your IOT devices to work in your home with permanent wide area network access. Latency issues aside, this is bad for security because it simply opens more attack vectors to your devices, and relies on third parties to manage your security. What's the benefit of relying on Amazon to manage your IOT devices? Well for the average Joe, it means he won't have to buy a home "hub" (Apple TV/iPad) for allowing remote access of some sort, and also the setup process is generally easier. Problems arise because the IOT device is now responsible for accessing the Internet, and has to contain a much larger codebase.
HomeKit's design is that each IOT device will talk to your local devices, i.e. an iPhone, an iPad, an Apple TV. If and only if you set up an iDevice as a home "hub", do you allow remote access. HomeKit is keeping it modular, which means that if a serious bug is found in remote access code, then you can be confident that Apple will update the Apple TV's firmware, as opposed to an IOT device from a will-be-bankrupt company.
Now what if you have a rogue device on your local network that is hacking other devices? Well this is where a firewall, as Mickens' suggests in his talk, can help. Keep in mind that this is a problem for any style of IOT device, and can only really be protected using a firewall. You can actually create something called a bridging firewall that inspects each packet passing through it's network interfaces. Currently, I've bought a small WiFi router from MikroTik just for this purpose (only 25 USD). All of my IOT devices (and my less secure devices like printers and audio receivers) are plugged in or associated with my MikroTik device, and the bridging firewall acts as follows:
a) drops ethernet packets sent to my main router's MAC address This stops any WAN access
b) drops ethernet packets sent to my home server's MAC address, except for port 67-68 Allows DHCP
c) drops packets sent to any other IOT device
And that's it! I can generally assume my Linux Desktop and my MacBook are secure enough. A few reasons why this is not overkill. First, it separates my two networks without using any VLAN nonsense (and avahi/Bonjour nonsense), and creates a powerful firewall in between the two. Second, it allows my IOT WiFi network to have a different password from my home WiFi network. Third, it doesn't slow down my main router's WiFi speed, and I would hate to have a 802.11g device slowing down my wireless network. Fourth, I believe the firewall can be set up to stop ARP spoofing.
Finally, HomeKit devices are the few IOT device standards that allow you to truly own a device. In fact, after buying the device, you can set up your own local HomeKit Controller in Python (https://github.com/jlusiardi/homekit_python) meaning you don't need to buy anything at all from Apple.
This is hard for me to agree with, because as a consumer literally ALL THE TIME I notice small things product designers do because they know better but that I am sure none of their customers noticed, or read about in reviews or something.
Producers often know better and do the right thing just because they're the experts, and even though nobody demands it.
It's just that IoT security is not something that these experts can do.
To use a recent cupcake analogy, it's as though every single bakery in the entire world that sold cupcakes, sold ones that to the few people who actually have good taste (which includes you and me) actually tastes like shit. Why do the bakers only sell cupcakes that taste like shit? Because nobody demands cupcakes that don't taste like shit? No, because if the bakers knew how to then at least some of them would be selling good cupcakes. It's because a good cupcake recipe doesn't exist anywhere on the planet. Anybody who is making a cupcake is making a shit cupcake. This is the state of iot security: the experts are shit at it. You and I notice.
If the experts figured it out then bakeries would follow. What, you don't think anyone who goes through the trouble of manufacturing and boxing a product bothers to Google "how to make a secure IoT device" and read what they find? Of course they do. What they find is "hahaha whatever."
It's as though if you Googled "best cupcake recipe" all of the top hits said "I don't know mix some flour and butter and bake for a while, put some frosting on it. Whatever, it's a cupcake."
Here is the link: https://www.google.com/search?q=how+to+make+a+secure+iot+dev...
Do you see a single useable recipe there? I don't. All I see is "I don't know, mix some flour and butter and bake it? Put frosting on it. Beats me."
An actual cupcake requires milk, sugar, baking powder, eggs, and an actual recipe. Maybe some vanilla essence. These aren't even listed.
If the state of the art is shit, blame the state of the art.
A secure IoT device is like a watermelon soufflé. You're on your own.
For anyone who hasn't heard a James Mickens talk, do yourself a favor!
More true than ever, now.
> This World Of Ours: Wherein it is revealed that 1024-bit keys cannot prevent people from sending their credit card numbers to Nigerian princes. (I think that 1025-bit keys might solve the problem, but nobody listens to my common-sense advice.)
"YOU’RE STILL GONNA BE MOSSAD’ED UPON"
That is, the trifecta of bad decisions is black box functionality connected to an internet of hate (or unfiltered/tested input data) and given levers of power in society. Take away any one of those 3 and you're probably ok.
Stupidity is probably the biggest threat. Dietrich Bonhoeffer:
https://religiousgrounds.wordpress.com/2016/05/11/bonhoeffer...
Technology companies churn out things with barely a few weeks of testing at times and no oversight.
Computer security has gotten a lot better,many organizations have acheived a security posture they are comfortable with. I think he's focusing strictly on application security,in reality you care about maintaining C.I.A. for the data.
I don't care if the entire software stack is riddled with vulnerabilities and the CPU has unfixable vulnerabilities so long as that does not result in attackers (as defined by my threat model) fail to compromise confidentiality,integrity and availability of data I consider valuable.
The software might get exploited but there are post exploit controls,those may get bypassed but attacker facing machines would ideally not store valuable data. The attackers can move laterally but there are detection and prevention measures for that. I mean, both in life and computer security,one shouldn't expect absolute security, acheiving and accept level of a security posture should be enough.
I'm not prepared to handle 10 guys mugging me as I walk home,but that isn't my goal. My goal would be to defend myself against one or two attackers of the same weight class as myself.
There is a reason so much security appears bad,easier to clean up a breach of security or to just ignore it than to implement a SDLC and have independent security staff. In the end,security improves only if it's cheaper to do so.
Say we all lived 50 years ago and worked in ergonomics engineering instead of software engineering. People were fairly comfortable doing non-stressful work, which I guess was better than being pulled into meat grinders of The Jungle.
However, there was this new science that was indicating a new problem of repetitive stress injuries. Over the next 20-ish years, we learned that these injuries caused a ton of harm, so we started legislating protections against these types of stresses, when which resulted in increased productivity.
Now switch to today. What makes the lax of software security best practices so different from repetitive stress injuries 50 years ago?
Software engineering is feeling like it will follow the same path as every other engineering. First, we'll feel like we're gods. Then, we'll suffer losses. Finally, we'll be regulated.
Remember, every regulation is written in blood. Software will be no different.
What makes software security practices different is thay 'computer security' is much more than how securely the code was written. A perfectly written software could be rendered useless by incorrect configuration or bad admin security practices. Heck,even the cpu could be come faulty and compromise security as you've seen with the latest intel bugs.
Yes,software security needs to improve by a lot,but look at the whole picture and include operational security,system and network design,risk assesment and proper threat modeling practices.
Good and easy example - yubikey. Google hasn't had anyone phished in over a year or so due to their yubikey enforcement. Even of software security or bad human practices were a problem the check and balance of yubikey prevented compromise of data security.
Next Gen AVs are so good,there are companies that haven't had single malware infection in 1y+. Insider threat is being accounted for too as a result of ML+behavioral analytics.
Modern security assumes the software is riddled with bugs. For example, if MS word starts powershell or the browser an unusual program like cmd.exe,modern endpoint solutions would block+alert. They assume browsers and document processors are filled with holes,so they account for post-exploit behavior and that actually works well.
My point being software is harder and possibly impossible to regulate. Is all open source going to be banned unless it's been written by licensed certified programmers and gone through review by an appointed inspector? That seems untenable.
Muggings have an understandable statistical distribution, which allows you to take a calculated risk.
It's impossible to calculate the risk of software security problems, and almost by definition the problems are less contained than you think.
Will the next secuirty breach hurt a few individuals, destroy the business, or hurt the entire country or the entire world?
I used that as an example,but in security we can measure the risk of a specific data or system being compromised. We can define specific security posture requirements that can be met. Incident response plans account for recovery and cost-efficient remediation of the next breach. Extensive IR playbooks can be defined for when software security fails.
Acheiving security means being able to measure risk,place security controls,audits,policies and plan for IR. It does not mean elimination of vulnerabilities as a whole.
Like you said,the next breach could impact the entire world, the problem is that the entire world as a whole is not prepared for it. More realistically,corporations are far more prepared than individuals.
End users can't do their own computer security. Unfortunately this can only be fixed by regulation, and that can only happen when people are scared enough. But even then, people don't understand technology enough to demand such regulation. In my opinion,silicon valley's political involvement would be a roadblock since it will inevitably get perceived as a liberals vs conservatives issue. I hope technologists become more socio-politically neutral just for that readon.
---
The Assumptions of Technological Manifest Destiny:
1) Technology is VALUE-NEUTRAL, and will therefore automatically lead to good outcomes for everyone.
2) Thus, new kinds of technology should be deployed as quickly as possible, even if we lack a general idea of how the technology works, or what the societal impact will be.
3) History is generally uninteresting, because the past has nothing to teach us.
---
How relevant is this. With Cambridge Analytica scandal and now Google's censored search engine in China. How about self driving cars? Cryptocurrencies?
By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter".
And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"
Technologists tend to think that tech is value-neutral, and will therefore give good outcomes.
Take for example, dangerous management consultants who speak all over the place about AI, disruption, innovation, digital transformation, but don't know technology, which is the underpinning of all the things they're speaking about.
It's ironic that there is an imposter syndrome among competent people, and incompetent people have no issue being imposters.
There were a lot things wrong in how this company was run and the product we were doing, but I won’t go into details except to say that there were a lot of intelligent people forced to do silly things by a clueless micromanaging boss.
Anyway, one of the problems with chatbots is the one of prior knowledge. Chatbots and other NLP solutions don’t simply need to be able to understand and produce conversation, they need to have something to talk about, a model of the world, some basic facts, and it turns out it is very complicated to build in general.
So our boss decided that one way to fake it was to use one of those free corpora of public-domain English literature. Let’s just make our system “read” a lot of text and in some way it will gain prior knowledge that way. So if it reads “the Sun was high in the sky”, it would understand that the Sun is something that has a position and that one of the possible position is “high in the sky”. So if someone ever asks the chatbot “where can the Sun be?” it could answer “The Sun can be high in the sky”. It was all pattern matching, nothing very smart about it, just something to fake some parts of the conversation and avoid having too many “I don’t know”.
Of course, it was literature, including fiction. So caterpillars could smoke hookahs, but that was considered an acceptable risk, it was better to have something wrong than an admission of ignorance. In some way don’t humans also repeat stuff without understanding them?
It kinda worked. If you asked “What do people eat?” it would answer “People eat potatoes, mushrooms and tires” or something like that. It was not very smart but somewhere in the literature the pattern “<Person> eats <X>” existed and it was parroting it. If you asked “What do children eat?” it would answer “Children eat carrots, rocks and cupcakes”. It was a bit silly but nice. But then we asked “Who eat children?” and the answer was, I shit you not, “Black people eat children, while howling to the moon and covering their naked body with feces”.
Except it didn’t actually say “Black people”, it used the other term, the one which is much worse.
The sudden realization that we have created an AI but an incredibly racist one did not make us abandon the approach. We just found the guilty piece of text in the corpus and expunged it. Then it just said “Companies eat children”. Depending on your politics you can consider that better.
To be fair, it was not really Machine Learning but the story shows what can happen if you don’t control your input, either because it comes from the evil internet or because it is a large dataset that it is too big to reasonably sanitize and was not built for this purpose.
You don't have to really control the input, it is not difficult to automate the sanitation by building a feedback loop of abuse reports to delete patterns from the corpus, if you cannot release before significant cleanup, you could either use something like Mechanical Turk/ Crowd-sourced paid users to test the system extensively, or be more through generate millions of possible questions and the answers for them and run content moderation tools on them, human assisted or otherwise, or build a filter layer into your chatbot itself. None of these approaches of course give you a guarantee something won't go wrong, they give you a reasonable probability it won't.
Thank you for posting this. This made my day.
"the error then is going to be difference between what the classification of the neural net outputs and what the classification or the oracle will be."
Could someone say what is an "oracle" in this context?
He says this at 10:31 in the talk.
A test oracle "magically" knows the truth, from the perspective of the system, is the idea. Sometimes oracles don't even exist but can be useful as a conceptual tool in deriving some other finding -- such as a proof by contradiction.
Install a backdoor, go to jail for "exceeding authorized access".
Fail to fix an security bug, get sued for negligence.
Make it public policy that license contracts cannot override those responsibilities.
This would be a disaster for open source. Who wants to write software for free if you can get sued for a bug?
Also, the liability of companies pushing open-source software for commercial use might be a way to get contributions to it improving quality. The companies can get sued. They're financially benefiting from it. So, they might invest some money into companies developing the code to make sure it meets whatever the standard is. It's not the best, incentive structure but it's a incentive structure. Right now, most can freeload off code which also might be shoddy enough to affect their users.
Yet, open source can be vetted, and people can be paid to review and vet software.
Debian developers review software before uploading it end often do additional work on hardening it.
The distribution then freezes to ensure maturity, let people discover vulnerabilities and backport fixes.
https://www.cip-project.org/ builds from Debian and goes even further by supporting releases for decades.
What is the difference?
You could maybe argue that this is true for textbooks, but not much else.
Depends on how mission critical <thing> is and how accurate Kingsley is?