The Code We Can't Control
slate.com
slate.com
http://arxiv.org/pdf/1305.5506v1.pdf
http://ftp.cs.ucla.edu/pub/stat_ser/r402.pdf
The do calculus attempts to describe causality as the correlations which remain present when you change the underlying composition of the sample set. In a certain sense, that's equivalent to saying it's the correlations which are not caused by Simpson's paradox.
https://en.wikipedia.org/wiki/Simpson%27s_paradox
So an interesting (but more difficult) question is whether the effects described here are causal or merely correlative. Personally I consider a causal link to be problematic, while a correlative one is completely innocuous. But regardless, it's important to distinguish the two.
The problem here is one of agency. Even true causality in humans is subject to change -- these are not laws of nature here but snapshots of human society, which changes all the time through political action -- but computers inadvertently perpetuate the existing condition, because they have no agency or interest in pursuing change. Perhaps this should be programmed: we could program computers for, say, affirmative action so that changing the status quo is one of their goals. But whether we do or not, the important thing to understand here is that once computers make decisions that influence humans, they are carrying out some political agenda (even if that is maintaining the status quo), whether we consciously program them or not.
But if it merely programmed to show ads for "Think Like a Man Too" to people who watched "Think Like a Man" based on a rule about sequels, there is no agenda there beyond profit.
In both cases they are disproportionately targeting black people, but in the latter case it's only a composition effect. If you run the sequel targeting program on a group of asians who enjoyed "Think Like a Man", you'll get the same outcome. If you run the AA program on a group of asians, you won't. AA is a causal relationship on race, whereas the sequel rule is not.
From previous interactions with you, I gather that you want statistical equality rather than equal treatment. That's great, but many people only care about equality of treatment. The do-calculus (or similar tools) helps with the latter problem, not the former.
But that's not the dynamics. It's like, there's this whole universe out there with its own forces and dynamics, but you're saying -- nah, I don't care about that, I just care about the current snapshot. Suppose for a second that every school taught blacks to like that movie you referred to. You just created a (causal!!) proxy for race. So now if decisions pertaining to your future income are based on you having watched the movie, then all of a sudden they become pure of ideology? First, we create a proxy, then we discriminate based on the proxy, and finally we claim no political agenda because we're "only" discriminating on that proxy.
> but in the latter case it's only a composition effect.
Why does that make a difference? Why do you think targeting correlations is somehow less political than targeting race directly? Your direct intentions are not what matters here. What matters here is what future your actions serve. Any directed (i.e. non-noise) changes (or lack thereof) to human society caused by humans are politics[1].
> I gather that you want statistical equality rather than equal treatment
Quite the opposite. If there's a society with built in mistreatment of certain races, then that mistreatment must first be addressed. You see equal treatment only because you don't try to learn how society works. Politics changes society all the time. What society looks like now is a result of yesterday's politics, and what it will look like tomorrow is a result of today's. You're simply suggesting implementing conservative politics that don't try to change society today. But there's absolutely no objective advantage to the current situation over a previous or a future one. That's what's known as the naturist, or is-ought, fallacy.
What I find quite interesting is that as a a scientist, you take absolutely no interest in learning how things work and how we can change them. You want to hack computers but not society. How is that not a political agenda?
P.S.
I tried finding a good starting point to learning the topic (of politics), or at least understanding what issues researchers are looking into and what they've found. I think a good introduction is: http://en.wikipedia.org/wiki/Power_(social_and_political)
[1]: http://en.wikipedia.org/wiki/Politics and in particular http://en.wikipedia.org/wiki/Politics#Politics_as_an_academi...
If there's a society with mistreatment of certain races built in, then that mistreatment must first be addressed.
It's a bit unreasonable to ask adwords engineers to layer on some post-hoc fix many miles downstream when they can't even measure the mistreatment or reasonably predict the effect of the fix. It's also completely unfair to the individual humans (individual humans are the only ones I care about) who didn't mistreat anyone and are now obligated to pay for the actions of someone else (in the form of fewer clicks on ads).
And of course, built into all of this is the assumption that statistical disparities are all caused by mistreatment. That is unlikely to be true.
Incidentally, as per the description given in your edit, unintentionded consequences of ad serving algorithms can only be interpreted as "political" in the broadest sense. So can anything else. So if you just mean "political" in the sense of "anything that ever affects anything", then sure - ad serving is political.
> It's a bit unreasonable to ask adwords engineers to layer on some post-hoc fix...
Maybe so. But this article notes the interesting fact that failing to do so is a political action. We can leave the discussion of what we should do to another time. Right now it's just important to understand what it is that we are doing.
> also completely unfair to the individual humans
Again, that is a question of policy and values. You may find it unfair and I may think the opposite is unfair; that's a matter of political orientation. But this article is not about what's fair and unfair, but an objective observation of the role computers play in the dynamics of society.
> And of course, built into all of this is the assumption that statistical disparities are all caused by mistreatment.
Not quite. What's built into this is the fact that human actions -- intentional or inadvertent -- affect the distribution of power in society. Whether or not there's mistreatment is a specific question pertaining to the specific histories of specific countries and societies.
> That is unlikely to be true.
Well, this is a question of fact, and research has conclusively shown a direct link from slavery and Jim Crow (that lasted for 250 + 100 = 350 years more or less, and ended, as official legal policy, only 50 years -- one generation -- ago) to the current distribution of power among races in American society.
EDIT: people who hold liberal values do make one relevant assumption: that it is very unlikely that certain groups would voluntarily (i.e. free of any influence and persuasion) yield power (over them) to others. Theoretically, this assumption could be wrong, but so far we have no reason to believe so. Also, this seems like a sensible assumption, and the burden of proof should be on those who think it's false.
> "political" in the sense of "anything that ever affects anything", then sure - ad serving is political.
I mean political as in political[1]: human actions that affect distribution of power in society. If a volcano erupts and destroys a town -- that's not political. If however, the volcano is known to be dangerous and therefore only poor people buy houses in its vicinity, there may be politics involved. The invention of the locomotive is purely technological, but choosing which cities the railway passes through might be political. Machine learning is technological. But programming computers to take actions which may affect the distribution of power based on machine learning extracting information from past data is political.
[1]: http://en.wikipedia.org/wiki/Politics#Politics_as_an_academi...
A link. They haven't shown that every disparity is caused by it. And the rapid social movement of various other groups (e.g., vietnamese, filipinos, mexicans) suggests that historical explanations don't have great explanatory power anyway.
I retract what I said. According to your definition of "political", ad serving, dating, renting a house, playing video games, etc, are all political acts. Very few actions by humans are not political.
True. But all they needed to show -- and they have -- is that this is the largest contributing factor, which explains most of the observed effect (in fact, they didn't even need to show that, only that it's a large contributing factor; if you were born with one weak leg and I take a hammer to the other one, doesn't mean that I'm not responsible for you being confined to a wheelchair). What, AFAIK, hasn't been found is other contributing factors (that you seem to allude to), but I won't deny their theoretical existence.
> According to your definition of "political", ad serving, dating, renting a house, playing video games, etc, are all political acts.
No. First, this is not "my definition" but the real definition, which you are free to look up on Wikipedia at your leisure. Here's the link -- again -- for your convenience: http://en.wikipedia.org/wiki/Politics#Politics_as_an_academi... Second, many actions by humans can be political, if they influence the distribution of power in society. Note that this is not a binary definition: one action can be more or less political than another. Finally, why are you trolling? Are you trying to understand this science or make fun of it?
Clearly you aren't that familiar with the research then. There are many disparities in work and academic effort, labor force participation, criminal propensity, IQ and career choice.
I've not seen any analysis that controls for all of these or any remotely accurate predictive model. I guess given your voluminous command of the literature, you'll provide one?
Finally, why are you trolling? Are you trying to understand this science or make fun of it?
I was merely linking to some interesting and relevant articles on causal calculus that relate to the issues in the article. You are the one who went on a random tangent about how everything is political and I should use the most expansive possible definition for certain words, and somehow I'm denying science if I don't.
Let me get this straight: do you think that scientists have shown an innate, i.e. biological, disparity between Europeans and Africans that explains any significant portion of the effect we see in American society? Because if you do, you have either dramatically and tragically misunderstood some real research (showing non-innate disparities) -- because that never, and I repeat, never happened -- or accidentally stumbled on an article in KKK weekly (BTW, I sometimes read "scientific" research in Christian publications for fun; for example, did you know that the second law of thermodynamics forbids the spontaneous creation of life? I kid you not).
The only thing out of the list you provided that was not misinterpreted is IQ. It is well known that there is a group of real researchers (actually, just one or two people) who claim an innate intelligence disparity between races. They are very proud of their research that "uncovers" an inconvenient "truth", but the real truth is that the effect size they have reported is too small to justify further study -- let alone explain any observed disparity in the real world (when reading a statistical study, you should really look at the effect size; blacks and women are often underrepresented in jobs that provide power with an effect size of 5x or even 10x).
As to others -- showing differences, say, between boys and girls -- those papers assume you know something about power, and are far from saying what you think they're saying. In other words, other than IQ studies who show insignificant effects, those results you mentioned simply don't exist. And, as always -- when reading statistics look for the effect size.
> I guess given your voluminous command of the literature, you'll provide one?
One? How about a thousand, starting with those you yourself alluded to. Start Googling, you'll find them in seconds (I am sorry, but I will not link papers proving that blacks aren't naturally inferior to whites; and I know what you're going to say now: "inferior? I didn't say inferior -- I said different", to which I'll say go read on power in society).
If you know you misinterpreted me, why do it?
Let me start explaining this, but please read about it, because I have no intention of teaching sociology 101 here. Power can sometimes be compared to a force and sometimes to potential energy, but the best analogy, IMO, is a force field. There are different kinds of force fields (e.g. EM and gravitational) and they affect you differently based on different properties (like mass and charge). But each of us emits power and is under the influence of power all the time (and different kinds). It is possible, however, to attain more power (or lose some), often by aligning with others (you can think of it a bit like unaligned magnetic bits vs. aligned ones -- they could either cancel each other out or strengthen one another). In the end, all power -- like force -- translates to influence, i.e. moving people in a certain direction.
Now, I don't think there's a good physics analogy for this, but people who have more power, are much less influenced by others' power (maybe it's like momentum?), while those that have little power are basically heavily influenced and directed by the power surrounding them. What's more crucial is that power can prevent alignment and thus ensure that some people (often groups) don't gain more power (no physics analogy here). And what I'm saying isn't theoretical: those processes have been studied at length by psychologists, sociologists and anthropologists.
When you say that blacks and white are "different", but the differences -- all of them -- correspond with less power (power can be intelligence, it can be money it can be other kinds of things), you are basically saying that blacks are inherently less powerful on many power axes (and never more powerful on any of them).
Now, power is what defines social ranking (when someone has a higher social rank than another, that's simply other words for saying he has more power), so when you say blacks inherently lack power, you are basically saying that blacks are inherently socially inferior to whites and therefore "belong" on lower social ranking.
To this you will say, "less power in our society, but maybe not in other kinds of society", but that means that whites have built a society, that by their very nature, blacks are destined to be ranked low socially. That is, of course, what is known as a racist society, but that's not even the truth of what's happening, because all this time I assumed the studies you referred to found effects that are anywhere near those seen in society, which is not the case. The data points to a different reality -- our reality -- where the forces in our society are aligned in such a way that it's very hard for blacks to increase their power, and this has absolutely nothing to do with innate properties. Of course, this, too is a racist society.
One thing beginners often get confused about is how power forms and works, and they imagine smoky rooms and powerful executives, and -- most importantly -- something resembling a nefarious conspiracy. That's not how it works. Power is subtle and constant and often invisible (to those untrained to see it). It rarely requires a conspiracy, and powerful groups are an emergent property.
>Politics is the practice and theory of influencing other people on a global, civic or individual level.
So yes, ad serving clearly falls under that scope.
There are, of course, many sub-types of politics -- as you'd have known if you'd bothered to look it up on Wikipedia. But your intentional ignorance combined with obsessive pedantism makes it very hard to provide the full science with every HN comment. Like I've said so many times: either learn this topic or don't, but why do you insist on flaunting your ignorance with such vigor? You seem almost proud of how little you know and how little you wish to understand.
No science is necessary for anything I've said - you havent actually made any falsifiable claims. I don't know why you keep bringing it up.
When we say an action is political, we mean it is best analyzed through political sociology/psychology, i.e. by using the techniques used to study about power dynamics in society, and drawing on the knowledge learned in that field.
Usually, those categorizations aren't even contentious. No chemist is offended when we say that metabolism is a biological process, even though it is also chemical. But I think that in this case (of politics), many people -- and especially here on HN -- don't even know what the word means, let alone its academic definition.
And consider this. If you were to travel in time to, say, Mesopotamia, and tell them that fire was a chemical process, they won't know what you're talking about. In fact, they may have used the word "chemical" to mean material or something. But you know that the process is really one of reactions among molecules -- they wouldn't even know molecules exist. So too in politics. The power dynamics shaping our society are often hidden from those who aren't trained to see them.
When we say "the invention of the fork was a political action that caused..." (we don't, but the invention of the fork was just something I studied once so it came to mind) everybody understand what that means, and no one is offended, because people who know about power in society immediately grasp what we're talking about. Those who don't might either don't understand at all, or even push back -- like you have -- and say something like, "but that was purely technological", which is not too much unlike saying that lightning is a spear of a god. You need to learn -- even a little bit -- to see politics at work, and that Wikipedia article on power is an OK place to start.
Also, being able to say what people want to hear at the time that is most beneficial to you, by running the statistics on giant datasets, is one of the biggest existential risks democracy has ever faced.
I do. I'm much happier that my Facebook includes ads for New Relic and MailChimp than condoms and sports.
More generally, I don't understand how algorithms or profiles can be prejudiced. If they're inaccurate, there is a strong economic incentive to improve accuracy over time. If they're accurate, I think that should be defense alone—if your history suggests you are likely to default on a loan, companies should have every right to not loan you money.
Current process seems purely political with massive rents extraction ($100k for a degree, loan guarantees), almost naked votes buying (subsidies, "free" plan for whatever). And moral arguments are used mostly for cover (non-dischargeable debts?).
Perhaps we can sidestep those problems, or at least escape from them for a while.
But we can't. Any decision we make -- even one based purely on data -- is a political one, because we can choose to change the data by changing society. If you say we should make future decisions (which affect lives and future data) based on current correlations only, you're executing a conservative political policy of maintaining the status quo. If you program computers to just take into account existing data (and not, say, desired future outcomes for society), you're programming conservative computers.
Also, the computer programs take into account the desired outcomes and the information they can obtain experimentally (through A/B testing for example), not just existing data.
Because feature-learning problems in the setting of nearly-unlimited sets of possible features are very hard.
The other issue is that humans are much more permeable to their environment than we would like to admit ourselves. We are influenceable. Ads invade our environment constantly for that reason, their goal is to influence us into buying something. They can take us to directions that we want to avoid. What if a fat person is trying to lose weight ? Ads have measured that this individual is likely to buy cake and present him with delicious assortiments everywhere he goes. It's beneficial for the companies but what about that individual ?
The algorithms can only be accurate to who you are, not what you want to become.
Conversely, what if the algorithms have decided that the person is a good lead for a local gym, and those ads push him or her into losing weight?
We're all influencing each other, constantly. In a way, banner ads and such are some of the most honest ways of going about it, since they're clearly delineated as such.
The algorithms can only be accurate to who you are, not what you want to become.
If aforementioned fat person searches for "weight loss" and Google starts showing ads for that, haven't the algorithms been accurate to what (s)he wants to become?
Of course not every example of a targeted ad is going to be controversial (like advertising merchandise of a sports team that I like versus the one that is just popular in my area) but its problematic that we don't have any sort of machinery in place to control/modify the examples that are politically/societally important.
At least if that machinery were in place, then we could transfer the agency of decisions to the humans that run google, rather than the algorithms that are just running in the background.
Okay, maybe they are "prejudiced" if by prejudiced we mean making decisions based on statistical heuristics. I still don't see how that's a problem if they're rooted in fact.
> The algorithms can only be accurate to who you are, not what you want to become.
Actually, the best advertising is aspirational. It's easy to sell a fat guy a cake, but it's even better to sell him on the prospect of no longer being fat (ie. a gym membership).
With these laws in mind, is an algorithm that says "people who tend to purchase cosmetics are better drivers" illegal? Is an algorithm that says "people who tend to purchase cocoa-butter-based skin creams tend to have worse driving records" illegal? Should their be some sort of test put in place for these algorithms to make sure they're giving out legally-okay results, just like government officials test the gas pumps to make sure they're pumping fairly?
It really throws a wrench in the algorithm, because there are a million little unidentifiable things that are correlated one way or the other with race, gender, marital status, etc.
For example, it's a fact that on average, women drivers (at least in the UK) are less of an insurance liability than male drivers. As a result, if a man and a woman both applied for the same insurance with identical driving/insurance history, and otherwise identical profiles, the woman would get cheaper insurance. This was ruled illegal by the European Court of Justice in 2011.
Yes, The personalized ads can help you to find solutions to problems that algorthms detect of your profile/information, but what is the cost? Freedom, you can say that you have the choice, but they take your information and control your choices.
If you want more choices of ads, you just have to look for it. Nithing prevents you from CHOOSING to see more ads. There is a limited number of ads a website can show you, if it is random, you will have as much choice as if it is not, but companies will make more profit.
The first paragraph establishes that the program bases its decisions on indicators of financial responsibility. It then makes a decision based (apparently) on race alone. This would require race to be a recorded indicator of financial responsibility as decided by the programmer (or his/her employer).
Had race not been recorded at all by the computer, then it would not have made a "racist" decision (for it would be based entirely on individual merit, rather than any particular connections with others of a specific demographic), and this article would therefore be turned over onto its head and rendered entirely moot.
The software makes decisions on metrics casually correlated with race. Basically it's the codification of institutional racism. Some contemporary political trends refer to "institutional racism" as just "racism," which causes other readers/listeners to hear "prejudice/bigotry racism."
By skipping a few steps in the causality chain and playing with definitions it is possible to make eye-grabbing statements. "Computers are racially biased" sounds WAY more sinister than the (not much) longer explanation.
The eye-popping version is at best a hook to get people to care about an issue. At worst the in-group signaling about tangential issues can cause something worth broadly considering to be thoughtlessly bundled up into existing narratives and political groups. Then nobody will care about the issue itself, but instead whatever their bandwagon has pre-determined to be the opinion to have.
FWIW: I think there is a lot to be considered about unintended consequences of removing humans from decision loops. After reading the intro to that Slate article, I'm pretty sure it isn't happening there and closed the tab.
Basically, the first paragraph effectively creates a strawman; I can only assume that the rest of the article seeks to defeat that particular strawman.