Palantir Goes to the Frankfurt School
boundary2.org
boundary2.org
"Big data analytics could be said to constitute an authenticity jargon in this sense: although they treat the data set under analysis as having something like an unconscious, they eliminate the temporal gaps and spaces of ambiguity that drive psychoanalytic interpretation. In place of interpretation, data analytics substitutes correlations that it treats simply as given. To a machine learning algorithm that has been trained on data sets that include zip codes and rates of defaulting on mortgage payments, for instance, it does not matter why mortgagees in a given zip code may have been more likely to default in the past. Nor will the algorithm that recommends rejecting a loan application necessarily explain that the zip code was the deciding factor. Like the existentialist’s illusion of immediate experience these procedures generate an aura of incontestable self-evidence.
As in Adorno, here, the loss of particular contexts can serve to conceal, and thus perpetuate, domination. Algorithms take the histories of oppression embedded in training data and project them into the future, via predictions that powerful institutions then act on. If the identities constituted in this way are false, the reifications they generate do real work, and can cause real harm. And yet, to read these figures historically is to recognize that they need not come true. This is not an interpretive path that Karp pursues. But for those of us concerned about the relationship between digital technologies and justice, this repressed insight of his dissertation is the most critical to follow."
The official answer to this problem of ML bias is "fairness" research. But really this is yet another cover for the unquestionable authority of the algorithm and the data, just this time with the added benefit of an overlay of the institutions intent. What is the alternative once we stop trusting Google and Facebook to fairly manipulate the outcome? What when we reject the authority of the data and the algorithms altogether?
But at the more mundane level, these limitations simply reflect the practical limit of data collection, and also the limit of the ability of the entity that wants to act based on the data collected.
To go to the example of mortgage default, it is not as though an inquiry (beyond particular cases) into the causes behind the correlation behind zip code and default is the best use of resources for the most altruistic organization. It is costly to investigate, and it's possible that the altruistic organization has no means to act on it in any significant manner in the first place.
And the analysis does not mention how this data is already emancipatory, which is also important when considering alternatives or replacement. Absent aggregated statistics, it would be more difficult for everyone to get a loan. These potentially false reifications are already far less false than previous methods of "my friend trusts this guy". It would have been more convincing if an alternative method was proposed, however unfeasibly it would scale, that would do the loan applicant more justice.
We (the USA) are supposed to be a "nation of laws and not of men." An algorithm is merely an artifact of human engineering, and so allowing algorithms to take the place of laws and due process is simply a roundabout way of putting the engineers of said algorithm into a position of arbitrary power. This is particularly true if the algorithm is a secret, can be modified without an audit trail (or can self-modify without auditability), or is inscrutable. Anything based on "deep learning" and the like is usually at least two of these three things.
I don't regularly check Hacker News, so saw it through the "HN Tooter" account on Mastodon which forwards links to HN front page articles. Then I was curious to see what the HN reaction was and went to scroll from the front page and on the "next page" links... and I couldn't find it anywhere! (even scrolling past the release date of the article)
I had to use the search feature and sort by date, then it showed up. Was this post removed from the front page archives or am I missing something?
Most of the "leftist cannon" still believes in psychoanalysis or it's increasingly absurd derivatives (schizoanalysis or analytic psychoanalysis being good examples).
Not sure what to do about disproven bullshit being peddled as facts by parts of academia. I shouldn't have to argue with a PH.D in critical terror studies about the existence of the Oedipus complex...
Paul Graham had a recent tweet on training an AI to detect authoritarianism. It seems such a "KarpBot" would be interesting.