How Palantir Is Taking Over New York City
gizmodo.com
gizmodo.com
What does Palantir do? “integrate[s] disparate data sets and conduct[s] rich, multifaceted analysis across the entire range of data.”
How does NYC use it? Tax fraud, fire code violations, fake security guards, fake IDs, fake cigarettes, fake marijuana.
So the data already existed in NYC databases and the crimes they're enforcing already existed.
And yet: "the potential for that kind of outright abuse is less disturbing than the ways in which Palantir’s tech is already being used. The city’s embrace of Palantir, outside of law enforcement, has quietly ushered in an era of civil surveillance so ubiquitous as to be invisible." -- total hyperbole!
If anything the most telling part of this article to me, was the small sums of money being made by Palantir which is frequently lauded as one of the most elite, selective startups for software engineering positions. It seems to operate in small change relative to all the hype.
Talking about how the NYC government is using it to invade privacy makes it out as if ANY data blending/visualization software couldn't do the same thing. We all know that it could, and that Palantir is bullshit.
By "work the data/analytics machine", I mean putting together the full spectrum of data analysis tools & people - something that anybody with enough money/time/staff can do - anything from the ETL/stream processing, integrating across different databases, big data processing systems, BI tools/analytical tools, graph processing, search tech, conventional data mining, recurrent/convolutional neural networks, etc. etc. etc.
Partially, I say this out of personal experiences, having observed how government & big business is often quite motivated to spend some money, when seeing what they perceive to be "cool" and advanced data/analytics capabilities.
That's not saying Palantir doesn't have advanced tech, I am sure they built some pretty cool proprietary stuff. But, just that maybe what they do isn't such a mystery to people who understand how it is done anyway.
Then when you buy them, a bunch of "forward deployed engineers" come in and carefully comb through all your data sources and figure out how to get it all ingested and linked in their software.
Which, of course, is actually the hard part of doing data analysis at large organizations! By default, in any large org, important data is fragmented and sitting in silos--and departments defend it that way. Being able to see all that data come together would seem like magic. But it's mostly because of the manual work upfront.
Maybe the right way to think about Palantir is a data-aggregating operation that uses marketing to convince large organizations to allow them to aggregate data.
https://www.buzzfeed.com/williamalden/inside-palantir-silico...
https://gcn.com/articles/2013/10/04/gcn-award-nyc-databridge...
https://www.accenture.com/t20150719T214424__w__/gb-en/_acnme...
Limits on acceptable data collection are more straightforward, enforceable, and fair than limits on acceptable data analysis. What would you say if a human had reached the same conclusion as Palantir by analyzing the same data?
I strongly disagree. In this electronic age, it is unreasonable to to expect companies and agencies not to store data electronically. Isolated, these data points are not illegal - the USPS knows your address, the IRS knows where you work, the DOT knows your license plate - these are all necessary for these agencies to do their jobs. Private companies know stuff too - your phone company knows about calls you make, your ISP knows about sites you visit, your bank knows about purchases you make. Some of these can be forgotten, some are necessary to do business.
The problem comes when someone cross-references innocent data to a level that results in an invasve unwarranted intrusion into your privacy.
Should we penalize Facebook for keeping our photos? No, that's what we use it for. Should we penalize shopkeeper for recording security videos of their store to help analyze thefts? No, that's within his rights. Should anyone be allowed to correlate all the images with all the security footage to have a camera-by-camera record of the motions of everyone in the city? No, that's a dystopian horror story in the making. The criminal intent is in the analysis, not the storage.
Here's one I was able to dig up: https://www.youtube.com/watch?v=5n2UjBO22EI
Why is it quiet when IBM does it, but horrific when Palantir/Big Bad Thiel does it?
It's either: a) no one cares about what IBM does anymore, because they're seen as "old school", or b) no one was paying attention before and only now caught wind of it?
Today, fake cigarettes and fire code violations. Tomorrow, aggregating the 1M+ or so surveillance cams around the city. Cross-referenced with facial recognition databases, cell tower logs, social media droppings, what have you.
Total hyperbole!
It's not hyperbole at all.
The most telling part of this article to me, was the small sums of money being made by Palantir
It's called "a foot in the door." Palantir knows that the potential of the "smart cities" market is deep and vast. So that's why its initial deals with NYC -- and what better marquee client to have? -- are priced at teaser rates.
Seriously? Is fake marijuana an actual problem?
https://en.wikipedia.org/wiki/Synthetic_cannabinoids
They typically cause more severe side-effects than traditional marijuana. There have been a few rounds of subway ads related to synthetic marijuana.
http://www.nytimes.com/2016/07/13/nyregion/k2-synthetic-mari...
The side effect profile of these compounds is worse than natural cannabis, with high incidence of psychotic symptoms.
https://en.m.wikipedia.org/wiki/Synthetic_cannabinoids
Edit: i completely overlooked rpedroso's substantially identical reply, haha...
"The planet already had Uranium-238 so Oppenheimer didn't really give us nuclear weapons."
Take a look at the top 10 US government contractors.[1] Most of the top 10 make weapons systems. But two are in information processing: Leidos (used to be SAIC), and L-3 Communications. Palantir isn't even in the top 100. Maybe they're more into state and local customers.
There's lots of potential for innovation in the state and local government space. A smartphone app for building inspectors, for example. One that involves lots of picture taking and GPS tagging. There are building inspector apps, but they're basically paper forms reworked for tablets.
An ambitious project would be a system which takes the video and audio from a cop's body cam and does most of the paperwork. Show it a driver's license or a face, and it's in the record and understood by the system. Cops hate paperwork, yet have to document much of what they do. Automate that and cops will be glad to wear a cam. Difficult and controversial, but useful.
It might be easier to sell in countries where local government is more standardized. In the US, you'd have to customize a system for every police department.
[1] https://en.wikipedia.org/wiki/Top_100_Contractors_of_the_U.S...
It is a continuous marvel that Peter Thiel, nominally an outspoken and prominent libertarian, is partially responsible for one of the most insidious powers that the U.S. government has over its people.
I say this as someone who is very sympathetic in principle to a lot of libertarian thought. In practice, that's just not how humans work.
I posit that there is a broader category, which is distinct from what people usually mean when they speak of or self-identify as libertarians, that adheres to the same principle. Basically, the idea is that government is always, by its very nature ("all power comes from the barrel of a gun"), an intrusion on some liberties - and so any extension of government requires a solid justification and thorough vetting. However, some freedoms and liberties have to be intruded upon in order to maintain others. Again, most bona fide libertarians would agree - say, the freedom to violently coerce other people is clearly not the one that you want.
But once you get into this mode of thinking, and ditch ideological stereotypes, there are many other limitations that appear perfectly reasonable. More importantly, you realize that whether some limitation is justifiable or not depends on your [inherently subjective] assessment of what is good and what isn't - but that is orthogonal to the minimal government principle. In other words, there are many different kinds of libertarians, who all agree on that basic principle, but disagree on what outcome they desire (and hence on how much government is "just enough").
So you can be a libertarian, but still consider public welfare programs to be a good way to spend money, because the alternative would be worse, in terms of overall individual liberties.
True, but all too often, I've heard otherwise reasonable people seriously argue for the repeal of the 13th Amendment (the abolishment of slavery) in the name of Freedom(tm) because, "You can't truly be free unless you can sell yourself into slavery."
Of course, we have seen this society, and even today can easily extrapolate what would be its effect due to proliferation of legal usury in the form of payday loans. But hey, we've got a Dark Enlightenment to usher in, for FREEDOM(tm).
A classic case of opportunism wrapped in the "someone had to do it" excuse.
This greatly oversells what Palantir does for the federal government.
Committing resources to quality of life improvements? Good
75% of enforcement done in neighborhoods of "color"? Yikes
CIA-backed data analysis firm Palantir Technologies? Dear god
It seems like it's kind of a trap. If you agree with the frame of the article, then you have to conclude that either 1. We passed those laws to make our neighborhoods better, but minorities and the lower classes don't deserve that, so we should ignore some set of laws there, and that totally won't become apartheid or 2. Maybe all of these nanny-state laws made by wannabe social engineers aren't such a good idea after all, since they give the Government too much arbitrary power to be enforced on whoever they don't like this week, so we should repeal them and tell people to mind their own business when they come up with this stuff.
The problem isn't the law. The problem is the selective enforcement. It's the selective suspicion. It's the selective police state.
Do you honestly believe that some cop in white suburbia is going to care about loose cigarettes? No way.
But that doesn't really matter anyways. I thought we were talking about not throwing people in jail and giving them criminal records for petty crimes. If a black guy gets in trouble for, say, selling loose cigarettes, is he supposed to feel better knowing that if a white guy in a different neighborhood did the same thing, that guy would be getting in trouble too? Is that really the best we can do here? I'd rather try and find a way to not throw him in jail in the first place.
You're just upset you got called on a false dichotomy.
Given the greater prevalence of things like stop and frisk in neighborhoods with large minority populations, is it any surprise more people are caught for things like drug possession? That's going to further skew the stats, leading to more enforcement in those neighborhoods (since they are "high crime").
Drug use is actually higher in young white populations than it is among young black populations, but because of where law enforcement spends their time the incarceration rates differ wildly.
Here is the NYC crime map: https://maps.nyc.gov/crime. Look at the maps for felony assault or rape. Stop-and-frisk isn't going to change the incidence rate of those crimes.
The general point holds - there are corrective measures that need to be taken to avoid skewing crime data as a result of increased enforcement, and in the case of some jurisdictions those measures are being taken.
What's not obvious to me is whether police are increasing the severity of the charges based on where they are, even if the charges wouldn't necessarily hold up in court. There's a case to be made that that would be an efficient tactic - public defenders will encourage plea bargains and it gives the DA more leverage to settle the case quickly and efficiently. The opposite may be true when booking a drunk banker who gets in a fistfight, or a privileged college kid who rapes his date behind a dumpster.
Also, the discussion generally is not about CompuStat, it's about "quality of life" improvements prosecuted with the use of secret databases that are not publicly available. So, you know, there's that.
What other metrics should we use? If you simply assign patrol routes based on population you are going to under-serve areas with higher crime and over-police areas that don't need it.
Of course it's complicated, and of course crime stats are a simplification. Most stats are. The point is, stats are much better than going by biased "gut feelings".
Yes there are underlying issues with stats such as underreporting, but the solution isn't to get rid of the stats. It's to improve the underlying cause of the bias in the data, such as improving police/public relations.
My solution, implied in my comment, was not to use naive statistical models or ideas (i.e. 75% of "crime").
And your now stated solution is easier said than done, and certainly not immune to the same kind of biases a naive solution may be subject to.
https://www.propublica.org/article/machine-bias-risk-assessm...
https://enterprisersproject.com/article/2016/9/beware-biases...
http://www.nytimes.com/2016/06/23/us/backlash-in-wisconsin-a...
Isn't enforcement of a law (through an arrest for example) the first indication that a crime might have occurred?
If so, how to figure out if the 75% attribution is fair or just a result of selection/confirmation bias?
I suspect there is no way for the public to know.
Source: http://fortune.com/2014/09/04/peter-thiels-contrarian-strate...
“Basically,” Thiel explains, “I thought that some of the approaches that PayPal had used to fight fraud”—which at one point posed an existential threat to PayPal—“could be extended into other contexts, like fighting terrorism.”
For contrast: Mongo Inc's Eliot Horowitz: https://medium.com/s-c-a-l-e/mongodb-co-creator-explains-why...
tinfoil hat
A LOT of hand-cranking integrations and massaging data, presumably that's where they make most of their money - billing hours to agencies with deep pockets.
Even the most bass-ackwards database designs can be reverse engineered by bright people in a day or two (we know Palantir can attract these people) especially if they have some scars dealing with dumb database design. And it's probably not likely that the person who built that database and the program that uses it went to great lengths to obfuscate anything. I'll even go a step further and say that most data sources are probably really simple programs. Throw in any kind of documentation as a bonus and the learning curve just goes down.
What the hell could they be charging that much for?
There's a rooftop patio, as many buildings in the area have, and gym subsidies as most tech companies have. I've been told they have emergency readiness kits (which are advocated by every level of government in every jurisdiction), but they're definitely not at the desks (or they are, but invisible).
How are any of these criticisms distinct from Google, which owns an entire street block (no joke street to street, Ave to Ave)? I've had more than a few meals there as well and it's pretty much the same situation as far as I can tell.
why anyone thought it was a great idea to name their company after the remote sensing device guaranteed to lie to you and make humans suicidally depressed has always been beyond me.
If anything, they are both a little too on the money.
Presumably this technology is supposed to be helping the people of NYC. Shouldn't these people know what data is being collected about them so they can decide whether or not they actually want it?
http://www.nytimes.com/2016/09/20/nyregion/cellphone-alerts-...
http://www.theatlantic.com/technology/archive/2016/04/linkny...
http://www.nyclu.org/content/automatic-license-plate-readers
When the government calls it "Open Data" (ie. https://nycopendata.socrata.com, https://data.ny.gov) it's lauded as an inspirational embracing of transparency and seen as just awesome. When the government uses it to make itself more efficient and effective, there's this implication that something nefarious is going on.
If you want to be offended by government software purchases, check out the POs for Oracle Enterprise whatever or the IBM Passport Advantage agreement that NYC has issued in the last year. Chances are, the city doesn't even know wtf half of the products they license are!
There seems to be a lot of momentum in the "canvas large data sets" space. It has always been on the wish list for the authorities (see many RFP's for the DARPA "Total Information Awareness" initiative) and the challenge has always been storage and algorithm development. Storage is becoming a non-issue when you can have a petabyte in M.2 class SSDs available across a 10Gbit network of processors. The challenge is the needle-in-haystack finding activity.