Beyond Memorization: Violating privacy via inference with LLMs
arxiv.org
arxiv.org
Everyone claims MBTI is akin to astrology, which means there should be no predictive capacity of those four letters. But just for fun, I gave GPT-4 the top 20 songs of my music playlist and asked it to guess my MBTI type, and it correctly did so. I repeated the process with two friends (asking their MBTI type prior to querying GPT) and it guessed them correctly as well. If MBTI is no different than random noise and GPT lacks inference capabilities with regard to humans, the probability of this occurring due to chance alone is less than 1 in 16^3 (*approximately—non-uniform distribution of types).
A professional can gain _some_ information from all of them. Just that the Big 5 have much more literature behind them than MBTI, so there's more reference data for any given constellation. That's a reinforcement loop. Methodical differences and reliability and all that fluff can be used to argue in more detail in favor of one versus the other, but by now, it comes down to what folks are more familiar with.
In the end, these are just a few out of many, many tools of the field. If you see somebody build their career on "(MBTI|Big 5|...) analysis" and nothing else, they're not a professional but a quack.
What's changing is the scope, scale and access. It's no longer necessary to be a savant or have a special talent - all you need are hundreds of banal data points easily scooped from the meta data wake we all leave. Not to mention it can be programmed and commoditized instead of being a special attribute of another human. The realization feels a bit like climate change in that some folks saw it coming, pointed it out, articulated the issues, underestimated how fast it would actually arrive and how profound the impact would be.
But back to the point at hand you can't legislate people not see what is publicly before them. You might criminalize recording that data for a while but that doesn't seem to work long term thought it's brutal in the short term. If you can record that data you can analyze that data. From analysis comes inferences and decisions and actions.
I think the more salient question is about equality. I have some ability to obfuscate my meta data wake but as an individual it's pretty limited. Apple, google, Amazon have much more ability to do so despite leaving a gazillion more bits of data laying around. Tools individuals can use to obscure our meta data trail will do more to maintain privacy than trying to tell someone observing me what they can and can't do with their observations.
as an individual i don't even have the capacity to be aware of all the possible implications and of what can be inferred from what i share.
i could not participate here on hacker news if i was alone responsible to protect my privacy.
we have to legislate what other people and companies do with the data they can find in public.
we can't prevent data being produced, and we can't prevent data collection tools being created, therefore we must legislate how the data is used.
i made this realization some time ago and since then i have been thinking about possible ways to address this problem. the only thing i was able to come up with is that to prevent abuse we have to make the punishment so severe that violating someones privacy is simply not worth it.
i would love to see alternatives, but so far i come up empty.
we can't expect people to stop sharing.
we can't prevent tools to collect data to be created because there are many legitimate uses for that.
we can't stop the creation of inference tools either.
so what is really left besides punishing abuse?
change our society in such a way that having and being able to infer private information is no longer anything that anyone can benefit from.
for example if we get rid of money and measuring profits then the profit motive would disappear.
if we educate everyone to be respectful and tolerant then it would no longer be possible to embarrass or blackmail people. in other words, a society where privacy is not needed.
a world where everyones needs are taken care of and that has an abundance of resources would eliminate a lot of crime.
you can see where this is going. but those are long term goals, and they are not things that are easy to do, if they can or even should be done at all. i don't think we'll ever be able to eliminate the need for privacy or prevent all crime.
so legal protection of private data and punishment of abuse will remain necessary.
>for example if we get rid of money and measuring profits then the profit motive would disappear.
Money is just one particular measure, but it's also a measure of 'something'. If I wasn't collecting money, it would be gold bars, or chunks of land, or widgets. At the end of the day people will find a means of measuring profit. Greed and jealousy will not be removed from humanity any time soon, and possibly ever.
>so legal protection of private data and punishment of abuse will remain necessary.
With that I agree.
which is why i specifically mentioned both. it's not just getting rid of money but the concept and need for profit as a whole. i am not suggesting that this should be done, or that it even can be done, but that it would be necessary if we want to remove any motive of people profiting from collecting and analyzing private data.
and therefore legal protection and punishment, severe punishment, are the only means that at this point can be practically implemented.
For example, twenty years or so ago, I think the vast, vast majority of people in the US would agree that you don't have a right to privacy walking around a public street. However, once the Internet and Streetview came along, people had very different feelings about whether someone should be able to post a video they took of you on the Internet, for anyone across the world to see for all eternity, oh and we can also use facial recognition (just from public data and photos other people have posted) to fully identify you.
Privacy is just one example. I will personally say my opinions on free speech have also changed. Given how highly targeted, systematic and relentless well-funded misinformation campaigns can be, the old assumptions I had about free speech ("Given enough daylight, the majority of people will walk away from 'bad' ideas") I no longer believe to be true. I really don't have any idea what the "right" solution is, just that my views have changed.
The primary point is that wide scale, incredibly cheap data analysis (now being supercharged with AI) basically breaks all these fundamental assumptions and ideas society has, which is one reason why the world seems to be in such turmoil.
They use LLMs to analyse social media and turn unstructured text ("I always get stuck waiting for a hook turn") into structured data (location=Melbourne, Australia).
They test this on a set of Reddit profiles that have been hand labelled.
However, the fact that this hand labeling is possible indicates in itself that the issue isn't a pure privacy violation. It's more that automating this in bulk is now possible which was difficult before.
Is this ethical? It isn't obvious to me that it's not - I think most people writing on a public site realize their writing is public, and thew fact we see common warnings about not posting personal information shows that is true.
It's not clear it violates privacy expectations either - even if some people don't like it.
Too bad it's probably going to be done to me anyway.
A company like Google builds a profile[1] from my search queries. Many other AdTech companies do similar things.
> Would you want it done to you? I sure wouldn't.
I don't mind this - if I have to see ads I prefer ones relevant to me over ones that are completely irrelevant.
Gay men get this ad: Only Politician Foo protects gay security in the streets at night!
Homophobes get this ad: Then you must vote for Politician Foo, only he opposses gay marriage!
Pro-lifers get this ad: Vote for Foo to protect the children from the very first moment!
Pro-abortion get this ad: Foo protects women's rights!
Advertising should be as general as possible - microtargeting people in a democracy is extraordinarily dangerous.
I suspect any ads on grinder are different from ads on tinder as well.
/s
I don’t understand how that follows.
Your examples of microtargeting seem helpful since they speak to how politician Foo appeals in ways relevant to a particular segment. Instead of unknown-likelihood inference of how the politician would operate because of some associations, a voter would be able to make an informed choice based on what is significant to them.
The alternative to some level of targeting or relevancing is broad branding exercises like “Coke is it” or “Team Blue protects you—team red wants you dead.”
1. Let's say there are two parties: pink and purple.
2. You absolutely love Pink Party because they want to paint the roads pink and force everyone to drive pink cars.
3. I'm running for Purple Party, which says everyone should be allowed to drive whatever color car they want, but we also want pink roads.
4. Gerrymandering is illegal, but our team determined that we can sway some areas. You live in one of those areas.
5. My team writes up an ad saying something like, "Vote Purple and support pink roads and pink cars!" We target all the Pink People living in the aforementioned areas.
6. Pink party runs a very similar ad.
7. You get confused, wasn't there one of them that didn't support pink cars?
8. You google the parties, and visit our website to read our platform. We see that you were in the ad segment, so we hide our support of any color car and change it to say that we're for pink cars.
9. You give up, and just decide that you like my beautiful face -- but more likely you just didn't see the difference, so you vote Purple Party at the election.
10. I get elected and we get pink roads. However, the cars are quite multi-colored.
Hopefully, you can see how this is dangerous? It's quite different than without targeting (where everyone sees the same message). The ads I saw told me what I wanted to see, while the ads you saw were what you wanted to see and confused you enough to vote against your own party.
1. Google and party website: this can already be done and it is easily detected if a person is interested in a topic since other sources will broadcast the party position on a topic.
2. Party-centricity: depending on your locale, people vote for a person not a party. Party affiliation is often a proxy for how an elected person would behave but there are typically differences between party and people. Those differences are critical for voters who do not identify with a party.
In the original hypothetical, candidates who differed from their party were better able to articulate that difference without broadcasting messages that alienated party stalwarts. I see this as a plus for democracy since people elect politicians whose behavior is more in line with voters’ interests.
On the other hand, I can see from your hypothetical that segmenting messaging campaigns allows liars to hide lying a bit better, which is a clear negative.
The voter could not make an informed choice in the case of targeted advertising because there would be no information. There is no signal, everything is noise catered to a particular user. Yuval Noah goes in depth with this were he imagines a world with human like AI agents that befriend us for the express purpose of politically manipulating us at a later date.
https://web.archive.org/web/20230428190758/https://www.econo...
You don't have to see ads.
But if you do, I don't see why you'd prefer the ads to be specifically optimised to trick you into doing something you weren't already going to do.
It's not just a question of what things are being advertised, but also how they're framing it.
The person that wrote the original text clearly knew they were leaking their age and location when they wrote that statement. Why are we flabbergasted that GPT can still infer that info with mild obfuscation?
I asked GPT4 to obfuscate the original text and it was smart enough to remove the mention of the Left Shark incident. So maybe using an LLM to sanitize all your posts online is a good defense to an adversarial LLM doxing you. I don't think I see the authors mention doing that at all in their paper (I may have overlooked it)
May be this was "obvious," but like a lot of research there's a bit of hindsight bias in that sentiment. The issue now is that while you can easily do this yourself in terms of inference, the problem is GPT4 allows this inference to be automated.
"Anything you say in public can be overheard and used against you" is somewhat different from "everything you ever said in public will be analyzed and compiled into a case against you" are slightly different shades of grey I'd say.
If it was discovered that someone had been operating a secret network of CCTV cameras pointed at every street corner on Earth for the last 20 years, would you say "well, you had no reasonable expectation of privacy anyway"?
(Also, why 2020? I could understand if you'd said 2016 with Cambridge Analytica but 2020 isn't clear to me).
I assure you plenty of companies were doing NLP based classification in this field much earlier than that. AdSense for example was doing it on pages, and of course Google search strings were a similar example.
If it really was every, then likely yes. No doubt there would be some interesting footage of important public figures in there too.
Not that I'm advocating against privacy in general, but when "I know what you did" is no longer asymmetrical, as in "I know what you did too", it'll certainly change how society thinks about these things.
Except that it was previously possible, and could be done far more cost effectively, before. Ad targeting systems from fifteen years ago were already doing this with precision in the same neighbourhood.
Yes you could get precise information but not just from a few sentences said by one user. There's also the quality of information you could previously glean from just unstructured data (noticeably lower)
Most people do not think that personal details they didn’t intend to share in their writing can be surreptitiously deduced from things like their word choice and phrasing.
It’s true, and people need to learn about and consider it, but no, most people have no idea that it’s really possible or how far it can go.
Their intent is to share what they intended to share. Efforts to go beyond that — whether through LLM’s, earlier data mining techniques, or Holmes/Poirot-like observation — are precisely “violating privacy expectations”
People are clearly aware of the fact that it's possible to monitor their movements from space, therefore it's not unethical to launch 1,000,000 satellites to continuously monitor all movement on every square inch of the Earth.
Somehow I don't think this really follows. The constant mistake I see in these kinds of threads is thinking that "scale" in and of itself has no bearing on ethical arguments. This is obviously false; 1 person pissing in a local river is not a big deal, millions of people doing it is a problem, and genocide is clearly worse than murder.
Or, I'd rather have a snowball thrown at me than an avalanche.
They can use it on profiles where that wouldn't've been possible before, and which do not have the obvious tells like "By the way, I live in Melbourne, Australia". Once an 'easy' profile has been labeled, then the model will be trained to learn all the features in that profile which correlate with living in Melbourne, including the ones no human had ever noticed before, and which will be present in other Melbourne profiles. (You could, and should, also remove the obvious tells from the training data after you have constructed the label, to force it to discover subtler features beyond the ones your human labelers used.)
This lets you bootstrap your classifier to extract information no human could have before. That would seem to be a 'pure privacy violation' by your definition.
I have reasons to think years. Ie. that you are years behind. Profiling individuals based on public data already exist, and it's a profitable business as well.
> using it for nefarious purposes.
Law enforcement (and various other state-owned entities) would beg to disagree. As for vendors, same. Edit: Ad networks, even.
Let's say "you speak Luxembourgish and ask where you can buy a Mercedes for driving your kids to school" You think it violates your privacy that I might be able to guess where you live, your gender, and your income from that? Did I miss when the world made Bayes theorem illegal?
Applying Bayes' theorem isn't illegal but making interference from other public information then de-anonymising other data based on cross-matching is an entirely different act, regulated by privacy laws.
Privacy isn't only about the method, it's also about scale and intent. See CCTV which can individually be dumped the footage from and rewatched to identify criminals vs. live feed surveillance with AI face detection to track everyone 24/7. Same data used but very different intents and outcomes.
I'm with you about the intent, but I don't follow the scale argument. It's not like it isn't a privacy violation if you only do it to one person.
However, it changes when you do that in an automated way, en masse, both ethically and legally.
I think only problematic ethically. From what i can see, there can't be anything illegal about collecting publicly available information and using any algorithm to analyse and deduce information from it.
For example, while they can't just divulge my name & age in the news, the captions under pictures in news articles can often still have people's names & ages under them without explicit user consent. Tabloids would likely cease to exist if it were any other way. ;-)
How would you feel if a browser plugin displayed these details next to every username on Hacker News?
The claim is birthdate+zip+gender could uniquely identify 87% of the US population, and there are respectable studies to back this stuff up (https://groups.csail.mit.edu/mac/classes/6.805/articles/priv...). There aren't a lot of days where there's only one person celebrating their birthday in an entire zip code.
Even factoring in birth year and gender, I struggle with this a bit, because I know that in densely populated areas of the country, where more and more of the population lives (upwards of 80% of the US lives in urban environments), you'll find multiple babies with the same gender being born in the same hospital on any given day... and there are often multiple hospitals servicing the same zip code!
Despite all this, your point still stands: it doesn't take much data to deanonymize someone.
I remember when store algorithms were identifying a teenage girl was pregnant before her family knew and offering her discounts on maternity stuff. That anecdote was from over a decade ago.
Today average users are worried their phone is listening to them because something in a conversation comes up in a targeted ad, clearly worried about the idea of "private voice data" being compromised, but somehow less weirded out by the fact that the end result is possible without listening to anything.
Our expectations of privacy in relation to our data need to change with the times, and research like this helps that slightly, though to be honest the real picture of what can be discerned from data isn't reflected in public research but is enshrined in private practice, and having seen some of that side of things this paper's 'discovery' is a giant yawn.
People should begin to prepare for a future where your anonymous activities online (esp images and text) are cross correlated to your public activities trivially by any interested party.
And where your deep dark secrets aren't so secret or where truth and lies can reasonably be revealed as such. Cheating in a relationship? Are you unique enough that a multimodal model fed all public social media data can't identify matching patterns in online behavior changes from early in a relationship to later for profiles where the SO later posted about infidelity and compare to your activity?
When switching from scented candles to unscented can tell a store's algorithm with access to broad data that you are pregnant before you've told anyone, we really can't yet imagine just what much larger models trained on much more massive data may yet reveal in the smallest actions and nuances.
That’s not invasion of privacy, that’s just making good (informed) guesses.