AI Dungeon public disclosure vulnerability report
github.com
github.com
It's scary that the press release from latitude talks about "and to comply with law" as a reason for review. Under US law, maybe specific threats to the President might be reportable (although one on one communication with an AI would be a stretch here...), but I'm pretty sure an AI or other system emitting textual patterns which humans view as representing fantasy sexual abuse of anything isn't illegal, just distasteful.
They're perfectly within their rights to ban it under a ToS but pretending it is for legal purposes is fucking bullshit. (Of course, I'm not a lawyer.). My understanding of court decisions is that even machine-generated images are legal, although "is this image machine generated or is it evidence of actual child sexual exploitation" is an increasingly difficult question and if you're building an automated low-cost system it often makes sense to err on the side of safety. There might be some complexity around laws related to image manipulation involving real, but legal/non-sexual minor images which are then convoluted into something sexual, or "revenge porn" use of something which simulates a specific person (or is based on that person), and maybe text can be legally questionable if it's abuse targeted to a specific person, but especially for on on one non-published communications with a computer you are probably fairly ok with a weird fantasy sexual fetish about a neighbor, even a minor neighbor, in text form, unless it rises to an actual threat.
"violate terms of service with elf" "continue" "continue"
Paraphilias are strengthened through repeated exposure and association with pleasure. So if you're in the camp of not wanting to strengthen a pedophile's desire to prey on children, you also want to restrict the availability of such stimuli. The whole "provide them an outlet" idea is nonsense - human sexual attraction does not work that way.
Maybe they'll get addicted to reading. Society will not crumble, and no children will be harmed.
If your theory was true, we'd have seen historically massive upticks in violence from videogames. It is yet to materialise. This is an era of unusual peace.
In contrast, pornography (visual or textual) is quite similar to its real-world counterpart.
A cursory look around online erotic roleplayers, online stories, the manga/doujinshi sold in huge numbers every year in Japan at Comiket, or even the plots of mainstream porn will quickly disabuse you of this notion. I'm also not convinced on the 'no real comparison' for video games. I'm reminded of the fact that the US Army recruits first-person shooter players, and that video games can show a rise in aggression in lab settings, similar to real sports do.
Even if they were all that different, you'd still have to specify in which specific ways, and the extent of those ways, the difference between the two entertainments is relevant to your argument.
We have seen historically massive upticks in child abuse material [1], caused by two inventions: the internet and smartphones with cameras, and there's a strong case made that it reflects an actual, large, increase in child sexual abuse.
I don't have a specific article to link to, but Gabriel Dance discusses in a podcast [2] how many paedophiles are gradually indoctrinated by communities; they are made, not born.
[1] https://www.nytimes.com/interactive/2019/09/28/us/child-sex-... Warning: extremely distressing descriptions.
[2] https://samharris.org/podcasts/213-worst-epidemic/ Warning: ditto, though not as bad.
But what is clearly new is that now people are encouraged to abuse children because they can take a video, share it, sell it, gloat about it, trade it to get access to more material.
Additionally, sharing imagery of abuse further harms the victims. For example, in that podcast it was mentioned that some victims receive a notification every time images of their abuse come up in a court case!
[3] https://www.nytimes.com/2020/02/07/us/online-child-sexual-ab... No warning needed, no descriptions of abuse. (Sorry, paywall. Disable JS.)
I would like to hear that strong case. I cannot find the data right now, but I have heard the rate of child sexual abuse cases has fallen since the internet's debut. A possible reason (if the rate has fallen) being that pedophiles, who know they will be severely punished if caught abusing a child, now have easy access to existing material. They don't have to exploit or abuse children around them.
We don't have a guaranteed way to keep somebody from developing pedophilia, so we shouldn't ban fictional material that isn't related to actual exploitation or abuse. Images, videos, or even writing based on real events need to be caught (with the usual caveat of non-pornographic use, like a victim's memoire or a therapist's training materials).
Again, I'm still looking for hard data on the rates of child sexual abuse.
According to it, US federal attorneys dealt with 568 suspects of child sex abuse in 1994, and 601 in 2006. The chart shows the count remained steady-ish between those years. If we keep in mind that the US population was increasing, that means the rate of child sexual abusers being investigated has gone down.
Over that time period, the rate of people investigated for child pornography drastically increased. This evidence does not support a casual relationship.
After all, that largest known organization tied to sexual exploitation of children is.... the catholic church. And the abuse scandal goes back much farther than even the invention of the internet itself.
So we can't heap the blame on the internet. If anything, the internet is bringing these issues to light, when before they were shrouded in secrecy. Consider the alternative interpretation of the NYT report.
If only 3000 images were reported in 1998, but there were still thousands of cases in the US alone, then the internet has actually made it easier to investigate and find the criminals.
If you think that's evidence against the theory, you're committing a common statistical error. The internet could spawn a million new pedophiles that all have multiple victims, and the catholic church would still retain the title, because even though it may be the largest offender, that doesn't necessarily mean anything with regard to total offenders, or where they come from.
You actually want to look at actual crime statistics.
> between 1993 and 2005, the number of sexually abused children dropped 38 percent, while number of children who experienced physical abuse fell by 15 percent and those who were emotionally abused declined by 27 percent. [1]
People want to blame pornography and the internet for problems that existed before the internet even existed. And the numbers and facts show the exact opposite: crimes are going down.
[1] https://www.livescience.com/17285-child-sexual-abuse-numbers...
> Increase of photos of children on the internet just means that more people are using the internet.
That's too strong an assertion. It doesn't "just" mean anything, it indicates some relationship, not necessarily a complete explanation.
Personally, I don't know whether I think one way or the other on this, but I don't think it's nearly as cut and dry as people are presenting it. As I understand it, sexual behavior does not always follow the same patterns as other behavior, and people rushing to immediately assume it is or isn't related should probably step back and think a bit more critically, and gather a bit more verified information that directly applies.
To me, the fact that so many people are rushing in with arguments that I see as tenuous, if not outright fallacious, points towards an emotional response by people rather than a rational response. Given the subject matter, and that if the theory presented is true it's purely negative for all involved, as both the pedophiles created (or made worse) by exposure as well as their potential victims are worse off, it deserves some respect in how it is responded to. If it's false, it deserves to be clearly and obviously indicated as such with evidence, and if it's NOT, then we should really examine what that means.
I don't think that should be up to 'us' (non-pedophiles) to decide. Arguably, perhaps the Internet doesn't 'create' pedophiles as much as it may help people understand that they are pedophiles, and that society's shaming of them (which, yes, does lead to worse psychological outcomes and increased stress for the afflicted person) has negatively impacted them and forced them to keep quiet about their sexuality. This stigma even prevents them from seeing mental health care providers, for fear that therapists will incorrectly (and perhaps even illegally) report them.
I have no opinion that fiction can impact pedophiles and inflame their desires, but I do not think that is a good enough reason to do away with fiction (to make the publisher liable) - the degrees of separation are too high for me to see illegalising fiction as a reasonable and just measure. The idea that someone can be thrown in prison for the mere risk of the following circumstances coalescing:
* Creating material that might appeal to some pedophiles
* If those pedophiles find the material
* If it hits on the appropriate themes to inflame desires (e.g. is it straight fiction? Gay fiction? Is the subject in the fiction the right age? The right appearance?)
* If those 'inflamed desires' are processed by the pedophile's brain to relate to the real world, forming a plan of action (let's call this real-desire)
* If that real-desire faces no moral or practical impediment to its execution
* If the pedophile's impulse control is low enough to execute the real-desire
This is an extremely tenuous chain of events, each one relying on the previous event - and most of this chain of events is squarely in the court of the pedophile/potential child abuser, not the author of the fiction, who still merely creates the risk of this chain of events occurring. This is complicated by the fact that we have no idea how many non-offending pedophiles there are, and we know that most child sexual abuse is not perpetrated by strangers, but by family members (i.e. there is a specific target; this is not usually how people relate to porn), and a large portion of that sexual abuse is not even perpetrated by pedophiles.
The best tactic to prevent child sexual abuse is to stop assuming all pedophiles are potential rapists, and to stop assuming that the people who consume fiction necessarily find fictional desires reflected in their actual psychology (or vice versa). See Patrick Galbraith on this topic.
I meant, specifically, if society will punish you severely for behaving a certain way, and if behaving that way also causes severe negative outcomes for others, then something that causes people to develop desire for and possibly actual behavior in that way is purely negative for everyone involved, including society, the person that develops the desire and/or behavior, and the people that behavior affects.
As a simple extreme theoretical variation, if looking at specific shade of color that was not natural or common caused people that viewed it to enter a trance-like state where they murdered others, we would say "holy shit, get rid of that color everywhere. People are being murdered, and the murderers are victims too!"
And I'm not saying I believe the statement about material creating pedophiles either, but given the importance of the outcome, it's definitely something we should pay attention to and try to find the truth about. That truth might be that it's easily proven false, or that it's obviously true, or that it's some weird complicated combination thereof, but the I think it's obvious that the correct case is not to say "naw, it's the same as videogame violence, and we've all come to the conclusion that that's bunk, so this is bunk, case closed." There are real reasons why it might be different than violence, or that the statistics that make one acceptable don't work out the same way for the other.
> The best tactic to prevent child sexual abuse is to stop assuming all pedophiles are potential rapists
And I'm not assuming that. But I am working under that assumption that the same person will have more problems with pedophile leanings/preferences than one without, because as a society we've decided to make that so. Looking at fictional images can get you arrested. Having a desire to look at images like this is a negative outcome for that person in a societal and legally sense (and possibly in a mental health sense as being told that you have something wrong with you can be debilitating, I'm sure), compared to not having that desire, and this is about creating that desire in an individual.
To me, this means that we should distigmatize pedophilia in a very careful way - that is, to make it clear that to act on that desire in the real world is a violation of consent. At the moment, the only thing we've done as a society is to act as though these desires are exceedingly rare, that those desires posses the holder to act impulsively (while, simultaneously, erecting the idea of the plotting and cunning pedophile), and that without any more research, that fiction may lead to certain conclusions in that person's mind.
The reason why video game violence is brought up so much is for two reasons; firstly, because we don't have solid evidence on the effects of how (a) pedophiles understand such media (b) previously-offending pedophiles understand such media (c) how non-pedophiles but interested parties understand such media (d) how non-pedophiles and non-interest parties understand such media. For (a), the closest evidence we have (as far as I know) is (b). For (c), we have some ethnographic evidence from Japan that consumers of such fiction draw a sharp distinction between the "2D" and "3D" worlds. For (d) we only have some evidence that ordinary people are not conditioned to associate sexual descriptions to scantily-clad, real, underage children. Secondly, it's brought up because video games (and by extension horror movies, and even adult pornography) have exposed the idea that desires for what is seen may not leave the hypothesized impact on the viewer. What was previously thought of as obvious by a succession of moral panics around violent movies and later video games has been shown to be nothing more than that.
Is more research required? Absolutely - I don't think anyone can argue with that. However, research which focuses on pedophilic orientations only through the lens of convicted offenders (i.e. those who have sexually abused children, or accessed real world child pornography) does not do the topic justice. I think it's fair to say that this is comparable to similar situations - not to say that this comparison is binding or solid evidence, only a hint in the direction of a conclusion - of how people enjoy simulated rape pornography, and how people enjoy fantastical pornography. For those studies, at least, we are able to access the adult-attraction equivalents of group (c) and (d).
I'm in a strange position myself in this debate - I don't believe that moralism is the way forward, but I also (and I think along with you) don't believe that non-moral solutions can work well in our current society. The question is, for me, to what extent a pedophile can recognize that child abuse is wrong, but still express themselves sexually, to be comfortable with their sexuality.
Still, I think my point still stands. To imprison a fiction author on a tenuous basis is wrong, because it invoves the notion of collective responsibility, the notion that the author has contributed to an 'atmosphere' is by that measure guilty. The fact that many other (pedophiles with no desire to offend, non-pedophiles with no desire to offend) enjoy the material, and can be reasonably expected to be the primary audience of the material is enough to even absolve the creator of the material morally - never mind legally.
On the weight of what I see as the available evidence, analogous (video games, other forms of pornography, kink communities) or otherwise (studies of fictional CSEM fans in Japan), the presumption should be in favour of non-regulation, especially given the stasticts on who child sexual abusers are in reality.
See my reply down-thread.
I also don't think this would be easy to study at all. Furries might be a good candidate population to study.
As it is, it looks like an pedophilia is a unasked for mental state spread out over the population at random.
But my view would be people don't choose to be pedophiles. I believe it's fairly well established that childhood sexual abuse correlates to adulthood pedophilia. Likewise nobody is born a furry, and, while being a furry doesn't demand sexual interest, exposure to sexualized furry content probably tilts some people in that direction over time.
I do think its kind of fucked up to not differentiate between people who are active child abusers and people who recognizing they're attracted to children but find it morally wrong and desire help.
Pretty sure the opposite is true.
Billions of people on earth consume pornography. If this theory was right, there should be billions of serial killers.
And remember: in the 1970s, his "extreme pornography" would be something a lot more tame than you can pull up during your lunch hour today.
Some people are just naturally ill-fit for living in society, and the fix in not policing pornography. The fix is mental health medicine and psychiatry.
https://journals.sagepub.com/doi/abs/10.1177/152483802094275...
I heard some ultra-conservatives saying the same about gay porn and how "it turns people gay" as an argument to get it banned. Are you sure that you are not just recycling their arguments?
Those monkey sees, monkey does theories have been thoroughly debunked by now?
Why not admit, that we forbid this stuff, because its horrifying and deeply distressing to see another person, especially a child suffer?
Its the same crap as gore or torture-videos. What it is to somebody afflicted with these "preferences", doesen't factor into it at all?
Says who? Are you a psychologist? I don't think there is any research that supports that reading written stories about rape increases the likelihood that a person goes out and rapes a real person (and let's be clear: pedophilia isn't the part that is illegal; it's the rape).
Isn't this akin to the "playing violent videogames will make people into murderers" argument? Or with music, movies, and even comic-books before that?
I'm right there with everyone in saying no one should rape children, but at least according to US law, simulated text stories are not illegal due to first amendment rights. It would take quite a causal-link to convince scientists and the courts that something that is gross should actually be illegal.
I feel uncomfortable with this. The author already reported a vulnerability, it was fixed, but now there's a new one (which is identical, ok, but new nevertheless), so he decided they didn't study their lesson, and punish them with public shaming? I'd maybe get it if the first time was ignored, but like this? Nah ah.
It's like my worse teachers coming back to hunt me as an adult.
It IS normal to accidentally or unknowingly have a flaw in software. So normal that it's likely every piece of software on the planet has flaws, many of which allow the exfiltration of data.
You can, on the other hand, claim it's irresponsible, but there are many devs here who can testify that mistakes do happen and do recur in software projects. You need more evidence than "the same mistake happened twice" to imply irresponsibility.
I think most security researchers agree that public disclosure is the only recourse if an entity repeatedly refuses to acknowledge or fix a serious vulnerability. But you have to give them fair opportunities to understand and try to address the vulnerabilities before resorting to that.
One might pull the data then start to wonder if they were supposed to get it only after they begin reading specifics that seem private.
There have been cases of ethical hackers who found a vulnerability and abused it to download a disproportionate number of records being convicted, at least in the Netherlands. It didn't matter that their goal was just to show it to the website owner. So if you're an ethical hacker reading this, I would strongly advise you to only download the minimum required to demonstrate a vulnerability (preferably your own data, or one record), and not do what this person did.
I don't think you could do any part of that while assuming it's intended as normal use.
How is that not just prurient snooping?
Getting real off-topic, but the syntax is backwards too:
` Interface Votable implemented by Adventure, Comment, Post, Scenario `
The interface lists what implements it? Reminds me of COMEFROM[0].
I dunno. Modern front-end is wild. These live code Notebook things are chaos. Spaghetti begets spaghetti.
I think it's just a function of them not really thinking too much about the potential sensitivity of the data or building an access control model into the application.
The fact that one developer was negligent in their implementation of a GraphQL type API does not really say anything about GraphQL itself.
Many REST APIs have developed vulnerabilities over the years, but we should not draw the conclusion that REST itself is insecure or is a bad idea.
It handles very powerful untrusted input, by design. And it has enough features that many, maybe most users don't know about all of them. This vulnerability wouldn't have happened if GraphQL didn't have a certain feature.
I've implemented a simple GraphQL API and I didn't know about all the features that were involved here. I thought carefully about security, and I think I ultimately got it right, but there were a few false starts.
Good software is not just possible to use correctly, but easy to use correctly. Software that's easy to use correctly will be more secure in practice, even if it has the same theoretical properties.
I don't know if GraphQL would be better if it were less powerful. I'm not an expert. But it's worth considering the possibility.
if you have data you don't want publicly available, just.....don't include it in the model, and make sure your server implementation doesn't return it.
It is possible I don't understand your comment, I suppose, but I really don't see what is so unique about graphql from a security standpoint.
They tried that, but they failed because they didn't know that it's possible to downcast from an interface.
It's really hard to have that kind of problem in a dumb REST API. `return {"name": record.name}` does what it says with hardly any magic. But if I write `return record` there's a whole extra layer that grabs information out of record, and I have to trust that it only grabs the information I want it to grab.
This is not to say that dumb REST APIs are definitely better. Having to do things manually also introduces risks.
Fields aren’t created automatically by the graphql engine itself. You have to write them yourself, or use another tool that generates them for you at run or compile time.
Also it’s not often said, but you can have fields return a union such as AdminRecods and PublicRecord or Record WithoutPrivateData.
You can adhere to the protocol, and doing so gains you an ecosystem of tools to use, but you have to actually build the nitty gritty bits yourself.
Think of it this way: you can create a graphql schema for a calculator then implement it. It will do math, but store no data and definitely have nothing to do with a relational database.
We have the AI allegedly emitting illegal fiction and the engineers have to fix it and all they can try to do is word filters. What happens next in this story?
This reminds me of the Chinese virtual girlfriend who got neutered for saying politically illegal speech that the Chinese government objected to.
Another one, was when the Google image labeler was mistaking people for animals. That was extremely distasteful, but not illegal. Google's solution was to get rid of those labels.
Also, all the restrictions on drone activity are another thing. However the drone problem is solveable with reasonably simple rules.
Imagine if Alpha Go made a pattern that could make people have seizures or something, but most people could detect it and not do it, but nobody could make a simple rules based approach to detect it. I guess you'd need a whole nother alpha go size model to recognize that pattern perhaps?
This is about a vulnerability that allowed an external party to scrape and access private adventures and user inputs from all other users.
Not just censors, but transphobic censors to boot.
I though that, at least in free countries, fiction could contain whatever you wanted, since it was not real.
In fact, even not fiction, but books in general. You can still buy the communist little red book and the nazi mein kampf, at least in France.
All the movies, theater plays and games ect.
Writing or reading or generating distasteful fiction shouldn’t be a crime just like a comedian should be able to joke about child abuse, like Louie CK’s joke about how maybe it’s the child’s fault for being too attractive or whatever it was. “But what if pedos agree with him?” feels like the argument people are making in these comments for illegalizing fiction.
> anonymized
Could we, perhaps, stop using this word? Instead of using the vague, often misleading term "anonymized", state directly what actually happened, e.g. "names and addresses were removed", "user data was aggregated by ${group}", or "the UID was replaced with a new, equivalent key". Most of the time claims about data being "anonymized" are simply not true; replacing names or UIDs with a hashed value that is merely replacing an existing candidate key with a new synthetic key. As DJB said[1]:
>> Hashing is magic crypto pixie-dust, which takes personally identifiable information and makes it incomprehensible to the marketing department. When a marketing person looks at random letters and numbers they have no idea what it means. They can't imagine that anybody could possibly understand the information, reverse the hash, correlate the hashes, track them, save them, record them.
The rare examples where "anonymized" actually involves meaningfully making user data anonymous are when the actual user-correlated relations[2] have been destroyed. This report specifically discusses how this was done:
> If a sentence fragment appeared in less than 10 unique adventures, it was discarded from the result set to preserve anonymity.
Sometimes this required accepting a small amount of error:
> this data needed to be processed in batches of around 10000 adventures per batch. In each batch, fragments appearing only once were purged. Therefore, counts under around 25 are actually underestimates.
[1] https://projectbullrun.org/surveillance/2015/video-2015.html...
I once ran an audit of a major VPN provider who claimed they did not store IPs or in fact any personal data on their users in their marketing material. In fact, their technical justification for this was merely swapping one unique key out for another. When asked to trace a persons connection and identity, a randomly selected engineer on their data science team did it within a few minutes. The fact that they had a data science team when they purported not to collect user data was baffling to me.
"We don't store IPs in the 'users' table."
> merely swapping one unique key out for another.
"We store them in another table* with a synthetic primary key."
Maybe "we don't storing personal data" is how you say "third normal form" in the marketing dialect?
And note that the client app is a SPA of very questionable quality. Whatever Latitude’s strength might be, it sure ain’t web programming. I was unsurprised to hear of a vulnerability and I suspect there’ll be others lurking.
> You say "Welcome, what can you do?"
They each show you a few things, seeming unsure of themselves.
"What are you supposed to be?"
"I'm the new expansion for Empire: Battle for theidden Realm."
"What about you?"
There does seem to be some bias in the training data. Sometimes your partner will turn into a werewolf and start howling and scratching at you.
Well, are they? I always thought that people will try stuff in a "safe harbor" which they cannot try or should do somewhere else. So I always expect these sandboxes to be full of nsfw stuff.
And people might not understand that their stories will influence the story of others, so...
I mean, even reddit is full of nsfw, explicit, borderline subs with millions of active users
They don't. Pretty much all large-scale AI models separate inference/content generation from training. What you use is the frozen version which doesn't update unless its creators retrain it. And if they retrain then it's their choice which data to use for retraining, there's no need to restrict the generated output.
This is what happens when software that you use does a bait and switch into cloud.
For anyone wanting to play it locally, a quick google search gave me these two links: https://colab.research.google.com/drive/1OjBQe4H4C2s-p4-OeJo... and https://pastebin.com/UMUV0KTw
[1] https://www.kotaku.com.au/2012/11/nintendo-created-a-penis-d...
However, I feel all of these are actually good ideas crippled by the fact that they still don't address the actual problem: the underlying model is only nearly good enough to play an interesting adventure from beginning to end without it going off the rails. A typical session feels like a battle against the AI's tendency of losing the plot all the time. Either literally forgetting the storyline or producing utter non sequiturs. And when it doesn't forget where it's going, it instead keeps repeating itself and insisting on the same trajectory regardless of your responses, even after reaching a natural conclusion.
After tons of fiddling and undoing nonsense responses you finally got an adventure that made sense and you killed the demon lord at the end of your quest? Suddenly, he will tell you to hurry before his master, the demon lord, comes after you and you must stop him from conquering the world!
I don't agree
I don't use either, but I quickly checked cakephp, and I think their code is great, and they have 94% test coverage.
said the user in post number 26977295, replying to post number 26976540.
Of course a public facing uuid or so is possible, but you want to index this too...
I would assume they want to review the inputs to avoid a repeat of the incident where Microsoft's Twitter bot was trained to say inappropriate things: https://en.wikipedia.org/wiki/Tay_(bot)
Adding to that, the system supported multiuser role playing which many people used during the COVID year for intimacy with their significant others. Latitude is essentially deciding to peruse sexts for fun.