On the dangers of stochastic parrots: Can language models be too big? (2021)
dl.acm.org
dl.acm.org
Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?")
There's still a lot of work to be done and the real progress will be made by researchers who implement systems in collaboration with their colleagues and employers.
I wonder if anyone's working on public models of this size. Looking forward to when we can selfhost ChatGPT.
I suspect if one wanted to fully replicate ChatGPT from scratch it would take ~1-2 million including label acquisition. You probably only require ~200-500k in compute.
The next few years are going to be wild!
I wonder if this is the first rumblings of the singularity.
But what will be needed is to create an AI that implements scientific papers. About 30% of papers have code implementation. That's a sizeable dataset to train a Codex model on.
You can have AI generating papers, and AI implementing papers, then learning to predict experimental results. This is how you bootstrap a self improving AI.
It does not learn only how to recreate itself, it learns how to solve all problems at the same time. A data engineering approach to AI: search and learn / solve and learn / evolve and learn.
That's what I didn't like about Gebru - too much critique, not a single constructive suggestion. Especially her Gender Shades paper where she forgot about Asians.
http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
I think AnthropicAI is a great company to follow related to actually solving these problems. Look at their "Constitutional AI" paper. They automate and improve on RLHF.
I recently saw a screenshot of someone doing trolley problems with people of all races & ages with ChatGPT and noting differences. That makes me not quite as confident about alignment as you are.
Basically you have to convince modern models to say bad stuff using clever hacks (compared to GPT-2 or even early GPT-3 where it would just spout straight-up hatred with the lightest touch).
That's very good progress and I'm sure there is more to come.
I remember seeing that on Twitter. My impression was author instructed the AI to discriminate by gender.
If I say to discriminate on some feature and it consistently does it the same way, that's still a pretty bad bias. It probably shows up in other ways.
There was a flag on one of the responses, though it apparently didn't stop them from getting the output.
Thus, any techniques aiming to eliminate bias must come in the form of a set of hard coded definitions of what the author feels is the correct set of morals. Current methods may be too specific, but ultimately there will never be a perfect system as it's not even possible for humans to fully define every possible edge case of a set of moral values.
I think the class of problems that remain are basically outliers that are misaligned and don’t trip up the model’s detection mechanism. Given the nature of language and culture (not to mention that they both change over time), I imagine there are a lot of these. I don’t have any examples (and I don’t think yelling “time’s up” when such outliers are found is at all helpful).
What really confuses me here is how this paper is somehow outside the realm of valid academic discourse. Yes, it is steeped in activist, social justice language. Yes, it has a different perspective than most CS papers. But is that wrong? Is that enough of a sin to warrant such a response that this paper has received? I'll need to finish the paper to fully judge, but I'm leaning towards no, it is not enough of a sin.
Your concern about motivated reasoning is a valid one – but motivated reasoning is different to persuasive language. If anything, persuasive language highlights the places you should be looking for motivated reasoning (but it also makes it harder to spot, for some people; I don't recommend using persuasive language in your own papers).
Page 3 of a 14 page paper already claims with a straight face that LLMs are responsible for the sinking of Maldives.
> Training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to require as much energy as a trans-American flight.
> Is it fair or just to ask, for example, that the residents of the Maldives (likely to be underwater by 2100 [6]) or the 800,000 people in Sudan affected by drastic floods7 pay the environmental price of training and deploying ever larger English LMs, when similar large-scale models aren’t being produced for Dhivehi or Sudanese Arabic?
"Author is not affecting a neutral point of view" is not the same as "author is distorting truth".
Like, it absolutely does not claim LLMs sinking the Maldives. How can you say that with a straight face?
I haven't read the paper beyond the abstract yet, so can't comment on it's contents, but can you specify what you mean by "how people have described it on HN"?
To be frank, I don't like this style of argument, because I literally don't know what you're referring to. I'm quite familiar with the hubbub that happened when Gebru left Google, but feelings around what happened there shouldn't be misconstrued as feedback on this paper specifically.
In other words, if you think that people are arguing that this paper is "outside the realm of valid academic discourse", I think you should call out what you're referring to, because I haven't seen that as a widely held opinion. It feels like a straw man, or at the very least you are discounting some very specific critiques other commenters have made by just broadly referring to "HN sentiment".
My understanding is that this is the main reason why Google wanted the paper pulled. You can't have a paper by Google authors saying incorrect things about how much CO2 Google is emitting.
The authors refused. They argued that they gave a citation for their numbers, and it's not their problem if the paper they cite is inaccurate.
Eventually Google did publish a response: https://arxiv.org/abs/2204.05149 But this got totally buried...
But they do solve many tasks correctly, even problems with multiple steps and new tasks for which they got no specific training. They can combine skills in new ways on demand. Call it what you want.
Could they solve tasks? Potentially. But how would we ever know that we could trust them?
With humans we not only have millennia of collective experience when it comes to tasks, judging the result, and finding bullshitters. Also, we can retrain a human on the spot and be confident they won't immediately forget something important over that retraining.
If we ever let a model produce important decisions, I'd imagine we'd want to certify it beforehand. But that excludes improvements and feedback - the certified software should better not change. If course, a feedback loop could involve recertification, but that means that the certification process itself needs to be cheap.
And all that doesn't even take into account the generalized interface: How can we make sure that a model is aware of its narrow purpose and doesn't answer to tasks outside of that purpose?
I think all these problems could eventually be overcome, but I don't see much effort put into such a framework to actually make models solve tasks.
I don’t have millennia, but my more than 3 decades of experience interacting with human beings tell me this is not nearly as reliable as you make it seem.
To claim that only humans can accurately solve a task using words and wisdom is to give humans too much credit. They are not that lofty, sacred, or absolute.
And submarines don't swim.
And more to the point, nobody believes that "it thinks" is sufficient qualification for a job when hiring a human, so why would it be different when buying a machine? Whether or not the machine "thinks" doesn't address the question of whether or not the machine is capable of doing the jobs you want it to do. Anybody who neglects to evaluate the functional capability of the machine is simply a fool.
There are swimming robots.[0][1] Swimming is qualitatively different to what submarines do. The exception is helical flagella, not robots.
Don't you think they're an exemption because they're alive? If seals had propellers we'd still say they swim. Squids use jet propulsion and we still say they swim; do jetskis also swim? Somehow not.
If you are an LLM, I will grant you that claim.
There is no falsifiable hypothesis to be found in it.
I think this paper will age very poorly, as LLMs continue to improve and our ability to guide them (such as with RLHF) improves.
You don't come out feeling "Voila! this tiny thing I learnt is something new", which does happen often with many good papers. Most of the paper just felt a bit anecdotal & underwhelming (but I may be too afraid to say the same on Twiiter for good reason)
LLMs dramatically lower the bar for generating semi-plausible bullshit and it's highly likely that this will cause problems in the not-so-distant future. This is already happening. Ask any teacher anywhere. Students are cheating like crazy, letting chatGPT write their essays and answer their assignments without actually engaging with the material they're supposed to grok. News sites are pumping out LLM-generated articles and the ease of doing so means they have an edge over those who demand scrutiny and expertise in their reporting—it's not unlikely that we're going to be drowning in this type of content.
LLMs aren't perfect. RLHF is far from perfect. Language models will keep making subtle and not-so-subtle mistakes and dealing with this aspect of them is going to be a real challenge.
Personally, I think everyone should learn how to use this new technology. Adapting to it is the only thing that makes sense. The paper in question raised valid concerns about the nature of (current) LLMs and I see no reason why it should age poorly.
> Finally, we note that there are risks associated with the fact that LMs with extremely large numbers of parameters model their training data very closely and can be prompted to output specific information from that training data. For example, [28] demonstrate a methodology for extracting personally identifiable information (PII) from an LM and find that larger LMs are more susceptible to this style of attack than smaller ones. Building training data out of publicly available documents doesn’t fully mitigate this risk: just because the PII was already available in the open on the Internet doesn’t mean there isn’t additional harm in collecting it and providing another avenue to its discovery. This type of risk differs from those noted above because it doesn’t hinge on seeming coherence of synthetic text, but the possibility of a sufficiently motivated user gaining access to training data via the LM. In a similar vein, users might query LMs for ‘dangerous knowledge’ (e.g. tax avoidance advice), knowing that what they were getting was synthetic and therefore not credible but nonetheless representing clues to what is in the training data in order to refine their own search queries
Shame they only gave that one graf. I'd like to know more about this. Again, miss me with the political garbage about "dangerous knowledge", the most concerning thing is the PII leakage as far as I can tell.
I feel this is a unavoidable consequence of using LLM. We cannot ensure all data is free from any markers. I am not a expert on databases/data engineering so please take it as an informed opinion
Being able to memorise information is demanded when we want the top 5 countries by population in Europe or the height of Everest. But then we don't want it in other contexts.
Looks more like a dataset pre-processing issue.
Is it conceivable that a model could leak PII that is present but extremely hard to detect in the data set? For example, spread out in very different documents in the corpus that aren't obviously related, but that the model would synthesize relatively easily?
> We are an interdisciplinary and globally distributed AI research institute rooted in the belief that AI is not inevitable, its harms are preventable, and when its production and deployment include diverse perspectives and deliberate processes it can be beneficial. Our research reflects our lived experiences and centers our communities.
Timnit tried to raise this to Jeff Dean to get help (Jeff was Megan's manager at the time). Jeff completely misunderstood what she was asking for, and instead sent some response about the environment numbers being incorrect (and they are, but that doesn't at all justify the way Timnit was treated). Not beginning to imagine that Jeff could have missed this signal, Timnit responded sarcastically. Jeff didn't pick up on the sarcasm and thought all was good.
Timnit then reacted by describing her frustrations with how she was treated in an internal diversity mailing list. She also emailed Megan Kacholia with a number of demands, mostly to be treated reasonably. Appalled at how she and her coauthors were treated, she refused to retract the paper. Megan reacted by taking her note that she would work on a resignation date if demands were not met in combination with Timnit's email completely pedantically and out of context, using them as an excuse to fire her by rushing her out, without allowing her to follow the actual resignation process. She also acted over Timnit's manager's head (Samy Bengio), who was so annoyed he later quit. (Megan cc'd Jeff, but hadn't spoken to Jeff about any of this, and was acting on her own.)
Interestingly, Timnit's email to the diversity list was so resonant that several of the changes it asked for in how Google approaches diversity were enacted after her firing. But Megan and Google's official line on all of this chose to obsess over Timnit's rhetorical devices and take them literally instead, using an email to a diversity list about diversity against her. People are still too afraid to talk about diversity on diversity lists, now, because of Google using that email against her.
Google reacted by gaslighting Timnit to protect its ass. After Timnit Tweeted that Jeff had fired her (Timnit probably really thought that Jeff and Megan had spoken to each other before Megan had sent that email), Jeff participated in this part in public, on Twitter, with a lot of serious consequences for Timnit and others, without considering power dynamics. (Jeff suffered a lot on Twitter, too, but that doesn't excuse not considering power dynamics in such a consequential way on such a consequential medium.) Timnit and others, including me, were harassed and threatened because of this, by way of third-party harassers. I was not even involved on the paper, just proximal damage. I was afraid for my life honestly.
Meg Mitchell, feeling lost, having seen the truth of how Timnit was treated internally, and trusting Jeff to protect her, tried to put together some things for Jeff to get him to see how Timnit was mistreated. She panicked and backed them up on her personal email because she was afraid of retaliation from Google (a reasonable fear---doing any diversity or community work at Google that at all challenges the status quo IME gets you retaliatorily reported to PeopleOps, who then try to get you in trouble and read your private communications and so on). She was transparent about doing this and gave instructions for Google to remove her personal copy if needed. Sundar Pichai fired her and then comms smeared her publicly with outright lies. She was harassed and threatened for this, too, and a number of places refused to hire her because of Google's treatment of her. Really tangible damage emotionally, financially, and reputationally.
Out of fear of being sued, Google's comms and legal departments reacted by continuing to censor and gaslight. Sundar was extremely complicit in this, too. Megan was moved out of Research, but not much else happened; she continues to send monthly emails about diversity, as if her continued contact with Research is not actively harmful to diversity.
So sick of internet people speculating about this without knowing anything about the situation. Sorry if I broke anyone's trust here. Just can't deal with this incorrect speculation anymore. (I have extremely thorough information about this, but to those directly involved, please feel free to correct me about any details I got wrong, or about important details I omitted.)
Nevertheless, hoping to check your sources I clicked through your profile and I have a question, totally unrelated to all this. Can you say something about the state of the art in "neural proof synthesis"? To clarify, I'm scare-quoting because I didn't even know that's a thing. For context, my background is in the European tradition of Resolution-based automated theorem proving (Prolog and all that) but also statistical machine learning, so don't worry about simplifying terminology too much.
Btw, the "proof engineering" link in your profile gives me a security alert on firefox.
Language models are showing a lot of promise for autoformalization: automatically converting natural language mathematics to formal definitions, specifications, and proofs. This is a task where symbolic methods do not seem particularly promising in general, and one that meshes nicely with synthesis.
A good conference to look at is AI for Theorem Proving (AITP). It's small but has a lot of relevant work. All of the talks from this past year are recorded and on the website. MATH-AI at NeurIPS had some good work this year, too.
There is a bit of a culture and citation gap dividing the work in the AI community from the work in the PL/SE communities; in PL/SE I'd recommend work by Emily First and Alex Sanchez-Stern. They are undercited in AI work despite having SOTA results on meaningful Coq benchmarks. In AI, I'm particularly psyched about work by Yuhuai (Tony) Wu, Markus Rabe, Christian Szegedy, Sean Welleck, Albert Jiang, and many others. Tony's papers are a good gateway into other AI papers since the AI papers tend to cite each other.
Autoformalization also sounds interesting. I've had some conversations about automatically turning big corpora of natural language text into Prolog with language models, for example. I don't reckon anyone is even researching how to do this with symbolic methods at the moment.
I'll check out AITP. Thanks for the pointers. I'm used to small conferences [and to underciting between disciplines] :)
BTW what you say about the sadly common assumption Timnit and other AI ethics folks don't have "real contributions" is too real. It has impacted me even though my work isn't on AI ethics at all, just because I bother to talk about it online in public sometimes. Similarly for any social justice work or any work improving the work environment in research. It is like some people cannot comprehend that one can be technically proficient and still care about social justice and ethics and other "soft" issues. I love how confused those people are when they learn my expertise is in formal logic and proof haha
Vanishingly few people can get away with acting like that at work without getting fired. She thought she was an exception and she wasn’t.
Everyone who tries to actually enact real systemic change within Google Research to improve how women and people of color are treated, or the work environment in general, burns out or gets fired. It is only toeing the company line when it's officially your role to do so, taking extremely conservative actions while ignoring real issues like abuse, harassment, and discrimination, that lets you survive while doing DEI work at Google in Research without burning out or getting fired. The sole exception I know of right now is Kat Heller, god bless her.
The feeling of "none of this is worthwhile, what's the point of any of it" is something everyone who tries to do real diversity and culture work within Google Research or even within the computer science research community more broadly has felt, and a diversity mailing list ought to have been a safe place to share that feeling.
Instead it was used against her. Now nobody feels safe sharing these feelings anywhere within Google Research.
Jeff to his credit is now working on many of these things. As is the DEI committee. I hope they all succeed. But I don't think any of that would have happened without Timnit highlighting these very real issues. It's depressing that she was fired for it. She wanted to change things and keep her job and her team, not to be a martyr.
Note to all: if you give an employer an ultimatum "do X or I resign", don't be surprised if they accept your resignation.
The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones.
Which makes me think- abstracts for scientific papers are high-effort. The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs.
That's not to say that all scientific abstracts are amazing, just that their goal is to sound intelligent and convincing, while probably 60% of the junk fed into GPT is simply clickbait and junk content padded to fit some publisher's SEO requirements.
In other words, ask GPT to generate an abstract, and I would expect it to be quite good.
Ask it to generate a 5-paragraph essay about Huckleberry Finn, and I would expect it to be the same quality as the corpus- that is to say, high-school English students.
So now that we know these models can learn many one-shot tasks, perhaps some cleanup of the training data is required to advance. Imagine GPT trained ONLY on the library of congress, without the shitty travel blogs or 4chan rants.
That's certainly true, but it's not by a so large margin, at least in biology.
For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important.
Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers!
So the ratio signal on noise is close to zero.
https://www.alzforum.org/papers?type%5Bmilestone%5D=mileston...
https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&...
From a technical standpoint, there is little new stuff that I found this paper offered in understanding why LLMs can have unpredictable nature, or what degree of data will get exposed by clever hacks (or if there are systematic ways to go about it). It sounded more like a collection of verifiable anecdotes for easy consumption (which can be a good thing by itself if you want capsule understanding in a non-technical way)
(Anecdotally, I have faced a bite-sized brunt: When discussion surrounding this paper was going on in Twitter, I had mentioned in my timeline (in a neutral tone) that "dust needed to settle to understand what was going wrong". This was unfortunately picked up & RTed by Gebru & the mob responded by name-calling, threatening DMs accusing me of racism/misogyny etc, and one instance of a call to my employer asking to terminate me - all for that one single tweet. I don't want confrontations - not my forte to deal.)
Wait until an LLM flags your speech and gets you in trouble. That'll be a real hoot compared to random individuals who likely have been chased off Twitter by now.
Why didn't any of you try to stop this verbal carnage? Also if Timnit was largely not at fault, why is that on every social media where any modicum of anonymity is possible, Timnit had been harshly criticized for her conduct.
We respect her contributions to advancement of ML, but that conduct was inexorable & in very poor taste.
It's hard to believe sometimes that someone so smart in some ways can miss such obvious signals. It took a long time for me to come to that conclusion and to understand, and I'm still very upset with how Jeff reacted.
These are all people though, they make mistakes, Timnit included. The way she was treated is still not at all OK, and sometimes when we are in positions of power like Jeff we still hold responsibility for our mistakes. Even if Jeff genuinely misunderstood, he should have apologized for his role and moved to repair harm, rather than reinforcing and exacerbating harm in his public response.
I won't say anything else about Jeff in public. I did try to help minimize damage and pain in all directions. It was painful and exhausting and not very fruitful.
(Also sorry but who are you? Just because you addressed me by name and it feels weird when that happens unless I know who is talking to me.)
FWIW, at Google in Research, many people have mentioned her to me as the only person they trusted to talk to about the things they went through when they were not treated well. After she was fired, many of those same people felt no longer able to raise issues about internal treatment and culture.
Timnit was not carrying just her own burden, but the burden of many at Google in Research who were not treated well, especially women and people of color in Research. And so she spoke not just for herself. She had witnessed for years how demoralizing it is to try to really change things within Google. I think the only person who has done that and not burned out is Kat Heller. I did a lot of it over the summer, and it really took a toll on my wellbeing, and my desire to stay (part-time) at Google to finish my own work there. I'm excited for my affiliation to end so I can remove myself more thoroughly from Google's internal politics and culture, though I hope I've made enough of a dent that some things actually continue to change for the better.