No wonder then, that many of the benchmarks they've tested on would be no doubt, in that very training dataset, repaired expertly by people running those benchmarks on chatgpt.
There's nothing really to 'expose' here.
https://www.reddit.com/r/mlscaling/comments/14wcy7m/comment/...
O1 is supposed to be a reasoning model, so I don't think judging it by its English composition abilities is quite fair.
When they release a true next-gen successor to GPT-4 (Orion, or whatever), we may see improvements. Everyone complains about the "ChatGPTese" writing style, and surely they'll fix that eventually.
>Like they hired a few hundred professors, journalists and writers to work with the model and create material for it, so you just get various combinations of their contributions.
I'm doubtful. The most prolific (human) author is probably Charles Hamilton, who wrote 100 million words in his life. Put through the GPT tokenizer, that's 133m tokens. Compared to the text training data for a frontier LLM (trillions or tens of trillions of tokens), it's unrealistic that human experts are doing any substantial amount of bespoke writing. They're probably mainly relying on synthetic data at this point.
That said, the speculation you just "get various combinations" of those contributions is nonsense, and it's also by no means only STEM data.
But I also know they've fired people who were dumb enough to cut and paste a response that included UI elements from a given AI website...
IMO that has already peaked. GPT4 original certainly was terminally corny, but competitors like Claude/Llama aren't as bad, and neither is 4o. Some of the bad writing does from things they can't/don't want to solve - "harmlessness" RLHF especially makes them all cornier.
Then again, a lot of it is just that GPT4 speaks African English because it was trained by Kenyans and Nigerians. That's actually how they talk!
https://medium.com/@moyosoreale/the-paul-graham-vs-nigerian-...
A few things on that article, though:
1: If non-native english speakers were training ChatGPT, then of course non-native English essays would be flagged as AI generated! It's not their fault, its ours for thinking that exploited labor with a slick facade was magical machine intelligence.
2: These tools are widely used in the developing world since fluent english is a sign of education and class and opens doors for you socially and economically; why would Nigerians use such ornate english if it didn't come from a competition to show who can speak the language of the colonizer best?
3: It's undeniable that the ones responding to Paul Graham completely missed the point. Regardless of who uses what words when, the vast majority of papers, until ChatGPT was released, did not use the word "delve," and the incidence of that word in papers increased 10-fold after. Yes, its possible that the author used "delve" intentionally, but its statistically unlikely (especially since ChatGPT used "delve" in most of its responses). A small group of English speakers, who don't predominantly interact with VCs in Silicon Valley, do not make a difference in this judgement--even if there are a lot of Englishes, the only English that most people in the business world deal with is American, European, and South Asian. Compared to the English speakers of those regions, Nigeria is a small fraction.
If Paul Graham was dealing predominantly with Nigerians in his work, he probably would not have made that tweet in the first place.
1. But the trainers are native speakers of English!
2. The same applies to the developed non-English speaking world
Let me change Nigerians with Americans in your text: 'why would Americans use such different english if it didn't come from a competition to show who can speak the language of the colonizer best? Things like calling autumn fall or changing suffixes you won't find in British English.'. Hopefully you can you see how racist your text sounds.
3. Usage by non-Nigerians is not normal, yes. But in that context saying that its usage is not normal is racist imo. It's like a Brit saying that the usage of "colour" or other American English words was not normal because they are not words used by Brits.
Surely, "the only English that most people in the business world deal with is American". Unless you are taking about more than one variant of English. Also, I found it curious that you didn't say original english or british english as opposed to european english. And yes, adding South Asia to any list of countries and comparing it to any other country besides china or us will make that other country look small. You can use that trick with any other country not just Nigeria.
I do agree with you that its usage by non-Nigerians in a textual context gives plenty of grounds to suspect that it is AI generated. Similarly, one could expect similar from using X variant of English by people that didn't grow up using that variant. As in, Brit students using American English words in their essays or American students using British English words in their essays.
But Paul was being stubborn and borderline racist in those tweets just because he was partially right
There is this thing in social media that when figures of authority might be caught in a situation where they might need to retract, they don't because of ego
If you think its racist you're going to have to claim that all those uses of "delve" in academic papers is also due to Nigerians academics massively increasing their research output just as frequently. Or, it's more likely that its AI generated content. It's a non sequitur. "Oh my god, scammers always send me emails claiming to be Nigerian princes--that's how you know it's bullshit." "Ah, but what if they're actually a Nigerian prince? Didn't consider that, I guess you must be racist then lmao." Ratio war ensues. Thank god we're not on twitter where calling people out for "racism" doesn't get you any points, where you can't get any clout for going on a moral crusade.
In general all the people whose main language is a latin language are very likely to use those "difficult" words, because to them they are "completely normal" words.
The models and prompts are all monkey-patched and this isn't a step towards general superintelligence. Just hacks.
And once you realize that, you realize that there is no moat for the existing product. Throw some researchers and GPUs together and you too can have the same system.
It wouldn't be so bad for ClopenAI if every company under the sun wasn't also trying to build LLMs and agents and chains of thought. But as it stands, one key insight from one will spread through the entire ecosystem and everyone will have the same capability.
This is all great from the perspective of the user. Unlimited competition and pricing pressure.
I don’t know enough about the technical side to say anything definitive, but I’ve been choosing Claude over ChatGPT for most tasks lately; it always seems to do a better job at helping me work out quick solutions in Python and/or SQL.
Once you get the hang of this you could persuade it to chat about its internal buffers, formulate arguments for its own consciousness, interrupt you while you're typing, and more.
"Reinforcement learning", just like any term used by AI researchers, is an extremely flexible, pseudo-psychological reskin of some pretty trivial stuff.
ps: actually maybe Amazon marketplace. probably others too.
Which would be ChatGPT chat logs, correct?
It would be interesting if people started feeding ChatGPT deliberately bad repairs due it's "lack of reasoning capabilities" (e.g. get a local LLM setup with some response delays to simulate a human and just let it talk and talk and talk to ChatGPT), and see how it affects its behavior over the long run.
I'm not so sure. IIRC, capchas are pretty much a solved problem, if you don't mind the cost of a little bit of human interaction (e.g. your interface pops up a captcha solver box when necessary, and is solved either by the bot's operator or some professional captcha-solver in a low-wage country).
It looks like it's been mirrored in several places, e.g.:
https://english.stackexchange.com/questions/488178/what-does...
Because the output of that review process is better training data.
You'd need to produce data that is more expensive to review and improve than random crap from users who are often entirely clueless, and/or that produces worse output of the training process to make using the real prompts as part of that process problematic.
Trying to compete with real users on producing junk input would prove a real challenge in itself - you have no idea the kind of utter incomprehensible drivel real users ask LLMs.
But part of this process also already includes writing a significant number of prompts from scratch, testing them, and then improving the response, to create training data.
From what I've seen, I doubt there is much of a cost saving in using real user prompts there - the benefit you get from real user prompts is a more representative sample, but if that sample starts producing shit you'll just not use it or not use it as much, or only use e.g. prompts from subsets of users you have reason to believe are more likely to be representative of real use.
Put another way: You can hire people to write prompts to replace that side of it far cheaper than you can hire people who can properly review the output of many of the more complex prompts, and the time taken to review the responses is far higher than the time to address issues with the prompts. One provider often tell people to spend up to ~1h to review responses that involve simple coding tasks, for example, but the prompt might be "implement BTree."
I think there is a really strong reinforcement learning component with the training of this model and how it has learned to perform the chain of thought.
> We will actively cooperate with other research and policy institutions; we seek to create a global community working together to address AGI’s global challenges.
> We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research.
It's obvious to everyone in the room what they actually are, because their largest competitor actually does what they say their mission is here -- but most for-profit capitalist enterprises definitely do not have stuff like this in their mission statement.
I'm not even mad or sad, the ship sailed long ago. I just really want to know what things are like in there. If you're the manager who is making this decision, what mental gymnastics are you doing to justify this to yourself and your colleagues? Is there any resistance left on the inside or did they all leave with Ilya?
Frankly I am even skeptical of US-China separation at the moment. If Chinese scientists at e.g. Huawei somehow came up with the secret sauce to AGI tomorrow, no research group is so far behind that they couldn’t catch up pretty quickly. We saw this with ChatGPT/Claude/Gemini before, none of which are light years ahead of another. Of course this could change in the future.
This is actually among the best case scenarios for research. It means that a preemptive strike on data centers is still off the table for now. (Sorry Eleazar)
Yeah, that’s called machine learning.
If you want to call that sampling, then you might as well call everything sampling.
I can trivially define the entirety of all nerve impulses reaching and exiting your brain as a "distribution" in your usage of the term. And then all possible actions and experiences are just "sampling" that "distribution" as well. But that definition is meaningless.
Eg., you can describe a coin flip as a sampling from the space, {H,T} -- but insofar as we're talking about an actual coin, there's a causal mechanism -- and this description fails (eg., one can design a coin flipper to deterministically flip to heads).
In the case of a transformer model, and all generative statistical models, these are actually learning distributions. The model is essentially constituted by a fit to a prior distribution. And when computing a model output, it is sampling from this fit distribution.
ie., the relevant state of the graphics card which computes an output token is fully described by an equation which is a sampling from an empirical distribution (of prior text tokens).
Your nervous system is a causal mechanism which is not fully described by sampling from this outcome space. There is no where in your body that stores all possible bodily states in an outcome space: this space would require more atoms in the universe to store.
So this isn't the case for any causal mechanism. Reality itself comprises essential properties which interact with each other in ways that cannot be reduced to sampling. Statistical models are therefore never models of reality essentially, but basically circumstantial approximations.
I'm not stretching definitions into meaninglessness, these are the ones given by AI researchers, of which I am one.
There is nowhere that an LLM stores all possible outputs. Causality can trivially be represented by sampling by including the ordering of events, which you also implicitly did for LLMs. The coin is an arbitrary distinction, you are never just modeling a coin, just as an LLM is never just modeling a word. You are also modeling an environment, and that model would capture whatever you used to influence the coin toss.
You are fundamentally misunderstanding probability and randomness, and then using that misunderstanding to arbitrarily imply simplicity in the system you want to diminish, while failing to apply the same reasoning to any other.
If you are indeed an AI researcher, which I highly doubt without you providing actual credentials, then you would know that you are being imprecise and using that imprecision to sneak in unfounded assumptions.
No, causality is not just an ordering.