Hacker News Reader

Comment by Semkas | Hacker News Reader

So, leaving aside the idea that OA might've used data from the researchers Codex sessions: Do I understand correctly that the internal OpenAI work on the problems was probably started after they heard Alpoge and Buckmaster had made process by using their models? And they used the publicly available info about the researchers past work to prompt their models?

If compute is cheap, and the difficult thing with scientific discovery is now mostly in steering agents into promising areas, there's an obvious incentive for OA mathematicians to simply monitor closely which researchers are close to releasing exciting results, make some assumptions about their prompts based on their past work, and quickly prompt their own (stronger) model to look into the same areas.

View on HN

> leaving aside the idea that OA might've used data from the researchers Codex sessions

Why leave that aside? That is _the_ story.

If a Chinese research lab did this we'd call it espionage.

Because they didn't do that. Tristan doesn't specifically claim that they did, and Anthropic employees don't think they did either. https://x.com/_sholtodouglas/status/2097218240397410733

By default OA trains their models on codex-sessions. If I understand him correctly this is something Tristan explicitly mentions in his post as a possible reason for the fast results obtained by the internal OA team. Anthropic obviously doesn't want to challenge the idea that training is transformative, even if it means agreeing with their competitor.

Well, of course Anthropic employees would say that, since they likely do the same. Claiming that your primary competitor doesn't engage in a certain malicious practice is supposed to make it look as if there's no way you would too. If somebody even says that about their competitor, then surely there must be truth to that, otherwise you would never give credit to someone you're opposed to.

It's literally a toggle in the options for ChatGPT, one which is on by default and most researchers probably have on without realising it.

So to say that it is unlikely is extremely suspicious. No, they did not literally pull user data. But user data is automatically added to their training set by default, so their latest in-house model would be trained on it if it is from several months ago. It isn't intentional on their part, and they probably realised they could not refute that they trained on Tristan's logs unintentionally, hence why they acted the way they did.

Its very easy for OpenAI to answer, yes or no, if the model they used trained on their chats.

But there's a bunch of people already in this thread calling that stuff unfounded speculation (which I disagree with), and my point is that even if that specific thing isn't true, OA's behavior here is obviously awful.

If they're going to try to beat researchers to discoveries like this it disincentives researchers to talk about their progress publicly, and basically breaks the ecosystem of scientific cooperation / discovery. It's also immoral.

Yep. The most uncharitable view of this might be: they stole the work of researchers to build their models, and now they're using said models to steal the proceeds of future work, too.

Why is that the story? Is there anything to back it up beyond a single accusation?

As a prior I would say that a math professor has about infinite times more integrity than OpenAI.

If you think that OpenAI won't look at your data to gain a massive advantage, you're naive.

I’ve thought a lot about publishing research and wanting to do more of it, but right as I finally had the time and energy to start writing articles LLMs start to take off. Now all of a sudden, I’m acutely aware that everything I publish will be used for AI training.

For math, a field that is built on incremental research it feels like AI labs will do nothing but discourage publishing research at all for fear that they will be able to spend the money for compute that publicly funded academia simply cannot afford.

It feels like publishing anything at this point just means that your work will be fed to a machine that will make sure your work will never been seen by anyone else because it will always be the ones making the “true advancements”.

Perhaps I’d feel better about this if AI labs really existed for humanity’s benefit, but for some reason I don’t think that comes up in their investor slide decks.

Exactly the same thing that's been happening with vulnerabilities and bugfixes the past few months.

I fear that AI is going to cause ossifying secrecy in many fields, much like what happened semiconductor design the past 10-15 years.

>If compute is cheap

This cost millions of dollars of tokens.

This is an excellent point...

It's honestly unsurprising and not a problem that they do this in my view. The problem really starts when you start taking credit for work that they would've achieved.

Like if i go to a talk on unfinished work, it's not really unethical for me to think about the problem--it's a problem if i scoop the authors but these problems can often be solved by collaboration or proper crediting and timing--IN MY VIEW

The difference is how credit and attribution works. And whether we feel it’s being laundered through models.

And also whether the AI moon laser pointed at your problem is just going to be the thing that writes the final conclusion on ten years of your work.

Reply on news.ycombinator.com