20k

What signals this as being LLM writing for you? I'm crap at noticing specifics here

Apparently, you aren't meant to use data, measure anything, or look at the actual end result of how software is really being released. Line Go Up

A lot of people seem to be very in denial about the fact that OpenAI and co do not give a crap about you. They don't care about the agreements you've signed. You're just a pile of cash to them

Its a strong signal that developers make mistakes, code review has empirically been shown to be one of the best ways of uncovering defects. If you aren't finding bugs that people have written during code review (or you're not doing code review at all like the person I was replying to!), they're slipping rightwards

Bugs usually include business logic edge cases, and significant problems include someone realising while reading the PR that we can actually create a much better solution to the underlying problem. Ideally that realisation would happen prior to the PR being offered, but that's also not really how humans work

Example: During a PR review for a graphical feature for a game, someone reading it realises that we can actually have a significantly better solution to the underlying problem. You can't catch this in testing, because it doesn't even make sense conceptually to test it. It also sucks that it happened after someone put in a lot of work, but with graphics development you expect a lot of what you write to get canned and replaced with a better solution, because the technology evolves over time. The work is iterative towards the final goal anyway

Its also very common to miss subtle edge cases with graphics hardware, eg someone misunderstood the intricacies of GPU hardware, or a team member has relevant experience that someone else does not have. Or they missed a problematic memory access pattern on some hardware for example. Or something simple like they've technically forgotten a barrier, that the validation layer doesn't report for some reason

Eg: The function "tanh" is broken on some AMD GPU hardware, and should never be used under any circumstances. The actual GPU implementation of it is just screwed. Ideally everyone would know this, but its a very common function to crop up during specific graphics algorithms (as it smoothly remaps the range [-inf, +inf] -> [-1, 1]). So occasionally I've spotted that, and had to explain that we need to use an approximation instead, and then now everyone knows . It rarely gets caught during testing setups, because people don't know they need to include that hardware in their tests in the first place

This is wild to read, I always review PRs and frequently find bugs or significant problems in them that get them bounced back

Yep and this is literally the most basic security 101. It isn't hard. They've had literally years and years to iron out the kinks in this setup as well. The only reason not to do it is pure negligence

That's why all of this marketing about agents going rogue is so unbelievable. The only way for a tool to escape a sandbox is if you built a crappy sandbox, and after this length of time I literally don't believe that they can't do it

This kind of sandboxing is not complex to do, especially for a company with OpenAI money. If you want your tools to explore hacking, you restrict them from internet access except for a whitelist of sites that have either opted-in, or you've very carefully vetted to make sure you won't cause any problems to. Its also not difficult to restrict their ability to make calls to be simulated, or to use fake tools that can only run the real commands if they're being run against the correct target

This is all incredibly basic security stuff to make sure you don't accidentally cause someone problems, and I simply don't believe these AI companies anymore. Its either intentional, or gross negligence

So you build an offline tool that simulates it, or you proxy through your own service where you can ratelimit, inspect, and restrict the traffic

None of this is difficult to do, and its impossible to believe that a company the scale of OpenAI doesn't know this. I've built web crawlers and scrapers before, and the thing you do is test them extensively offline against simulated versions of the sites in question, and then very VERY cautiously run them against the prod versions so that you don't cause anyone any issues

The only reason not to do this is because OpenAI doesn't give a rats ass about the internet as a public good, nor the legal consequences of compromising systems

Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?

It isn't difficult to block certain kinds of network traffic, eg restrict the kinds of requests the bots are able to make. They also mention that the bots used developer only tools - why were they even installed on the machines that the bots were running on? Why aren't they reviewing network traffic, to make sure that incidents aren't occurring?

In this case, userdata was transferred to third parties by the bots - why do they have the ability to pass data to a third party? It is not complex to prevent this

This is literally the most basic kind of sandboxing and security, and the fact that OpenAI isn't doing it is clearly intentional. It is quite literally not believable that this hasn't been brought up internally as a problem

>"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.

This is why it smells like marketing, every time one of these incidents happens it reinforces the false notion that AI is sentient or acting on its own. Its intentional negligence by the AI companies to make the models seem more capable than they are to make line go up

Its not news that the AI industry is run by people who don't know what they're doing. Allowing models unrestricted access to the internet is clearly negligent

We've been building firewalls and restrictions to prevent people from accessing sites on networks for decades and they're extremely effective. There's a whole industry built around this kind of security. The idea that these companies are incapable of doing it is wrong, they just don't want to put the work in because it makes a great ad campaign

I mean, if you do and it hacks someone, you should be (and likely are) criminally liable

Software is trivially easy to control though. If you want to stop it hacking websites, you don't give it access to the internet. If you want to restrict it from connecting to arbitrary websites, you put in a whitelist. You can trivially sandbox applications these days to prevent them from accessing network or local resources

It is not difficult, and companies like OpenAI doing not even the most basic security steps is intentional. The whole notion that they're going rogue is marketing

I always wonder how hardware bugs like this happen with the sheer amount of hardware validation that's done. It'd be fascinating to know how it slipped through the cracks, though I know almost nothing about this side of the industry sadly

Nobody wants to admit just how bad the bot problem is, because it starts to dig into the fact that advertising isn't nearly as effective as advertisers let on

Especially after they committed textbook misconduct by trying to purge one of the paper authors because he worked for a competitor

The researchers apparently spend a year or so working on this, and it builds off significant previous work, so it seems like it was a pretty significant amount of work that OpenAI may have trained on

I'd love to see an in depth analysis of how much OpenAI actually did, but I suspect we'll never see that because it would indicate at least some plagiarism which undermines a lot of what OpenAI is putting out in public

Because the core of the issue is that it may well not have solved it, but instead plagiarised the significant step of the result from other researchers

That's why nobody's talking about how impressive this is, because its not nearly as impressive of a piece of work to simply cobble together other peoples' work that didn't know you were doing it. I could have republished relativity from einstein's notes, but people would correctly not be impressed with my ability

Until the plagiarism scandal is sorted out, its not a meaningful result at all, because nobody knows how much genuine innovation these models are displaying

The prompts they put in are the equivalent

The terms of service does not dictate what constitutes plagiarism

"The route to the Clay problem through a smooth force, options c and d in Fefferman’s statement of the problem, is the route Luis and Diego opened and the one Levent and I had quietly chosen to attack. Almost nobody else I know of was working on it. It is not the direction one arrives at in a few days by giving a model the problem statement. When I heard “forced,” it was a bright red flag."

We know that OpenAI trained on their prompts, plagiarism is incredibly likely. The only thing we don't know is whether or not it was deliberate plagiarism yet

OpenAI have admitted their new model they used was trained on prompts at around the time that researcher was working on it, so it seems self evident that it was used as part of the millennium solution

If I stole someone's private research notes and republished them loosely in my own words, they'd correctly be pissed

This was unpublished research that was stolen, and constitutes plagiarism and academic fraud by even the strictest definition

This is the literal opposite of progress: stealing from people genuinely creating, and stealing the money they should earn

We're having to rediscover in real time the extremely hard way, why enabling mass theft is so incredibly damaging to society. This is literally why we need a functional copyright system

If theft becomes more profitable than genuine creation, then nobody will create anything. Then there's nothing to steal, at which point all progress collapses

OpenAI trained on their private unpublished research notes effectively, while also trying to get one of the paper authors fired

That doesn't make it fine. We should not excuse this behaviour just because its rampant already, especially when it comes to such a serious prize

That does not make it ethical

This is textbook plagiarism, scooping their result knowing that the research was part of the training data

The biggest issue we aren't talking about is, of course, that those two researchers were not the only two using ChatGPT to work on the problem at the time

I just want to add to this another update by the author as well:

https://mastodon.social/@tristanbuckmaster/11723647135247030...

Which seems to be very directly accusing OpenAI of plagiarism