Hacker News Reader

Comment by kcexn | Hacker News Reader

Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing.

It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?

Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.

View on HN

The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves.

The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.

The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.

Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results.

The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.

They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day".

If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."

> but proving a group was non-sofic was out of reach

a colleague was telling me that the base idea for proving that something is not sofic already appeared in the literature around 2019 or so (this is the "expanders graphs" that are mentioned in OpenAI s paper. no one had managed to find a concrete example though. this doesn't make the result less impressive in any case.

The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.

I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing.

But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).

Circuit complexity lower bounds (and lower bounds in general) are notoriously difficult to come across.

For example, despite our best efforts, the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n). In contrast, our best algorithms for the task run in time roughly O(2^n). That’s an exponential gap. This is despite decades of trying to find lower bounds.

> the state of the art lower bounds on time complexity of algorithms for solving 3SAT is O(n)

Wow, that’s pretty stark.

“What’s the minimum time it would take to solve this problem?”

“Well, at the very least you’d have to read the input the whole way through”

> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing

Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.

I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either.

I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.

They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.

"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together.

Mundane incremental research is cobbled from existing citations that already appear nearby in the record.

Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.

So we can know pretty conclusively.

Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)

[1]: https://www.youtube.com/watch?v=guLDNMAOn24

Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.

I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.

They differ in that there hasn't been a solver you could have thrown them at. I guess you could argue their harness + LLM setup is a "solver", but the approach is so different from what we used that word for in the past that I don't think it would be appropriate.

You’ve received the expert answer several times. You just don’t seem to like the answer.

Forgive me for taking everything salesmen say with a grain of salt.

It’s a marketing. They are a sham company. If this article was by Scientific American or something it would be worth a lot more. They are literally trying to keep the hype train on track.

Also on HN front page today: AI's debt binge can't last, hidden borrowing reaches $1.65T (fortune.com)

https://news.ycombinator.com/item?id=49160699

It is marketing, they are a shady company, and yet, if someone had access to these results before today's modern AI tools, they could get tenure at any university in the world.

As others have said, it's hard to know how significant these results are without more transparency around the methods used to obtain them.

Not really. Why would that be the case?

If the prompt was crafted by an expert mathematician with pages of context and insights provided to the LLM to put it on the right path, that is a lot less impressive than a prompt that just says "find a counterexample to this conjecture".

Similarly, if they are spending millions of dollars in inference just churning on thousands of problems and these are the 10 solutions they came up with, that would be less impressive than if they chose these 10 problems specifically and were able to come up with solutions.

Reply on news.ycombinator.com