hodgehog11

By category, I meant other language models in general. The point of others putting something up to beat Jev is to show that, to date, no one has bothered to produce something like Jev, because anyone with decent LLM experience can roll their own for purpose with little effort and have been doing so for years. And can beat it on any metric you choose.

Let me put it this way. OpenAI and Anthropic have a slight moat over the Chinese labs because they have strong training data and the most advanced RL strategies. It will take the Chinese labs significant R&D effort to bridge that, especially in math (and there is a good chance they will, provided they want to).

Jev has no moat other than the fact that no one else has bothered to package a model in this way. Another lab could build a strong competitor very quickly if they want to put the effort in. That's the point of this post. There is no uncertainty about what they have done, nothing to figure out. Someone just needs to do it. I'm not sure what to say if you can't see the difference between the two. Jev is worth celebrating because of the idea to package it in this way. But it is not a paradigm shift and that is likely a problem for them.

Architecturally, it is trivial. That's something the community would have consensus on, so not a hot take.

I see your point, but Jev doesn't exist in a vacuum. When one (like me) says "trivial", they mean it relative to other attempts and developments in the field, all of which require everything you've mentioned at minimum. Commercialising any product, and doing it well, is hard. But the R&D factor here is substantially more straightforward than almost any other product in its category, because there is no architectural breakthrough here.

An LLM is a zero-shot classifier with a large number of classes. All you need to do is establish what the output means and you can fine-tune an LLM final layer for this task if you like (and others have done). A student of mine did this as an exercise two years ago, and it was cool, but not publishable.

I agree with you on the "ease of use" business though. No one thought to make this sort of thing commercially available.

But there is no hot take here. Jev is not some new paradigm; engineering-wise, it is a trivial modification to the existing pipeline. That doesn't mean it isn't commercially viable.

A group already (sort-of) has for the broader theory: https://arxiv.org/abs/2604.21691

It's a good list, but "alignment" is much more targetted. I attended a workshop with researchers from OpenAI and Anthropic also present, where the objective was to hash out what the problems in alignment even are. This, it turned out, was extraordinarily difficult.

The problems with alignment are to do with how we define and search for this vague notion when it is mathematically ill-posed at present.

This honestly sounds like holy grail tech at this point; what kind of battery life are you looking at?

I don't understand what you mean. Test error is literally out of sample error. Marginal likelihood is designed to estimate out of sample error. The whole discussion is about out of sample; nothing has been about in-sample error. The in-sample error for my examples are all trivially zero, so only out of sample error is worth discussing.

Ah, sorry for the misunderstanding, I can see how my comment reads that way. That is done in the Gaussian process context, not in my first example, and yes, it's a dirty idea, but you can justify it using differential privacy arguments (basically you are optimizing few parameters and these do not have full interaction with the data).

Just to add on top of the quality reference provided by srean, I like to first drill in Bayesian principles and then use this article to derive PAC-Bayes from that: https://arxiv.org/abs/1605.08636

Regular PAC falls out by taking a uniform prior over a finite hypothesis class (and then building up VC dimension if desired, but usually by this point you realise why the bounds are unlikely to be good).

No, I am talking about out of sample error and estimates thereof. It is "overfitting" to data, but it also has lower out of sample error than the case where you do not "overfit".

This is why the notion of overfitting is not nearly as cut and dry as a basic ML course would have you believe. Just because you fit data exactly does not mean that your estimator has high error on out of sample data. A trivial counterexample is a spiking model that spikes to fit to the data but otherwise follows the correct trend outside of the dataset. The bias variance tradeoff gets thrown out at enormous scale and overfitting is not a meaningful concept. What matters is regularization and robustness, not how well you fit the data.

The reason why bias variance tradeoff and considerations of model size are a good approximation for smaller models is due to concentration of measure in the data which effectively kills any regularization in your modelling procedure. Once you enter settings where concentration of measure begins to bite in parameter space, everything changes. This isn't really that mysterious; any textbook on Gaussian processes (e.g. Rasmussen and Williams) will tell you this.

I agree that this is a good nuanced take. However, I find that students who have learned PAC (which usually takes quite some time) often have to unlearn certain principles to do PAC-Bayes, so my comments come from a fair amount of frustration with the topic. On the other hand, I find that teaching students PAC-Bayes from the get-go is easier, it still works for simpler models (you can derive the same PAC bounds, so you can't do any worse), and they get the full story. Obviously for those of us who learned both, it is good to know both. But I am skeptical for the next generation coming through whether it is worth teaching PAC at all.

You can choose the prior according to any selection rule that does not see the data (actually, you can do more, but justifying this is the realm of empirical Bayes and requires some more precise arguments). In this case, you can choose it according to the model size and provided that your Jacobian is full rank, you will get increasing marginal likelihood.

Apologies, I'm skipping details, because that's how I speak with my colleagues, but I realize this is an external environment without context. No references since this is folklore (you can look at Hastie et al's Surprises in High-Dimensional Ridgeless Regression paper for the non-Bayesian version, Bruno Loureiro or Andrew Gordon Wilson probably have a paper with something similar).

Concentrating a density around a zero set means that I raise it to the power of 1/gamma (appropriately normalizing) and then take gamma to zero. If the likelihood was Gaussian, this would be equivalent to taking the variance to zero (yielding a point mass). But in overparameterized settings, this concentrates on a submanifold describing the set of interpolating solutions. In least-squares linear regression, that is the solution space. Reducing the variance on a Gaussian prior is treated as an asymptotic expansion by Laplace's method. If you choose the variance to decrease (inversely proportional to the parameter size, for example), then the marginal likelihood will increase monotonically with model size.

By any measure of model size, I mean that you can pick your favourite among the common ones, such as information metrics (e.g. mutual information / KL), statistical metrics (e.g. marginal likelihood), test error. You should be able to show the same phenomenon happening for all of them, so it isn't a quirk of marginal likelihood. It is concentration of measure working in your favor to reduce the variance in the estimator.

Absolutely not. This link is a reference on PAC learning, which is thoroughly misleading in the land of deep learning and inevitably leads to vacuous bounds. This is common knowledge in deep learning. I would not recommend that any student learn any part of this theory at this point, since we have far better alternatives in terms of simplicity, accuracy, and generality.

PAC-Bayes is genuinely superior, for example. Instead of a uniform weighting over all hypotheses (effectively encoded in the supremum), you get to weight the hypothesis class in virtually any way that you want. This is critical to ensure that you exclude absurd hypotheses that you have almost nil chance of reaching. If you do consider a uniform weighting, then you can just easily reduce to PAC anyway, but you do so in a cleaner package.

This is provably not true, and you can use the marginal likelihood / PAC-Bayes to prove it (or any other framework for measuring model quality). Increase the number of parameters in a linear model way beyond the point of interpolation, and concentrate the likelihood around the zero loss set. Then reduce the variance on a Gaussian prior. You can balance the two temperatures at exactly the right rate so that any measure of model quality will monotonically increase with model size and achieve a maximum at infinite model size.

Even easier, just take a limit of polynomial regression to a Gaussian process while optimizing the marginal likelihood over the prior temperature.

In all of these cases, the model with the least parameters is not the simplest in principle and does not have the best chance of not overfitting. The reality is significantly more nuanced.

This is an insane thing to read. Bubeck had a reputation even before he started with OpenAI. Of course it was him that was involved in this drama.

This is such a sad mess, and it really didn't have to be this way.

It's literally a toggle in the options for ChatGPT, one which is on by default and most researchers probably have on without realising it.

So to say that it is unlikely is extremely suspicious. No, they did not literally pull user data. But user data is automatically added to their training set by default, so their latest in-house model would be trained on it if it is from several months ago. It isn't intentional on their part, and they probably realised they could not refute that they trained on Tristan's logs unintentionally, hence why they acted the way they did.

Yes, mathematicians are. And yes, most of my colleagues did not even know the opt-out was an option.

Touche, I used to work in pure probability where that ridiculous Hardy-Littlewood rule used to cause all sorts of problems, but now work in statistics, where it is no longer an issue.

To be clear, the colleagues I am referring to mostly work in math phys. They are sole author papers, but I refer to them as first author, since that is the language I am now accustomed to.

And no, the culture in maths vs. theoretical stats is not really that different at the end of the day, and the latter is most assuredly like other STEM fields. I still collaborate on pure math papers (geometric analysis and PDEs mostly) from time to time, and it isn't really a different head space. My name is typically later in the alphabet, so I never even think about the alphabetical ordering.

Uh, care to explain? I have several colleagues that stopped submitting to journals once they reached full professor. They only submit papers from their students for the benefit of their careers. First-author papers, not so much.

Exactly, and the advantage is that checking that the problem is "formalized" here is essentially isolated to verifying that the final theorem statement matches the claim. If there are no 'sorry's and the program compiles, then it has been proven. That's the point of Lean.

You can prove that doing this will spiral training into a fixed point. There was a lot of research into getting this to work in the past, but it never truly worked well. The hope was that if RLVR was used quite a bit, and the general performance crossed some threshold, that it would then be possible. However, since it has been shown that RLVR only concentrates the distribution of outputs rather than truly shift it, I doubt this will ever be a viable strategy.

Why would they send it out for "expert review"? Every time, they have just made the AI generate a Lean proof. In fact, it seems like the most plausible direction to NS is computationally assisted detection of a blowup solution, which has fantastic automatic validation.

It basically is a formality at this level. Many top math researchers now hardly even submit to journals at all and just put up a preprint.

At this scale, peer review happens by the audience. They don't need a journal to get people reviewing their work.

I also believe this. Post-training LLMs with vague metrics can only be achieved with RLHF, which is not impossible, but extremely costly and difficult. Instead, companies will opt for RLVR, focusing on math and programming tasks. This pushes objectives away from writing quality; often far away. That is why older models, in my view, actually read better than newer ones. It's by design.

I agree that this should be something that researchers reflect on. GPT-2 is one of the primary models to research on nowadays, and many recent developments have come from studying it as a test bench.

Imagine if CRISPR was considered "too dangerous" to publish because of the potential ethical ramifications, and that only a special few should be aware. It is utter self-righteousness, and it is shameful behaviour. The world cannot adjust itself to what it cannot see, so you risk greater catastrophe by keeping it secret.

The open dissemination of knowledge at every increment is the only way for society to truly deal with what is to come.

It's good to see validated numerical proofs seeing a resurgence now that they are substantially easier to achieve.

Others might be able to chime in, but my experience is that AI is effectively taking proofs that were once iterative (pushing current ideas further, tightening arguments) but considered time-consuming, and turning it into very low-hanging fruit. So stuff like this, as well as the new lower bound on the asymptotic ratio of zeros on the critical line, really aren't that impressive anymore. But it is likely necessary to do nonetheless, just as low-hanging fruit has always been.

The real frontier remains (even if many were not operating there) the development of new definitions and ideas that push right past the walls that were previously there. AI still seems to be shockingly poor at doing this, and you can feel that when you use it for proper hard problems.

I hope you understand the context in which that was said. The point of that statement is that the only way to rigorously verify correctness of a program is by using formal methods. Those are often too difficult to achieve by humans, which is why there is an entire program of developing autoformalization using LLMs. You are supporting what I have said.

In practice, no one rigorously "proves" that their program works at present, and well-written tests do suffice. The definition of "well-written" here is circular, granted, but there isn't really an alternative. Even strong programmers should be writing high-quality testing suites.

No it really is about the test suite, and provably so. As another poster pointed out, speed is a superoptimization problem and the test suite provides the constraints. If the constraints are appropriately set, even a naive genetic algorithm will eventually improve the outcome over time, provided suitable mixing of the proposal scheme. LLMs provide measurably better proposals than naive approaches, so the entire chain is sound. The issue really is an inability to set appropriate constraints on what the user is looking for, and poor quantification of the multiple objectives one should try to balance in practice. What's great is that's a human problem. Diverting that to the models is obviously a disaster.

I agree that there has been a glut of subpar developers for years, and that has lowered the bar significantly. This is mostly because core values shifted. So let's keep our eyes on what really matters rather than acting elitist.

The Chinese labs are picking up on the low hanging fruits on efficiency, and no, you do not need to abandon transformers, you just need to push them closer to the more computationally efficient architectures of the past. OpenAI and Google seem to be trying a few things too.

Anthropic clearly are not though, and to call their operations wasteful is an understatement.

> The game often has its own DRM though which will stop you

I think you missed the "know where to look" part. It's called a Steam emulator, for starters. Note that I speak about this strictly for preservation purposes, as people who raise Steam DRM as an obstacle to preservation have not done their homework.