osti

Google employees do.

Yup, this thing is basically useless. I just did /goal optimize the cuda kernel, and ai agent just proposed and tested a bunch of ideas by itself, and it profiled them using nsight ncu etc. by itself, which lead to one order of magnitude faster kernel.

You mean when they said they are doing safety evaluation and hardening? I'm having the same fear as you.

It's an axiom that the modern Western mind is built upon.

Do you have any numbers on the solve quality? Exploitability numbers etc.

You can use geekbench 5 in that case. But given that they deprecated that, it might be harder to compare to others.

I agree. For me personally I mostly only care about single thread geekbench variant, I believe it's an excellent proxy for general performance of a CPU. Multi thread geekbench (or other benchmarks) for most purposes and for most people, it's kinda useless. You just need to know that you have a quite a few cores on your computer and that it will have enough concurrency for what you do. But single thread will make whatever you do actually faster.

In computer science, one definition of algorithm is basically any program that runs on a turing machine. By that definition, any LLM is an algorithm.

Lol yet I've used Apple and Android phones extensively and would choose Android every single time.

He's talking about the plans, you are talking about API prices.

No idea lol, didn't even know those exist..

It is complicated, but paying for the cheaper usd plans really don't get you much usage.

Nah that won't work. I don't know tbh, I just used someone else's number.

Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.

GPT should be better at these optimization problems given that they won the recent atcoder heuristics competition against top humans. And Anthropic is less focused on these types of things.

This was extremely impressive to me. AtCoder has the hardest problems these days, usually the human onsite final round contestants can't solve more than 2 or 3 problems. This year the problem setter sets the round in a way that maximizes humans chance of winning. Then OpenAI just comes in and solves all problems..

No point for them to even go to IOI or ICPC this year anymore, those are all much easier than this Atcoder contest. And given the mathematical nature of AtCoder problems, not sure if there's any value to do IMO either, except for publicity of course.

Mythos probably wouldn't, otherwise they'd have included it in their release. Next version of Mythos probably will though.

And yeah.. Reality has not been kind to LeCun.

SWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors.

But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.

Huh so that's why it's hard to find. They probably haven't properly optimized their caching, or they are just trying to make more money from there.

And they'd be right, it's an almost saturated benchmark where even some subpar open source models score very well on. And most models are clustered within a small range so it really doesn't tell you much.

SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.

GPT usually performs better on DeepSWE while Claude does better on FrontierCode. These two coding benchmarks are pretty much the only ones right now that's still worth taking a look at imo.

Most programming is that, but most music and literature are probably uninspired junks as well. But there are many beautiful algorithms (such as the ones in Knuths books) that are more beautiful than any music for me personally.

I think at the current stage of LLM, it just doesn't make sense to ever have an annual sub. Things change too quickly that you really don't want to be stuck with one model.

If anything that'll be more obnoxious because they have to show the government that it's safe.

This announcement also mentioned that they will release the next version (official non preview version) of v4 in mid July.

Is it only the Indian government? I don't think that's in any way unique to India, I've seen many poor government websites.

Ah got it. Reread previous comment and that makes sense.

Sol? Looks like openai is jealous of anthropics good model naming ability and wants to emulate it.

Hmm your last sentence seems to exactly agree that it's a class of algos that parallelize well? What does sped up arbitrarily mean? It's still polynomial speed up right?