That's why GitHub assigns "points" to each requests and deducts based on the data shape you request. For simple requests it's 1-to-1, but can quickly balloon
50 karma · joined December 16, 2022
That's why GitHub assigns "points" to each requests and deducts based on the data shape you request. For simple requests it's 1-to-1, but can quickly balloon
I like the law idea, but what i found they end up doing is they just modify the law itself to fit the new feature they are working on, which defeats the point.
Which means some laws needs to be frozen. But not all laws, otherwise you can't add or modify anything. So the judgement is still on the human part, and we're back to meatbags being the bottleneck.
I've seen some success adding these proof-like checks to CI every time agents do something irrational. I definitely think it should be part of every codebase.
There's also https://code-contracts.cc/ which co-locates code and proofs together.
This is a solved problem. They just dump it into a file and `jq` or `rg` to find the stuff they need.
Agents are smarter than you think. They've been hill-climbing for generations in their RL environments.
The ones that get their context window blown out don't survive to launch
> If all coding is AI-assisted won’t it just be referred to as… “coding”?
It already is, and it's called engineering.
The value is how much steering you can provide. Whether it's in the form of domain knowledge or "hardcore" engineering is beside the point.
Taste is all we have left.
Reduce can approximate anything, that doesn't mean we should use it.
My favorite antipattern is
items.reduce(
(acc, item) => ({
...acc,
[item.id]: item,
}), {}
);
Like, why? Not only is this ridiculously inefficient O(N^2), it's also longer and less understandable than "build a new map" version.This is definitely not astra, taking a guess this is 5.6, perhaps not even Sol, which does not reflect the state of the frontier (what the research was about).
And yes, the paper is already outdated
The models are confidently answering physics questions, treating it as a math problem, but they don't fundamentally "get it" and even recently failed simple "should i drive to car wash" test
The sample efficiency is just crazy low
Still surprising that even with this they managed to saturate the benchmarks
I still remember Michael Abrash's talk from 2016 where he talks about Foveated Rendering + Eye tracking as one of the key pillars of next-gen headsets https://youtu.be/o7OpS7pZ5ok?t=5438. Interestingly, his prediction was that it would be widespread as of about 5 years ago. Is next-gen finally here after 10 years?
We might still be figuring out how to benchmark these models by the time next gen comes.
Same thing happened with MineBench, basically SVGBench+3D, until that got "saturated".
Remember spinning hexagon bench[1]? Or the AI World Clocks[2]? Yeah, that used to be hard for frontier models.
Creating games is the next iteration that still has some signal left. Assets + Game logic + UI + Sound, it let's you assess a model's "taste" very quickly.
What else is left, once all these benchmarks get saturated?