6,586 karma · joined March 13, 2013
Dumbed down quantization?
No. Full intended inference weights preserved, so far so good.
Slow performance?
No again. Looks like you could get over 150 tokens/second.
Give up context window size?
Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this range, and quality does start to drop off toward the full size.
When they just loaded the weights alone, it was taking 156GB in vLLM. After warm-up and adding a KV cache pool, it took over 200GB.
And this implementation is already cutting down the 1M token context window you would normally get.
All else being equal passing triple the amount of tokens through a model to solve the same problem makes it slower.
Doesn’t mean this model is bad, and it has to be considered how cheap it is, but it’s a factor.
It’s also so inefficient, when they release the full performance numbers it’s not going to be good.
One example, it takes about 3.6x more tokens to finish the same work as Gemini Flash 3.6.
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
Did it really cost $1 million instead of the 150k that’s been floating around?
If you don’t like the price now just give it some time.
The point is that these kind of heroics can actually work now and they’re only going to get cheaper.
I don’t mean to sound negative about what you wrote, it was an interesting read.
Google watches over the last few months a flat out assault on the Pareto curve from American and Chinese companies. Release after release pushing the boundaries of frontier intelligence and price/performance.
And the response from arguably the biggest AI research labs in the world by headcount is Flash 3.6.
What do you do when you are given essentially unlimited resources and still find yourself falling behind?
Other papers have tried to beat it with temperature and it didn’t work, although I haven’t seen anyone try insane levels.
Give it a shot and let me know if you have any success.
But yeah, in general I think you’re right, the actual utility is pretty niche.
The comment I was responding to, about changing LLM output, referred to prompting, not temp/sampling tricks. I’m not aware of Pangram being beat by clever prompting. There’s some interesting work on creative writing using contrastive prompt techniques, but I haven’t seen it tried as evasion.
Even if you control temp and sampling, they’re not magic. If you raise the temperature too much writing can go to hell, so you may beat the detector but end up with junk. There are some ways to mitigate such a quality drop like raising temperature in conjunction with min-p, but still, I haven’t read any research that shows it getting good results at anything close to 10.
Now you want to get more clever and manipulate logprobs…well ok, you could come up with elaborate strategies designed to evade specific detection methods. But I don’t see that getting done as a weekend project while maintaining writing quality. And if it does happen there’s no guarantee the detector can’t train on its characteristics and start an arms race.
I’d want detectors to be as accurate as possible, false positives of 1 in 10000 seems like a good starting point. I believe their results have been independently tested.
And as a separate matter, any tool for evaluating students should be applied fairly, safely, and with adequate human review and due process.
You need good tools and good oversight.
Not really. The false positives for the SOTA detector are very very low.
"It's also very easy to change the pattern of LLM output."
Not in a way that can reliably avoid detection. The problem is the patterns are baked into the distribution itself. It's smoothed over, so it becomes difficult to prompt your way out of that.
Not true at all. Pangram is highly effective and has a very low false positive rate.
The post here is impressive for a small project, it looks like they independently thought of one of the core ideas Pangram uses of creating twins to compare.
You can see how it works here: https://arxiv.org/pdf/2402.14873
I used to think that was a personal limitation, until I saw an interview with Bjarne explaining that he used to understand all of it but at this point it’s too big, no one can anymore.
As for the builds, your agent probably already knows how to do a lot of this from the command line, although explicit suggestions can help it build faster for different situations.
As for XcodeGen, you may find it unnecessary overhead if you're already using Xcode file system synced groups.
For iOS my biggest suggestion would be to enable App Store Connect skills for your agent (https://github.com/rorkai/App-Store-Connect-CLI).
With this not only do you not need Xcode all the time, you also don't even need to be near a MacBook.
Just make changes via Codex on your iPhone, the tell asc to build and upload to TestFlight, download and run the new version, iterate.
It might be a better mathematician than most humans at this point. Kind of like when chess software started beating everyone except grandmasters.
What’s left? Proposing and building out entirely new theories and frameworks? Then better than any human? Then alien math results we struggle to comprehend?
It’s amazing how much work you can get done on your phone now, especially if you already have a design mapped out in your head.
Otherwise, it seems like preferences could be competing.
For example I asked
“Why should LLM attention use dot product instead of cosine similarity, being that we often hear vector magnitude does not encode most of the useful information needed”?
The voice response was directionally right but lacked detail and was a little hand wavy.
The answer to the same question in a text chat was much higher quality.
The voice response replied “let me think about that…” so it appears to be invoking 5.5 as advertised, but it’s definitely weaker.
I had reasoning set the same for both.
It’s advertised 7MB, but also comes with a 5MB mini version.
Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility.
It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.
I’ve had good results with Tavily so far, might be worth checking as an alternative for agent search.
“expert developers whose skills have reached the point where they outclass any and all “frontier AI models” in their area of expertise”
Are any developers saying they outclass any and all frontier models? I’d say at best it’s mixed at this point. The best developers still do certain things better, but not even close to all things.
“The problem is that even code written and/or reviewed by Fable 5, will stink”
I’m skeptical. Example prompt and output please.
The problem is being more auditable does not automatically make it more audited.
There have to be enough people with skill taking enough time to work on it.
Do they really correspond roughly? Seems like they’re trying to suggest a discount while still being worth a significant amount of monthly spend.