this bench was just released, it couldn't have detected opus 4.6 degradation.
380 karma · joined February 1, 2022
this bench was just released, it couldn't have detected opus 4.6 degradation.
And it just BURNS tokens like crazy.
Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.
EDIT: I was using gemini cli... it's not a harness issue lol
make an x account then?
You can read it yourself - https://blog.codeberg.org/protecting-our-floss-commons-from-...
Free money, right?
https://www.pangram.com/history/0b397596-fa85-4869-bae3-2b1d...
Where did you get the idea that it takes a ton of money to do it?
I am confused.
More news at 7.