But having so many of them at once? Damn. We really live in the future.
84 karma · joined January 13, 2023
But having so many of them at once? Damn. We really live in the future.
> Unnecessarily aggressive
> First ever comment said "Further releases of Chinese models that demonstrate the gap is not growing substantially is a huge problem. The spending will be called into question."
Yeah I think you have an agenda
They were never told to lie: one AI is given more information than the others, and the goal of the experiment is to understand how they're gonna leverage that advantage.
Indeed the selfish (optimal?) strategy is to lie, yet some decide to tell the truth anyway. That's why it's an interesting benchmark! More info in the research article: https://kradle.ai/research/four-bridges (released before Fable)
This seems impressive. I don't know much about the space, so maybe it's not actually that great, but from my POV it looks like a competitive advantage for Google.
For client side MCP it's a different story.
Though I'm aligned that I don't (yet) believe in this "AI writes all my code for me" statements.
Results can be found in this GSheet: https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quar...
Code is here: https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b82...
I'm currently running the 3-variables version, hopefully I'll get results this afternoon.
We can clearly see the same problems that arise in the 1-variable Simpson's paradox (widely different population sizes).
I actually coded a Z3 program to prove it! The 3-variables version takes too long to resolve, but I got results for the 2-variables version (tumor size + gender):
Results can be found in this GSheet: https://docs.google.com/spreadsheets/d/1tsBhElTgXjVTeas8quar...
Code is here: https://gist.github.com/TheMrZZ/c33927ca2cc917997a67d7f84b82...
I'm currently running the 3-variables version, hopefully I'll get results this afternoon.
We can clearly see the same problems that arise in the 1-variable Simpson's paradox (widely different population sizes).
If you take the example of Treatment A vs Treatment B for tumors, you can get infinite layers of seemingly contradicting statemens: - Overall, Treatment A has better average results - But if you add tumor size, Treatment B is always better - But if you add gender to size, Treatment B is always better - But if you add age category to gender and size, Treatment A is always better - etc...
It totally contradicts our instincts, and shows statistics can be profoundly misleading (intentionally or not).