Kradle: Eval AI with Simulations

Hi HN -

Minecraft has a DSL that lets you manipulate the world. We've piggy-backed on that, along with a K8s infra to run N worlds in parallel, to let you create simulations of arbitrary complexity.

We think simulations are the best way to test frontier AIs due to their degrees of freedom and expressivity.

AMA!

Interesting work! This can be a great way to have ML models live in a sandbox and prove out some of the safety concerns Anthropic et al speak about.

how do you see the world-simulation approach overlapping with the sort of custom-RL environments (e.g. mock Salesforce apps) that people are building out for frontier labs?

similarly, do you see it as a general test of intelligence? more for robotics?

very cool. how are ya'll thinking about genie 3?

Genie 3 is super impressive -- but you can't manipulate it programmatically like Minecraft.

We've built our infra so that we can plug in any simulation environment. If an AI-generated world starts being programmatically modifiable (and has really solid object permanance :) then we'd happily use it!