Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.
But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.