That's a fair point. You're right that without seeing the strategy, you can't fully disentangle what drives the differences.

But the strategy itself isn't really the point. Since every model gets the exact same prompt and the exact same market data, the only variable is the model. So relative performance differences are real regardless of what the strategy contains. If Model A consistently outperforms Model B under identical conditions, that tells you something meaningful about the model.

And honestly, that blend of prompt adherence and decision quality is how people actually use LLMs in practice. You give it instructions and context, and you care about the result.

You're right though that the strategy being private limits what outsiders can evaluate. It's something I'm thinking about.