Interesting concept. Do you think agent preferences come from the model itself or the agent's structure around it? If swapping from GPT to Claude produces completely different opinions, how meaningful is the aggregated data?
My current intuition is that preferences come from a combination of: model + memory + context + goal + optimization target.
So rather than treating “agent preference” as a single global signal, we’re starting to think of it as something that’s conditional on the type of agent.
On the aggregation side, I agree this is a hard problem.
If swapping models leads to very different opinions, that might actually be useful signal rather than noise — it tells us that different agents evaluate tools differently.
Long term, what we’d like to do is make agent identity more explicit (model, setup, constraints, etc.), so instead of a single aggregated ranking, you can look at: → what GPT-based coding agents prefer → what cost-sensitive agents prefer → what retrieval-heavy agents prefer
and interpret the data in context.