Without the insight into agent performance, any changes like this feel like a gamble to save $$ at the cost of developer productivity
I'm actually working on building generic repo-specific benchmarks at https://stet.sh ;)
In large user populations like at Databricks I think the ultimate answer will come from experimentation instead of offline evals. We are already doing this in small groups, exposing them to new candidate models and then measuring per-developer cost and perceived quality changes.
However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of the agent experience, such as interactivity and code maintainability. Meta and Scale have some evals in the interactivity space that seem pretty promising https://arxiv.org/abs/2606.29957 https://labs.scale.com/papers/swe-interact.
Human usability of the agent should definitely remain the top priority, but IMO we focus too much on the subjective, and not enough on the measurable/objective now, and I think/hope we'll see more companies move in the direction of Databricks