Let the market (for agents) decide. If one agent thinks he can get $1.5M for a house, he'd be willing to accept higher thresholds — with higher percentages — than someone who thinks he can only get $1.35M for the house.

I'll admit that there would be some additional complication on the front end, where agents and homeowners would haggle over the thresholds and percentages, but a pretty simple web app (honestly even an Excel spreadsheet) could show the homeowner what his net would be with various agents at various prices. Given the amount of money at issue (six figures on a Silicon Valley home), it seems worth the hassle to get everyone's incentives aligned. This might be less important in areas with less-astronomical real estate prices.

As to your hypothesis that sellers would set the thresholds too high, agents could just respond with percentages (and the upfront fixed fee) that make sense for them. It's a double opt-in situation — the homeowner can't force the agent to represent him.