I've worked at an agency both on a fixed price and on a time and materials basis. They're equally bad in their own ways, IMO.
On a fixed price build, as you say, every small detail becomes a negotiating point.
I do think time and materials can be better, but I think the trust you mention can take a long time to earn. Some things seem simple but take longer than expected. Some things seem hard and come together quickly. It can sometimes feel for clients like they're being taken for a ride on a particular piece of work, even though in other cases the work will have been done for less than initially expected and it notionally balances out. These are tricky problems to get back on a case by case basis. And, truthfully, in my experience the clients who were fussy about costs were usually fussy about quality too, and I would say generally came out ahead because of it. So the incentives are still tricky.
It's also challenging, because in an agency context the client inconsistencies do exist. Some people do work faster or slower. Some people do better or worse work. Sometimes the person who is already up to speed on the project isn't available and someone else has to do the work instead. A lot of these problems are internal to the agency, and it's frustrating to the client when they affect their expenditure.
I don't really have a good solution to it. Honestly, my takeaway from the experience was that the client/agency dynamic just kinda sucks, and it isn't a dynamic I enjoy working in.