I see the problem as follows: Buyer/OP found and purchased the domain from a broker/escrow agent/middle-man, as a result the buyer was subjected to the additional time required for seller to transfer the domain to broker/escrow agent/middle-man.
Solution: Once the domain was transferred to broker/escrow agent/middle-man it appears the buyer had the domain a few hours later. Therefore, the broker/escrow agent/middle-man could require all domain listings be transferred to their account upfront. In theory after the payment, the buyer will have the domain within hours with no need to rely on the seller. Perhaps the only downside is the seller agreeing to an exclusive listing, but there is nothing prohibiting a system where the seller could cancel the sale and request the domain be return prior to any closing.