Maybe for a query or two, but the problem sorts itself out.

Auth servers are very rarely running from a completely cold cache. Additionally most resolver software uses a system called scoreboarding where response times for each IP are recorded and it will prefer the fastest known nameserver for a query (some number of queries are still periodically sent to "slower" or unknown servers to gather metrics for them).

Clients never treat a system resolver as "broken." Most operating systems will round-robin the list of available recursive servers when there is a timeout before returning failure to the software. In the case of web browsers and such, they'll retry with a resolver operated by the browser vendor before admitting defeat.