(I work on Route 53).
We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers.
In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider.
That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.