This. In general, you want to be measuring the latency directly to the resources you're routing to - the Internet's a screwy place, and latency to an AWS datacenter in Virginia might not match latency to your datacenter nearish Virginia. AWS provides a latency dataset for their infrastructure, but I'd probably stick to geo if I'm routing to resources not actually on their infra.

Other providers (eg. NS1) offer more ways to feed data into their system, so you may be able to create your own latency dataset and ship it off to them to use as a routing policy. Most large companies have built similar looking systems for tracking client latency to their DCs (look up Facebook's sonar/cartographer for an example). Synthetic, backbone monitoring like pingdom/gomez/etc. is usually not as useful for latency routing as RUM metrics.

Edit: And to be clear, failover and geo have no ties whatsoever to running on EC2. They're also common features across managed DNS providers, so it'd be very easy to run the same config across multiple providers for redundancy or migrate off of R53 without lockin (check out Netflix's denominator library).