My grandma is 90 and drives 5 miles to the grocery store, a slow road. I don't think she'd pass a driving test but she drives during the day when barely anyone is on the road, chances of serious injury are nil.
Is it worth it to spend large amounts of money on testing these people, taking their license away if they fail? Getting rid of their car will force them to replace it with someone else driving or cycling which could be a problem in many places. Worst case scenario they'll need to go in a retirement home.