1) Many drivers from your town would likely geofence themselves to your town, reducing the supply of drivers that will drive to the metro area, increasing wait times and driving up prices, making riders unhappy.
2) Drivers would use this feature to avoid driving to "bad" parts of town, making transportation discriminatory. (In ways that are illegal for traditional taxi companies, so Uber would eventually run afoul of the same regulations.)
It's a shame because there are reasonable use cases for a geofencing feature (you seem to certainly be one of them), but it can be easily abused and can hurt the rider experience depending on locale. (Not saying the rider experience is inherently more important, but certainly Uber/Lyft would see it that way.)