>I suppose Uber could validate the GPS position of the driver and my phone and use that to validate my story, but I doubt they went through all that trouble :(

For that, I assume they would need to have tools built for that specific purpose, with security/audit in place. I doubt any support guy could just randomly query for GPS data for drivers/users.