Uber Using Driver Phones as a Backup Datacenter | Hacker News Reader

Uber Using Driver Phones as a Backup Datacenter

highscalability.com

165 pointsantman123 comments

Off topics but the fact that Uber requires you to have the latest version of its app to book rides is ridiculous. Not everyone has the newest smartphone with tons of space to update apps.

I actually got stuck out at Fort Mason one day during the Launch Conference last spring because of this issue. I used an Uber to get there, but when I went to leave the app required an update and I couldn't get a decent enough connection to download the update near the venue.

They really should let you use the N-1 version at the very least. That should prevent these same day/trip issues.

I have a feeling they probably do.

I hadn't updated in a week or two, just out of laziness to not check for updates everyday. So it's possible I had missed more than one update that caused the issue.

I heard the force-upgrade window is 3 months. To verify just deny updates until the app doesn't let you use the uber app anymore. It might also be necessary fixes pushing up the minimum version.

It perhaps allow you to book 1 ride on an outdated version and warn you its the last one. Then lock account until you've updated so you don't get stranded like that 1 person

They should just use a decent website (web-app), in my humble opinion.

They have that: https://m.uber.com/

Ironically, this sounds like a customer experience that's way worse than anything described in the article. Especially since this is hardly advertised to users.

Parent got downvoted, but I think it's a legitimate concern: refusing people rides because for whatever reason (old phone, poor / expensive Internet connectivity) they can't update the app is problematic.

It would be neat if they gave you a short list of temporary codes in the app that doesn't rely on connectivity that you could punch in when you call into an automated phone system.

They just need a freaking "Round trip button"!!

All it needs to do is take you back to where you started from based on where it last dropped you off.

You can use their mobile site instead: https://m.uber.com

That also has the advantage of not requiring invasive app permissions.

The last few times I tried to use the mobile site, it gave an error message claiming that requesting a car via the site was not authorized (it had worked for a few months before that).

Send their support a mail or tweet. See the thread at https://twitter.com/josh_triplett/status/587614288686096384 .

What? So every app update is a breaking change? That's awful software engineering practice.

Agreed. I'm gonna hazard a guess and say that they do it in order to avoid the cost of supporting older versions of their internal APIs. Interesting contrast to Facebook's engineering team, who maintains mobile apps for at least two years after their release:

Backwards Compatible: In a world of deployed native mobile applications with no forced upgrades, backwards compatibility is a challenge. Facebook, for example, releases apps on a two week fixed cycle and pledges to maintain those apps for at least two years. This means there are at a minimum 52 versions of our clients per platform querying our servers at any given time. Client-specified queries simplifies managing our backwards compatibility guarantees.

via https://facebook.github.io/react/blog/2015/05/01/graphql-int...

I didn't know they had such a liberal backwards compatibility policy until I saw this GraphQL talk, which is a pretty interesting overview of why / how they do it: https://www.youtube.com/watch?v=WQLzZf34FJ8.

Makes me wonder how much money / how many users Uber loses by not supporting older versions.

At work we (like Uber) own the servers and the clients, and even then it's very time-consuming to support backwards compatibility. Occasionally, we all but have to require an update. Ours don't leave you stranded, so it's not apples-to-apples here, but it's a thing.

Unfortunately, this could be as much Apple's fault as Uber's.

Only if Apple changes their requirements, like disallowing access to the UDID, but even then you have plenty of time (and such changes are requirements on future versions) so it's very doubtful Apple has anything to do with this.

Could you explain how so?

iOS has a history of breaking the API with version updates. Sometimes it's such a PITA, large companies choose to remove their apps instead of upgrading them:

https://toucharcade.com/2015/09/23/bioshock-removal-2k-suppo...

I think if you're a driver, an up to date phone is a business expense. The latest version of the app doesn't require the latest phone - just enough space.

Contract employees are supposed to be able to determine the tools they will use to complete their tasks, by law.

As part of the independent contractor guidelines, they have to supply the tool (versus the employer buying it for them).

But tools and equipment are commonly specified (even down to a specific tool model for insurance requirements or material brand for warranty requirements) as part of contracted labor.

If the job requires certain certain strength tools, you need a good enough tool to complete it. Like digging soft dirt tools vs breaking up hard stone tools to dig a hole somewhere...

And they are. Android versus iPhone, Nexus versus Galaxy, etc.

If I hire someone to cut down a tree on contract, I can't dictate their tool of choice - only that it complies with what's necessary to complete the work.

There are plenty of fair criticisms of Uber. This isn't one.

>Not everyone has the newest smartphone

The latest version of the app supports Android 4.0.3 and up, as well as iOS 7 and up.

>with tons of space

The iOS app is 59MB. In 2015, you're claiming it's unreasonable to ask for a few megabytes? What? 8GB flash drives were party favors 5 years ago.

8gb flash drives might be party favours, but as phones no longer let you stick a memory card in and the latest greatest handsets (eg: nexus 5x) are still shipping with 16 or 32gb of storage, the 50mb isn't insignificant.

Even if the OS uses 8GB of space you are talking about well under 1% of the leftover space. Sounds insignificant for something as important as transportation.

My phone is 2,5 years old and still works very well. Just has tons of bloatware (damn you HTC!). Might have to root the phone to uninstall that junk.

My friend had to update his otherwise-fine iPhone because of Uber's disdain for backward compatibility. Admittedly it was an older model phone. But everything he cared about worked fine with it until Uber pushed a mandatory new version of the app that required a newer iOS than the phone could run. To Uber's credit they did have a useful support response with a pointer to the mobile website. Still seemed awfully awkward.

Interesting requirement. I work on a large app with a large install base that is not just early adopters: we use 6 month minimum window to phase out services.

Used to be you could actually book a trip using SMS -- I did it that way for almost a year before I ever used the app, and then they phased that out.

Or maybe they are using it as a screening device for "better" drivers. I keep hearing that when Uber comes to a new city, the pay is pretty good so it attracts non-ESL speaking drivers. Maybe this is just a barrier to enter the marketplace to maintain those 'better' drivers. Probably not, but I wouldn't be surprised.

That's the elephant no one is talking about Uber is obviously going to be attractive to those in the black economy.

I wonder if Uber realizes in the uk they will be on the hook if they employ those with out the right to work.

So if you hire a guy on the road to help you move, are you responsible to verify his immigration status?

I think these ride sharing apps also report giving you money as required by law, and you have to put in such info as an SSN in the USA so they can do background checks. I'm pretty sure they also do background checks in the UK and part of that will be your immigration status. It's just they are efficient and automated in doing so, so it seems like they don't check anything because of the speed.

Yes and in the UK its strict liability there isn't a defence of good faith if you accept fake id's even if its a very good fake your liable for I think a £10K fine

does that apply to contractors?

To the employers and I suspect that HRMC will class Uber drivers as employees as they did for Contractors in IT

Or they could just pay their good employees better and then fire the poor ones.

P.S. I know that they claim that their drivers are not employees but if it walks like an employee and talks like an employee it's an employee.

Please pay me per hour or give me a salary, I must be missing it. Oh wait, drivers are paid per ride, can start and stop work at any time, and can work other apps simultaneously.

What job lets you work with competitors while you are working?

Please pay me per hour or give me a salary, I must be missing it. Oh wait, drivers are paid per ride, can start and stop work at any time, and can work

I made this point upthread but contractors are free to use whatever tools they like to accomplish their tasks, by law.

Full-time employees are subject to using specified equipment by the employer.

Not exactly how it works.

Contractors can choose but it doesn't mean they can choose tools that don't do the job. If part f the job is storing backup data on a phone the tool needs to be a phone capable of storing backup data.

The biggest argument I've seen for uber drivers being contractors is that they can work for competitors at the same time. Any employee would get fired pretty quickly.

Only if it gets paid like an employee.

Its huge. Takes a couple minutes to start on my Android phone. Having Spotify in there is the very definition of bloatware.

Where can I verify this?

It's very hard to maintain compatibility for different versions of mobile operating systems.

API's, etc. change or break completely all the time. Often it is simply a trade-off between acquiring new, perhaps more affluent users, and loosing those with less money to spend and cheaper phones.

The morality behind this is questionable off course.

Really? "It's very hard"? That's the explanation for such poor software engineering practices? I guess their engineers are only paid to solve easy problems.

Engineers are paid to solve problems that help the company make money.

If they ran the numbers and the cost was greater than the reward why would they both maintaining backward compatibility?

Uber is not a software company. The only code they hack is the legal code.

The good software engineering practives of yesterday are for old people. Young people are just smarter. They move fast and break stuff. Makin' omelettes, man. Shhhyeah.

Good software engineering practices of yesterday were sometimes dumped for good reason.

oh but young people are idiots. That's right. Back to waterfall guys.I'll stand up the new CVS server.

And then you have people like in this comment.

https://news.ycombinator.com/item?id=10446977

> It's very hard to maintain compatibility for different versions of mobile operating systems.

Except it's not. It requires thought, but the reality is, they aren't trying to maintain compatibility, they are merely saying "Nope, you have to have the latest and greatest or it won't work" regardless of whether it would work or not.

If anything, this demonstrates that Uber itself is not concerned with quality. Considering my experiences with Uber in the past, this is not surprising.

Quality is many faceted. One aspect that's relevant here is predictability and Uber chooses to meet their prescribed level of predictability by controlling the phone OS/version.

> Have to assume driver phones can be compromised which means the data must be made tamper proof. So all the data is encrypted on the phone.

> In the background the Replication Service encrypts the data and sends it to the Messaging Service.

> The Messenger Service sends the backup to the phone.

As long as the phone can't ever read the encrypted data this is probably secure, but shifting costs onto the driver through using their bandwidth to store encrypted backup blobs seems unethical. The uber app on the phone isn't using this data to the benefit of the user, it can't as it is encrypted.

I don't think you should use a person's device for anything other than serving that person. This is the kind of thing though I would totally expect from Uber. Based on what the media has reported in the past, the whole company's culture seems toxic and dishonest from top to bottom.

But the driver is served by this -- the driver wants the Uber platform to be scalable, available and ensure the 'successfully customer experience' the article refers to, so that they can continue to get paid for providing the rides.

Now, if Uber were offering a free 'lol cats apps' on the app store that surreptitiously ran this protocol in the background of unsuspecting users I think it would be a different case, but running technology that supports the goal of the main app doesn't feel out of bounds to me.

To your point, those phones are not the drivers personal phones, but they are provided by Uber. Uber is just storing information on their own hardware basically.

Apparently that's not true anymore. I was talking to some Uber drivers a couple of weeks ago in Vegas and they have to provide their own smartphone.

Exactly. You can download the driver application from the Uber Website. They are distributing it as an Enterprise application(on iPhone) thereby circumventing the App Store.

> I don't think you should use a person's device for anything other than serving that person.

I might have missed something to the contrary, but to me the article only describes how they use a drivers phone to store information about the trips he is making. Which the phone is involved in anyways.

Similar to storing stuff in the session cookie of a web app.

I agree - I read this as they are storing the current state for the trip the driver is engaged in on the phone so when datacenter failure occurs restoration source is the uber driver phone.

You have to store data on the client endpoints in realtime logistics applications; its just how it's done.

If you don't, networking failure scenarios will destroy your user experience since you'll end up with frequent irreconcilable data gaps with cellular networking being what it is.

In general I think it can be fine to use a person's device for other things.

A famous example is Skype: they were able to set up a global VoIP system with very little capital, by making it a peer-to-peer application rather than requiring a central server to route all calls. So your computer was used to forward unrelated calls, but that's generally fine, the burden on each individual user is low.

In the Skype case, there were tons of computers out there, owned by individuals but mostly idle, and Skype managed to use them more productively--that is a win for everybody. With mobile phones it might be a bit different because the bandwidth is limited, but still, people can see how much data the uber app is using, and decide whether it is worth the benefit to use it to hail uber cars or not.

Skype example proves exactly the opposite: the main reason they had to switch back to a centralized setup is because as mobile became a more dominant usage scenario P2P became untenable. Supernodes had to serve quite large and -- worse -- constant traffic which kept them alive (draining battery and consuming data).

From the article, it seems this is not a true p2p architecture. It's more of a hub and spoke model, with each phone connected to the cloud, but not to other phones.

Yeah, that's why I was saying that Uber (mobile) might be different from the original Skype (PCs).

Is that compatible with this?:

  Why choose this approach? The traditional approach would be much simpler. I
  think it is to make sure the customer always has a good 
  customer experience and losing trip information for an 
  active trip would make for a horrible customer experience. 

  By building their syncing strategy around the phone, even thought it's 
  complicated and takes a lot work, Uber is able to preserve trip data and make
  for a seamless customer experience even on datacenter 
  failures.

If the phone can't unencrypt it, it can't take advantage of the data if the datacenter goes down surely?

edit: p.s. dang if you read this, please can we get a way to do quotes with automatic text-wrapping >:

> I don't think you should use a person's device for anything other than serving that person.

It's not the person's device. Uber requires all drivers to have a separate phone just for Uber usage. I had always assumed it was so that drivers wouldn't hit on passengers by calling them after dropping off. The distributed data backup seems like another advantage.

> assumed it was so that drivers wouldn't hit on passengers

To my knowledge, Uber relays calls between drivers and passengers through its own phone network, which effectively anonymizes each end.

For example, an Uber driver once called me, and then my phone lost service. I tried to call back with a friend's phone, and an automated Uber message said that there were no active rides associated with my friend's number.

When I took Uber outside the US, this wasn't true. The app faithfully showed my real phone number on the driver's phone (the driver took the +1 as a queue to not call me and instead drive slowly in hopes of finding me).

Are you sure it was your real phone number, or was it just an American number that Uber owned? That's what they use in the US -- a pool of numbers that they bought to function as masks.

It's to protect the privacy of drivers as much as as passengers.

I think its a solid solution from a tech standpoint. Phone (or any end point for that matter) holding on to data/state during a DC failover and being aware of an outage happening at the DC makes the recovery process, so much more graceful. One could argue that if this solution was not in place, driver would have lost the trip data during a datacenter outage which would have cost him his earnings. So this is directly helping drivers in exchange for him allowing to store encrypted data on their phones during such outages.If anything I would explicitly say this to the driver installing an Uber app and seek a acknowledgement. You are also over indexing on what you hear in media about Uber. What you are quintessentially saying is, there were two solutions available for the same problem and engineers ended up choosing this one, because he wanted a dishonest solution over a honest one, so he can fall in line with the company culture, which is dishonest :). That does not make sense at all.

Which is a shame because the actual boots on the ground are usually very kind, clean, and courteous, and the ride is far, far superior to cab experiences. At least anecdotally, here in Philadelphia, PA.

Most people are great. Ridesharing is great. There are different mentalities in the coordination organizations, though.

For example, a Lyft driver mentioned to me they had an Uber recruiter in their car the other day. The person was pushy, insistent and distracting, which motivated this driver's loyalty to Lyft and gave me a subtle, mental "-1" for Uber.

The concept is obviously sound, but that doesn't make the companies driving it immune to failure.

Are you sure it was an recruiter? Uber doesn't hire them afaik. Probably a driver wanting to claim the $500 referral bonus. I've had plenty of Lyft drivers doing the same and my response is always "cool, give me your card!"

I'm sure the distinction is important to people in the industry, but from the outside, someone who pesters drivers in the hopes of receiving $500 referral bonuses is a recruiter.

Not only Uber used to hire them (I don't know if they still do), it also provided them with burner iPhones and credit cards so that they can continue recruiting and cancelling rides as Lyft tries to ban them from the network. There was a bit of media shitstorm about this some time ago.

Seems inline with a lot of the other ways they have sneakily taken a greater share from drivers. They used to give drivers the phone and service for a refundable deposit. Then they started charging $10 per week for use of the Uber phone, eventually they allowed drivers to BYOD and now they are using their data plans as a backup service.

The so called "safe rides" fee is another example of Uber dishonesty. The percentage of the fare that goes to Uber w/o having to actually say they are changing the split has gone way up. That was set initially at $1 and it looks like it is up to around $1.50 now. This is in addition to the 20% cut Uber takes off the top of the actual fare.

" Updating a stored trip happens like: set(“trip1, version2”, “yyu”); delete(“trip1, version1”). The advantage is if there’s a failure between the set and delete there will be two values stored instead of nothing stored."

So simple yet could be overlooked. Great read overall.

This is Copy-On-Write in disguise: the different versions can actually be considered different objects, in which case it's just a copy and an eventual cleanup of the old object.

Right. Just read what Copy-On-Write is. Follows the same basic principle. I might be wrong, but Copy-On-Write is formally referred to the case of a shared resource? Though I think we could consider the trip information to be shared between the phone and the data centre.

See also Postgres multi-version concurrency control. Doesn't solve the problem of merging incompatible inatomic-concurrent changes.

thanks will surely have a look.

I wonder why they didn't use something like CouchDB for this purpose and instead engineered their own solution? I see the CouchDB replication protocol + conflict detection perfect for their usecase.

I often wonder why CouchDB didn't take off more. It wasn't a panacea but it was clever enough and you could use it in mobile apps. Perhaps the decision of the "host" company to merge with another company and head in the direction of CouchBase made people wary. Or there's some technical reason I'm not aware of. The latter seems more likely.

I am one of the original CouchDB folks, and a cofounder of Couchbase. We have spent the last few years quietly building an enterprise-class suite of Couch sync compatible databases. Everything we do is open source and Apache licensed.

Couchbase Mobile is native on iOS, Android, and C#, and compatible with PouchDB and Apache CouchDB. http://developer.couchbase.com/mobile

Here is an example of how General Electric is using our open source platform to power the Internet of things. http://www.couchbase.com/nosql-resources/presentations/offli...

[edit] we've been able to do this, funded by the success of Couchbase Server. Which looks like a high-performance NoSQL database, not an offline mobile database, but it uses many of the same data structures.

Thanks, looks like I have some research/reading to do.

Aside: This is why I love HN. Thanks for all of your hard work.

Yes, although we have a release of 2.0 coming up, and the PouchDB project is really active. The problems couch can solve (e.g data safety and offline capability) probably aren't considered when initially building a product. Also moving from SQL to CouchDB is quite tedious, although my company just went through that process :)

I feel stupid maybe, but can pouch/couch sync anything other than an entire database? The lack of granularity there always turned me off.

Yup, filtered replication supported by pouchdb + couchdb - http://pouchdb.com/2015/04/05/filtered-replication.html.

There are also other models, sharding individual databases per user or some other application logic. There are definitely some use cases we dont handle amazingly right now, but its all heavily being worked on.

This seems like a great approach when the client devices are all friendly and collaborating. What sort of countermeasures do you need when some of them might be malicious attackers?

The tampering issue is described in the article.

> Have to assume driver phones can be compromised which means the data must be made tamper proof. So all the data is encrypted on the phone.

Their explanation leaves me wanting more. If the phone is compromised, surely the encryption process might also be compromised.

The replication system encrypts the information before it is transmitted to the phone. The phone never has access to the key.

Ahh, yes it's described in a different section. Cool!

Vector clocks and Conflict-free replicated data types should make it safer and easier for more and more applications to follow a similar approach to distributed data storage with delayed synchronizations to the reference data-center:

https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

This looks like the prime example of FOG computing [1]:

"A network architecture that uses one or more end-user clients or near-user edge devices to carry out a substantial amount of storage (rather than stored primarily in cloud data centers), communication (rather than routed over backbone networks), and control, configuration, measurement and management (rather than controlled primarily by network gateways such as those in the LTE core)."

[1] http://fognetworks.org

Looks like there might be a surplus of engineering talent (or non-talent, perhaps) at some companies, which results in these bizarre ad-hoc projects.

What's bizarre about this?

Wow... always makes me very happy to see any move toward pushing a bit of intelligence out to the endpoints.

what if in the distant future they could do away with most of their datacenter with a solution like this (improved of course), imaging the savings... maybe this is their first step into looking into it.

Unless phone storage and bandwith become cheaper than data centres, there is no saving, and even if there were, the cost hasn't vanished, just been hidden.

That depends on the type of messages; if one device is sending the data, and the other is receiving it, you could absolutely save bandwidth by implementing a P2P system instead of having the data center as the middle man.

An example would be the rider seeing the position of the car they have hailed - P2P would save DC bandwidth without having the devices spend more.

There's overhead to P2P; if there wasn't we'd never have had servers, the desktops on the LAN would have just done it.

Previous discussions: https://news.ycombinator.com/item?id=10428508 https://news.ycombinator.com/item?id=10253158

None of these really are a "discussion" (IMHO "previous discussion" links have two purposes: either to show something has recently had active discussion and therefore shouldn't be reposted, or to link to very old discussions that might be of interest. For me, linking empty discussions doesn't add anything useful)

Threads aren't considered duplicates until the story has had significant attention on HN.

https://news.ycombinator.com/newsfaq.html

Just noticing: https://news.ycombinator.com/item?id=10428508 is a 3-day old submission from the same user, and had modest attention.

Same user is fine and might actually mean that the mods asked the user to repost (although dang probably would have mentioned it if they did in this case)

No, you're right, it was one we invited the user to repost. I didn't see that earlier.

This seems to poke an ever bigger-than-usual hole in their fantasy that drivers are contractors rather than employees.

So, they use the internet, which is generally considered "two nines" reliable, to achieve 99.99% reliability? Oh, tell me more. <wonka-meme />

I skimmed through the youtube preso and saw nothing about results observed in production, either through actual DC outages, or chaos monkey, or game days.

Seems highly overwrought architecture astronaut'ing and also very difficult to get any telemetry on how well/if it works, to me.

Do you experience 90 hours of Internet downtime every year? That's "two nines".

Okay, good theory crafting...

From http://www.nytimes.com/2011/01/09/business/09digi.html?_r=0: "A home user’s Internet connection, with a laptop using Wi-Fi, would be available about 99.8 percent of the time, estimates Mr. Hölzle at Google, which equates to about 18 hours of cumulative downtime a year."

So yeah, two nines.

Also, here's the results of me pinging google.com from the network of my "large technology company" who employees me.

652 packets transmitted, 646 packets received, 0.9% packet loss.

Two nines.

We're talking about mobile internet service. "Two" is probably doable. "Four" OTOH would be less than 53 minutes of downtime a year, which I might believe if you were standing still, on a high point, in an area that didn't have much going on electrically. I would not expect to see that while driving, or perhaps I mean that I have not seen that while driving. But GP is wrong anyway; see sibling comment.

By spreading data to enough mirrors, one can achieve arbitrary reliability. I doubt that any part of their business is only replicated to any particular device. After all, batteries die all the time, unpredictably.

They sure know how to offload their costs to society, while reaping the benefits.

I'm looking forward to the day when this "fat client" strategy will have lead to a "decentralized Uber" (based on open-source technology, etc.).

Securely storing information about a driver's trip on that driver's phone is hardly "offloading their costs to society." I assume the driver does not want the information about the trip they are providing to be lost, and therefore that they do not get paid, if an Uber data center goes down. This is not about storing all of Uber's data on the driver's phones (this would make no sense), only the information for that particular driver's activity.

Further, as the article describes, this may actually be more reliable than separate backup data centers.

This is the reason I don't use uber unless I have literally no other alternative. They continue to show a willingness, damn near an eagerness to make their drivers suffer for their unwillingness to work with local government by chipping away at the earning potential of drivers.

And even when I was out of work for a while-I've got a car that would be perfect for Uber Select, I refused to drive for them because their business practice makes it abundantly clear they do not care about you as a driver. You're a means to an end, no matter the cost.

Reply on news.ycombinator.com