A former Uber engineer's disaster story | Hacker News Reader

A former Uber engineer's disaster story

twitter.com

710 pointsepaga332 comments

This may be an egregious question, but what does the Uber app actually do that makes it so big? It displays a map with some moving dots on them, allows you to pick a location, and asks the server for a route and price. Plus a bit of workflow for signup, credit card, reviews; not a huge number of screens.

I appreciate the difficulties of getting an app to fit in a fixed space having worked on a point of sale system that had to run on a MIPS device with 32MB of Flash, of which half was occupied by Windows CE. It's not really clear to me why Uber's app needs to be so large and complicated other than the availability of a large amount of money, high-powered engineers, and shiny new unproven technologies.

(The compact app fit in about eight megs in the end, plus another meg or so for the file which defined all the screens; it was written in the least trendy, unsexiest framework known to man, Windows MFC in C++)

Follow-on tweets address this somewhat: Every airport has different pickup locations, every region has different payment rules including cash, etc.

Apparently all this location-specific business logic has to be in the app itself and is not downloadable on demand.

There are a lot of ways to frame these problems as interesting algorithms but it doesn't really strike me as something that is troubled by scale.

I've worked on similar problems where "everything is a special case" in some vein and it takes a few very clever minds to construct the algorithms and datastructures that can come to a good solution. But that's just it, it's the work of a few engineers and not thousands writing if-else blocks for each airport.

I still don't understand what Uber does that requires so many people getting paid so much money, the only thing that makes sense to me is they don't want them working for Lyft. It would be helpful for people to shed more light on how engineering resources are spent and what particular systems require so many hands to realize, but that's too valuable information to leak I'd imagine.

I had similar questions.

Having a large number of engineers is fantastic for making the company valued higher.

Directors love having more engineers under them.

Overall I've been highly speedy with 4 engineers, and slowly adding a 5th. Sure there are limits, but I think limits makes us think about being more efficient with how we build. It is an entirely different mindset.

Once you decide to "go big" you end up in this situation where you usually literally throw a dozen engineers at the problem and the problem goes away. I've seen designers spend 6 months on a _single page_. Because they can. and they get paid crazy money for it.

At a growing massive company it's not about efficiency but total outcome. If 10x the engineers gives you 10% more revenue then that's worth it economically if the pie is large enough. 10% of 4 billion a year can pay for a lot of engineers. Once you're no longer growing then you need to focus on efficiency as more engineers rarely translates to more total revenue in that case.

A good place to start would be to go through the Uber UI and click through every button and every page that exists.

Now create a driver profile and do the same thing.

Now create the multitude of other special profiles they have and do the same.

Now do the same for Uber Eats.

Now do the same for Eats but using the restaurants version of the app.

And the couriers.

Now get in on Uber Freight (and any other projects they have) and do all of the same.

Now do all of that again for their other platforms (iOS, Watch OS, Wear OS, Android, Web)

Now access their internal tools that make all of this possible and do the same.

Now think about localizing all of that. Handling special cases for different places/languages/legalities. Think of all the different payment integrations. Security issues. Infrastructure. Technical debt. AB testing. Metrics. Dev tooling.

And I've only begun to scratch the surface here. The point is there is a lot more than meets the eye, especially when operating on a global scale.

This is a common but specious argument. It is possible to make large, complicated apps without creating a monstrosity. That you are making something like this is a manifestation of an engineering culture that has too many cooks in the kitchen, a lack of specific engineering talent spent on finding issues and telling the teams shipping them to stop, people shipping libraries that they don't need and tech debt that was never pruned away. An app with 100x more people working on it should not be 100x larger, because it certainly won't have 100x more features in it. Sure, Uber is more than "a map and a couple views" but the fact remains that their app should never have been in the state that it was.

For my own curiosity, what large, complicated apps are not monstrosities in your eyes?

The Apple Maps app on iPhone, for example? If you make it fair and count the code that goes into the frameworks specifically for it, it's still tens of megabytes at the most. There is a huge part of it running server-side to support it, of course, but it is difficult to say that it is not a complicated app. Or consider the Mail app? It needs to deal with IMAP, and has custom flows for a number of named mail services (Gmail, Yahoo, etc.), and then it has to have special code to handle all the edge cases of "what happens if the user deletes a message here but it didn't sync, or Google sends us 503s sometimes if we do this, or…" You can make arguments that maybe it is slightly less or more complex than Uber, but it is just a couple dozen megabytes and not hundreds.

10s of megabytes? Where did you pull that number from? Since it is a system app, its true install size is hidden away from users. The comparable google maps app is 190 MB.

I counted up the system frameworks it relies on that can be reasonably considered to exist for the purposes of serving the app.

We used to pack entire operating systems with userspace in a couple dozen Megabytes. I bet they had more screens than the Uber app.

You should scratch below the surface because this isn't really explaining much - it does not sound like something that requires more than 100 people. Their legal team sounds like it should be twice that alone.

Or am I crazy?

Take look at this thread: https://news.ycombinator.com/item?id=25376346

My point is that there is 100s of screens/features in the Uber app that you don't even know exist. And you have those same 100s of screens across their other apps. Multiplied across all of the different platforms. Plus any internal tools they use.

I'm curious, have you worked at a software company of Uber's scale? Not trying to be a dick here, genuinely curious because I hear these types of comments often here on HN and I wonder how many of those commenting have seen the scale from within.

But like, why? Do all of these things make them tons of money? It's a dumb taxi app, ultimately.

You're not answering the question except as a tautology:

- Q: Why is Uber's engineering team so ridiculous and bloated?

- A: Because they need a ridiculous bloated team to make this ridiculous bloated app.

Just like Alphabet is just a search engine, and Microsoft is just an operating system, and Amazon is just an online store.

Companies grow into new products, services, markets, etc.

Google really is just an advertising company. Everything else "Alphabet" does is a rounding error next to ad revenue.

Microsoft and Amazon have multiple profitable lines of business.

Is Uber more like Google or more like Amazon? Is "Uber Eats" (underpaid contractors drive around takeaway dinners) distinct from "Uber" (underpaid contractors drive around people), or "Uber Black" (underpaid contractors with unwise car leases drive people around)?

> Google really is just an advertising company. Everything else > "Alphabet" does is a rounding error next to ad revenue.

It's not like people would go and visit a page with just a list of ads; no matter how little revenue the other products around ads bring in, they bring eyeballs to the ads, and serve as a moat around ads.

> Google really is just an advertising company. Everything else > "Alphabet" does is a rounding error next to ad revenue.

It's not like people would go and visit a page with just a bunch of ads; no matter how little revenue the other Alphabet products bring in, they bring eyeballs to the ads, and serve as a moat around ads.

My whole point here is I don't understand why "Uber scale" is so large. I have worked in large organizations - smaller than Uber, but we did more stuff (as in more complexity and diversity in products - shipping software and the hardware to run it, for example).

"100s of screens" is as meaningless to me as "thousands of lines of code" and wholly uninteresting because it seems like the front end is the least significant reason why Uber requires so many people to operate. That's why I'm interested in seeing how many people are assigned to projects and what they actually spend their time doing, because at scale you begin to experience inefficiencies due to scale for bad reasons if your incentives aren't aligned. From the original twitter thread that sounds like it was rampant at Uber.

Not sure how long you've been around, but "Uber scale" used to be a joke a few years ago because no one could understand why they would reinvent so many wheels because "they don't work at Uber scale." The joke being that Uber didn't need to be Uber scale to begin with, they just had investor money to burn.

I'm just a layman, but I often ponder this same "Why are some companies so big when a small team could do about the same thing?" and my mental theory currently is:

1. The amount of engineers scales very badly with respect to application complexity. I.e. you may need 100 or 1000 engineers to do something that "looks" about 2x as complex as something 10 engineers can do, and

2. The more complex solution tends to win in the market because on the one hand people value the application handling edge cases better, having shiny graphics, having slightly better UX, etc. and on the other hand you redeem the cost of paying those 1000 engineers thanks to software being free to copy and network effects / monopoly.

A helpful way to think of it is by comparing how Uber and an OS GUI are different (since that's the comparison people are making, anyways)

With an operating system UI, the developer may reasonably know what the UI will look like, what features are present, where they are placed. With Uber, a single person may not know all the rules for a single feature, and the rules change frequently, sometimes in the most obtuse ways. Like when San Francisco decided to ban cars from one of its most important artery streets earlier this year (market st). Now not only you need to indicate to a rider that this is a thing, your UI also needs to to provide walking instructions to the nearest designated pickup spot, and the routing algorithm needs to not put drivers on market st. Oh, and don't forget Hebrew speaking tourists read right-to-left and that food delivery bicycles _are_ still allowed on market st. That's just one change in one city.

Consider that just about anything might change anywhere, any time. From how earning breakdowns are displayed for drivers in Sao Paulo (this differs by jurisdiction), to where the pickup locations are in Pearson airport in Toronto (airport rules change all the time), to what promos apply in Mexico City this week (which depends on marketing and finance analysis), to whether surges should be disabled in response to a terrorist attack, to what documents a driver signup rep should be scanning into the system as local regulations change.

Now build all the infrastructure required considering that the people on the ground reporting their local changes are not programmers, and conversely many of these changes may require making changes to the UIs that these people use, how the new data is aggregated, etc. There are literally thousands of greenlight hub / customer support employees worldwide using these internal tools, many don't speak english and timezones are all over the place. The nature of problems are often fuzzy, hyperlocal-specific (e.g. certain forms of fraud), or extremely meticulous (e.g. legalese, etc).

The trouble of scale comes with having to deal with numerous nebulously defined things from numerous stakeholders and having to distill which engineering teams are even responsible for which aspects of which thing. As the number of stakeholders grow, so does the risk of miscommunication, unclear chain of responsibility and other well-documented large organizations challenges.

That's a bad comparison and I'm not sure why people are making it - you're mistaking fundamentals. This isn't a UI problem, it's datastructures and algorithms. There are plenty of desktop apps with similar issues (representing a complex problem as a set of constraints and writing a solver to find a solution, and querying the solution for the data to display to a user).

I don't reject that there are troubles in concerting real world data into a representation for the constraint solver. But this isn't untrodden ground, and there's no reason a front end developer should need to care about it. The problems you're describing are far behind the UI; and if they're not your organization has immense problems.

> there's no reason a front end developer should need to care about it

Well, yes and no. You're right that trying to express every nook and cranny of the business rules as solely a UI problem is a wrong way of trying to understand the complexity. But a frontend developer does need to care about how to surface whatever complexity needs to be surfaced, and likewise, backend developers need to know how to model the complexity, and there needs to be some level of orchestration between all the moving parts. Remember, Uber is built on a microservice architecture, and many things that would be arbitrary hardcoded values in projects elsewhere (e.g. pricing) are instead tied to complex dynamic systems in Uber.

In the pricing example, when Uber added the option for drivers to set their own multipliers, it wasn't just a new slider in the driver app. It also affected the pricing AI model, the matching algorithm, how and when prices are displayed in the rider app, geofencing rules, etc etc.

When new regulations mandate that every commit that makes production changes in your city must have an appropriate audit trail, when you need to deploy a cheat-proof face mask detection AI model in response to a global pandemic, when you need to figure out how to make customer support cost $1 per engagement instead of $10, when you need to figure out how to make your network protocol faster in Bangalore, when you need to figure out how to make GPS more accurate in Manhattan, or when any of hundreds of fuzzy problems need solving, then saying that variations of all of these problems have been seen elsewhere misses the point that going from problem statement to deployed solution isn't always a trivial task.

The original question was why so many people work at Uber. The answer is that there are a lot more fuzzy problems than meets the eye.

Seems like a reasonable architectural choice early on when your app is live in 2 cities. Instead of changing the language of the app, they should have focused on its API architecture.

Yet, the last time I was throttled to 2G speed due to a problem with my carrier, the Uber app couldn’t even start (getting to the first interactive screen) in like five minutes. What’s the point of distributing the entire world’s data, which gotta change from time to time, when you can download the tiny requisite piece each time, especially considering that a relatively high speed Internet connection is required to use the app anyway?

Edit: https://news.ycombinator.com/item?id=25376346 goes into more details.

Uber actually has a not-so-well-known product for this very problem: the mobile web version (m.uber.com)

It doesn't have nearly as much functionality as gregdoesit mentioned, but it's enough to service the use case of getting you from point A to point B when you're stuck w/ crappy connectivity.

Wow, I had no idea! On app load, maybe checking speed and then displaying a link to the web version for low bandwidth areas would be a good move.

Why display a link? Hybrid web/native apps are a tried-and-true strategy when you have massive complexity or dynamic content that you don't want to ship with your app. If you use it judiciously and tastefully the "native UI everything" people don't notice and come after you, either.

Not an app developer ;). But this makes sense. Thanks

Fun fact: if you open the Uber app and hit the Food icon/tab, the experience is entirely a web one: it is powered by ubereats.com in a webview.

The other not-so-well-known product is the Uber Lite app (iOS and Android) which was specifically made by Uber for this problem, particularly in the developing world.

Wish they still had what they did when I started using it: send an SMS to a shortcode number with your pickup location.

I actually built this product out (for both Uber and Lyft) - SMS/MMS-based ride requests (MMS only for sending google maps PNGs to confirm pickup location). I was about to convert it from a personal project (I used it to get rides on both services for a few years) to a "real" one open to customers, etc and then ran into snags with both APIs at around the same time. Uber's API changed to potentially require a web flow to confirm surge pricing (IIRC - now it's scheduled to be disabled entirely very soon) and Lyft's API was basically shut off entirely (there's a way to apply but I was unable to make any progress even through internal contacts at Lyft). I'll try and keep an eye out for any changes...

We've previously spent a lot of resources also looking for ways to render native components through backend APIs (there was a team called screenflow that was laid off) but this was complex and wasn't able to support complex business needs. Uber does not want to remove app size at the cost of UX (ex. webviews).

I’ve often wondered the same thing about Airbnb, DoorDash, and other unicorns where they hire the best engineers from Silicon Valley, yet their product seems like a rather basic CRUD tool.

Apparently with Uber it’s the infrastructure design and back-end algorithms for trip routing and driver management that are “so great.” The SV hype machine helps a lot too.

I doubt that the truly best engineers work there. But certainly at their scale you need really good ones at least.

They also have a huge need for speed. Shipping faster and winning a market is worth 10s of million in annual revenue and billions in valuation.

A lot of this is resume driven development as well. Solving a hard problem, even if it's an invented one, looks good on your resume. So does using the latest and greatest tech stack instead of a boring but functional one.

Joe the Data Scientist says that if the app dynamically does X then that could be worth $10 million a year. Steve the product manager says his change could be worth $1 million a year. And so on and so on. Most are wrong, some are right. If you have a small team you'll never implement any of it so you won't know. If you throw people at it then you can test all those things. And when your company makes billions a quarter it doesn't take much of a revenue lift to pay for a few dozen teams.

>This may be an egregious question, but what does the Uber app actually do that makes it so big?

It justifies the employment of 400 iOS engineers, it also lets you book taxi rides.

Experimentation / data-driven product development. You’re actually downloading 583 small variations of the same app, plus all the localized rules mentioned above, and it quickly adds up.

Former Uber engineer/EM here: I worked on the Rider app.

The “there are only a few screens” is not true. The app works in 60+ countries, with features shipped in the app that often for a country, and - in rare cases - a city.

The app has thousands of scenarios. It speaks to good design that each user thinks the user is there to support their 5 use cases, not showing all the other use cases (that are often regional or just not relevant to the type if user - like business traveler use cases).

Uber builds and experiments with custom features all the time. An experimental screen built for London, UK would be part of the app. Multiply this by the 40-50 product teams building various features and experiments outside the core flows you are talking about (which core flows are slightly different per region as well).

I worked on payments, and this is what screens and components are in the Uber app:

- Credit cards (yes, this is only a a few screens)

- Apple Pay / Google Pay on respective platforms

- PayPal (SDK)

- Venmo (SDK)

- PayTM (15+ screens)

- Special screens for India credit cards and 2FA, EU credit cards and SCA, Brazil combo cards and custom logic

- Cash (several touch points)

- AMEX rewards and other credit card rewards (several screens)

- Uber credits & top-ups (several screens)

- UPI SDK (India)

- We used to have Campus Cards (10 screens), Airtel Money (5), Alipay (a few more), Google Wallet (a few) and I other payment methods I forget about. All with native screens. Still with me? This was just payments. The part where most people assume “oh, it’s just a credit card screen”. Or people in India assume “oh it’s just UPI and PayTM”. Or people in Mexico “oh, it’s just cash”. And so on.

Then you have other features that have their own business logic and similar depths behind the scenes when you need to make them work for 60 countries: - Airport pickup (lots of specific rules per region)

- Scheduled rides

- Commmuter card functionality

- Product types (there are SO many of these with special UI, from disabled vehicles, vans, mass transport in a few regions etc)

- Uber for Business (LOTS of touchpoints)

- On-trip experience business logic

- Pickup special cases

- Safety toolkit (have you seen it? Very neat features!)

- Receipts

- Custom fraud features for certain regions

- Customer support flows

- Regional business logic: growth features for the like of India, Brazil and other regions.

- Uber Eats touchpoints

- Uber Family

- Jump / Lime integrations (you can get bikes / scooters through the app)

- Transit functionality (seen it?)

- A bunch of others I won’t know about.

Much of the app “bloat” has to do with how business logic and screens need to be bundled in the binary, even if they are for another region. E.g. the UPI and PayTM SDKs were part of the app, despite only being used for India. Uber Transit was in a city or two when it launched, but it also shipped worldwide.

And then you have the binary size bloat with Swift that OP takes about.

Thank you for all these details. I think it's a good sign when people think your app is simpler than it really is - the Uber app does a good job of hiding this complexity.

I had a theory about why very very skilled people aren't promotted as they deserve : because they make hard problems look simple.

On the other hand, the not so skilled ones...

Look, I get it that the app is complicated. I look at these apps from the perspective of a reverse engineer but invariably I find places where there is bloat just for the sake of being bloat. There is code that is provably not used, entire libraries that included just so that one small part of it can be used. I see time-related assets that have existed long past the date that they could have possibly been useful. I have seen apps that bundle entire scripting runtimes, apps that have half a dozen in-house telemetry libraries, apps that have megabytes of files in them in the hopes that they may be useful someday.

At this point it's really hard to believe the argument that "but our app is more complex that you could possibly understand" is valid. Compare your list with a Linux distribution from a handful of years ago: do you have more "screens" than that? Can you do more?

Eh. It doesn't seem particularly relevant to bring up Linux, with its hundreds of thousands of person-hours worth of volunteer contributions and an extremely technically focused BDFL gatekeeper.

It's a relatively well known fact that multi-team corporate projects have tech debt. There was a thread recently about Google Cloud being slow due to too-many-cooks-in-the-kitchen syndrome, and many ex-amazon folks say AWS UI is notoriously difficult to refactor for similar reasons.

The Grand Bazaar of Istanbul and the Tower of Babel.

I mean, Uber has had a huge number of people working on it too, with the incentive that they are paid to do a good job. Regardless of the eventual outcome, I have no misconceptions that Uber doesn't have some engineers who are extraordinary talented. And, the project is newer, too, with fewer guarantees on stability and fewer users to boot. I think it's a valid comparison to make at least–they are both huge, complex projects with a massive number of people working on them. Tech debt is inevitable, but I still think it is valid to call it out when it occurs and not try to excuse it with "but our app is super complicated, you can't possibly understand how complicated it is".

I think the argument is more that even if you were to magically get rid of the cruft introduced by institutional bloat, the app will still necessarily be large given the nature of the business. Uber isn't some outlier here either; Lyft, Ola, Yandex, GoJek, Grab, etc etc all have comparable iOS app sizes.

It sounds stupid, but I'm going to say that they're all doing it wrong. One commenter in this thread brought up the Uber app for Android and it's a fraction of the size–clearly, market constraints can make it possible to do better?

To be clear, I do think you have a valid point about e.g. unused assets and loading big libs to do trivial things, but IMHO, that's really a commentary on software development at large, rather than anything specific to Uber.

And there are cases, even in the Uber app, where thought has been put into optimizing both for upfront download size and internal complexity (the legal pages, for example)

Something that is worth considering is that a lot of what was said about the app being complicated is that it is also _dynamically_ complicated. For example, in the driver app, a feature was added to ensure drivers are wearing face coverings. The backend for that does AI-based fraud detection to prevent shenanigans like photos of photos. The picture upload flow is just one small part of a much more complex project that had to be rushed out because the covid pandemic happened. There are a bunch of similarly large impact things that touch several of the Uber apps simultaneously, often in non trivial ways and with tight timelines (e.g. features for compliance with specific legal requirements)

The reason I think Linux seems like a bad comparison is that if Linux contributors don't want to support some random driver for 3 years, that might be annoying but is generally just the way it is. Uber can't really afford to not do some of the complicated things it does in a timely manner.

What I don't really understand is why so much of the dynamic content ends up being native code on the client. For example, let's say you need to have some of legal compliance page. But there is always changes you have to make in a bunch of regions, and the flow is constantly changing, or whatever. But why can't this be a DSL you download and deserialize into views? Why does there have to be a hundred different native views for this that all must be part of the app? I'm hearing all these tail-end cases that one region hits and interacts with for seconds and I guess I don't really understand why these things are not handled server-side, making the app a dumb client. After all, isn't that how the website surely works?

I understand that Linux doesn't necessarily always have the product marketing deadlines or whatnot that Uber might have, but it's saddled with legacy code too. When things have security issues, or the API is just awful, at some point someone needs to fix it. Especially if your business is being decimated by a bug or you are lacking in APIs that meet your usecase. I don't think it's as poor as a comparison as you might thing it is.

I think a lot of it is driven by dysfunction and legacy at Apple. Their whole mindset is they want to review every version of the app manually and every bit of code that executes must be signed. They don't allow a lot of obvious software engineering approaches as a consequence, like partially downloading the app and streaming the rest on demand even though it's an obvious thing to do for mobile apps.

Note how in this story the Android and iOS teams started a big rewrite at the same time, but the story consists exclusively of fuckups on the Apple size. Apple impose an arbitrary cellular download limit. Apple designed a language that can't handle more than 9 dynamic libraries before it hits scaling issues. Apple prevent you from using interpreters or other techniques to change the app after it's installed, unless you use a WebView in which case it's OK for mysterious reasons. Apple don't help even when major brands are weeks away from wrecking their business by hitting these limits. Apple Apple Apple. Meanwhile the Android guys are sitting pretty with Java and now a migration to Kotlin, which is (a) a much better language than Swift and (b) is dogfooded at large scale on the IntelliJ project so you know it scales (had no scaling problems with Kotlin and I've been using it for the last five years).

It's a bit unfair to compare the mobile app with the Linux distro, given that mobile apps aren't supporting dynamic feature loading (or weren't at that time), so everything should be bundled. Imagine that any "screen" you need to access _must_ be bundled within the distro. So, compared to Linux distro – you should have your local mirror of all possible features any particular user would like to have. For example, Gentoo handbook says "Hosting a mirror requires a minimum of 550 GiB available disk space." 550 GiB just to run a Chromium? No – to have an opportunity to run it when you'll need it, because it's all or nothing. Either you get every program with all of its features, or get nothing at all.

Oh, no, I wasn't including "you can download packages" in this–I want it to be a fair comparison. Take a medium-weight Linux distribution and put it on a laptop and go on an island somewhere, if you want to tilt the comparison in the favor of Uber, which can of course still download assets from the internet and be backed by a server doing significant work for it. At that point I think it's still difficult to make the argument that Uber is somehow composed of more "screens" or "edge cases" that it needs to handle.

I am not an app developer, so ignorant. Why not have different apps? Uber-India, Uber-Mexico, etc? Or if it is to be all one Global Uber app, why not have expansions that are downloaded for other regions as needed?

Friction and convenience. Always assume the user will not take the desired action because it most closely models reality.

Would you want to download a different app every time you traveled to a new country? If you're in the EU/APAC with many countries close together and travel for business?

Uber works relatively well in every country you go to, including at one point, China. Not many apps have ever accomplished that.

I wouldn’t mind if the app started up and said, “download the $country_or_region extension,” but that is just me (assuming it doesn’t require re-creating an account, or re-entering billing details).

Pretty much the last thing I want to do every time I step off a plane is download, install, and sign into a new app before I can get into a car to go to my hotel.

Get off the plane in a developing country. Some of those airports are not nice. Some of them are nice shells that are almost deserted, with no wifi and a couple vendors that won't take your American credit card. Your "World Plan" often works, sorta, but you can't always get data due to different radio bands or because the network is simply not good enough for a 100+mb download. You'll either have to hop into a hopefully licensed taxi, or start walking in the dark in an industrial area (where airports usually are) in order to find a wifi hotspot or a street vendor that will sell you a SIM card. It's not fun.

Uber made the right choice.

Why does it take 100MB to download the business rules for that particular country? It will take some data, but you will need some to book the taxi anyway.

Imagine yourself at an airport and trying to order Uber, but be forced to download new 50 MB app. You would immediately try some other app like Bolt or Lyft.

I'm from EU country and it was so convenient to be able to use Uber in NYC the same way as I would in my home town.

I'm now wondering why Apple doesn't let you ship different binaries for different regions?

They do (well, you can have different listings limited to different regions).

But basically the only time I use Uber is when I travel to another region. I don't want 30 different Uber apps on my phone...

As someone who lives in the US, but (pre-covid at least) regularly travels to the UK (where I am from), I wish regional listing was not allowed _at all_. It's incredibly frustrating to have to carry two phones with different App Store accounts to be able to switch between the UK and US versions of things - Uber is one of the (vanishingly) few that gets this right and why they have consistently been my default choice for rides.

Maybe iOS could keep on hand the different versions you have, and automatically switch based on your current location? Generalize the "lazy loading" so app developers don't have to deal with the problem themselves

It sounds like app developers would still have to deal with the problem, and pray that the OS gets the location logic right

You don't need to carry two phones, you can sign out from App Store and then sign in with the other country account to install new apps. Updating works for both accounts even if the second country account is not currently logged in.

Pretty bad UX, it's clear it wasn't designed for this use case, you have to go into Settings, but it works if needed.

I also worked for Uber but on backend systems in the payments flow. Every single piece of app functionality mentioned above has at least one, if not multiple, backend systems and teams supporting it.

Like in the app, "payments" sounds like it would be a simple Stripe integration but it isn't. For one, that's way too expensive at Uber's scale. They build their own payment integrations to save fractions of a point that adds up to tens of millions of dollars in saved expenses.

There's real time systems to facilitate the trip, then the order processing systems to process that raw data, hydrate it, store it in multiple data warehouses, and publish it to Kafka to be consumed by literally hundreds of other services. The main money pipeline involved many services covering things such as rider incentives, driver incentives, rider fraud, driver fraud, tax calculations, rider payments across all of those aforementioned payment methods, receipts, invoices, driver disbursements, rider support, driver support, and more.

Then take in to account that almost all of those services were written for a world designed only for taking passengers from point A to point B with Uber Black but have huge added bloat to accommodate:

* Uber X

* Uber Pool

* Uber Eats

* Uber Freight

* Uber Moto

* Lime

* Jump

and proof of concept work for all sorts of new business ventures like Uber Elevate and ATG. All of that cruft is expensive to maintain and all of those actual product lines started as experiments.

When I left in 2019, there were two versions of the vast majority of those systems in production as all were being rewritten to handle the significant changes to Uber's business model from inception to now.

Additional edit: These services also require massive numbers of server hosts. All of the old stuff was written in Python, all the new stuff is in Go. That alone is saving Uber tens of millions per year just in hosting costs.

It genuinely blows my mind anyone who has written software couldn't have guessed at all of this from the beginning.

Uber PMs and engineers should be super proud. The fact that each user thinks that their one use-case is "all the app does" is fucking brilliant product design.

I've had over a thousand Uber rides (used it to commute) in 10+ countries and it's obvious there the way it seamlessly adjusts to each region but even otherwise I think I could have guessed just from the difference between SF and Seattle. There was a sort of joy of opening the app in a new region and finding unique options. Some sort of (maybe intentional) superstar-level product design.

Rickshaws in Seattle? Cash payment in India? Uber Taxi in SF? Sweet!

> The fact that each user thinks that their one use-case is "all the app does" is fucking brilliant product design.

Alternatively: The fact that they try to bundle all the world's use cases in some 100+ MB monstrosity of a client app is a tragicomic commentary on the misaligned incentives in the tech world's VC-driven wannabe-monopolies.

Anything else seems like an even bigger mess:

What happens when a user who travels unexpectedly discovers they need to download a new version of the app for where they are now? And what if they're on a low-bandwidth or capped cell plan and can't download a 100MB app easily, where 3/4 of the data is duplicates of stuff in their other 5 copies of the app?

What happens when a user tries to book an Uber from one area with a special app version to another area with a different special app version? This could easily generate tens of thousands of permutations.

What happens if the user is physically in one country right now, but is a national of another country and wants to pay for their Uber in their usual currency and payment method?

If the user ends up needing to have 20 copies of the app for various locations, how well will they sync ride history, changes in payment methods, and other such info?

I don't even work for Uber, and I thought of all of these complications in 10 minutes. The real Uber probably has 100x more complications that would arise from splitting location-specific functionality into location-specific apps.

Initial install bundle should be minimal and only have code that is required everywhere and or for downloading/installing more stuff.

It's not about creating 10,000 different permutations of an app, but downloading/installing content on demand.

If we are afraid that there might be problem that user later on has not enough bandwidth or issues like that, we'll start immediately downloading/installing after initial bundle in order we think this user might need them.

This means much faster initial usage, much less bandwidth used in total.

Same with developing experience. You shouldn't have to compile code that is not important for your development exp. This way compiling wouldn't take much time, dev productivity would be increased tremendously.

You're not always allowed to do that. E.g. Apple requires all code to either be in the bundle or run in a webview (and even then you risk removal if you introduce "significant" features): https://developer.apple.com/app-store/review/guidelines/#2.5...

For Android purposes: maybe it's allowed in the Play store? I'm not sure tbh. There are certainly other stores with other rules (and Uber is in many of them too), or they could distribute it themselves (but we can see how well that has worked for e.g. Fortnite), but you still need to handle any scenarios where it's required.

There's also something to be said about supporting 60+ versions of a client...

Not following that thoughtline?

By downloading dynamic code, in effect you're running N (or some combinatorial number of) different apps.

Granted, you can/should strongly isolate them so it's infeasible for one module to interfere with another. But as always, YMMV in practice. And good luck testing them all either way.

Sounds like base things, common for everything should be done using native app code and everything that varies that much should be made as PWA in WebView. You can download this PWA in the background too once app has installed if you are afraid you might not be able to do this on demand. I don't see why not use PWA? The performance implications in this case are minimal imo. And you have all the other fully fledged native features since your app is combination of both. Main arguments against PWA is that iOS doesn't support that well, but hell in this case you skip ALL problems with PWA since you have full capabilities already thanks to being generally native and you are downloading it from AppStore anyway.

In this case it could easily be reused within both Android and iOS without having to create duplicate business logic which I think already is a nightmare.

Airports logic I think should still be something that is defined using JSON/XML or w/e and algorithms should handle parsing that.

Or come on, there must be a way to get airports logic in an easier way. Does user have to update the app every-time when there's some minor airport update?

PWA won't give you access to random device APIs that you didn't open up for a specific use before needing them, which e.g. cameras, payment systems, etc often need. All that still needs to be native, and that can be substantial in some cases.

If it does, say you made a generic shim, you're effectively downloading and executing dynamic code. Maybe not by a technical definition, but that's not the point of Apple's rule, nor will they care about technical arguments when they reject it. There are plenty of examples of this happening in practice, I'm not talking hypothetically - it's why they have a carve-out explicitly for education apps. Even if you don't do stuff like that, and stay entirely in the webview, PWAs (unless it's actually being run in the browser, of course) are not allowed to make significant changes without going back through review.

edit: you can absolutely shrink your binary size with a PWA or similar though, yeah. There will definitely be other tradeoffs you have to make though, I'm not deeply familiar with those. Performance and integration with native libraries at the very least (e.g. map rendering).

No, that's why it's beautiful product design. They know something about me. I am at location A and I have a problem: I want to be at B. They then solve that problem smoothly with reasonable defaults. They know this is the prime problem I have and that I have some top few variants on this.

Speaking of the defaults, the auto-suggests these guys have are fucking ridiculous too. Both Lyft and Uber. It's 4 AM in the morning and I've opened the app at home: offers me a ride to SFO. It's 11 PM in the evening on a Friday/Saturday and it offers me a ride to my friend's place. It's midnight and I'm at that friend's? They offer me a ride to my favourite club. I am one-click touching everywhere. These guys are good.

Superb user functionality.

What solution do you imagine that isn't a worse user experience? I can think of lots of obvious ways to simplify the client, but they all seem likely to make things worse for me as a user in pretty predictable ways.

If anyone has a good reason to bundle every location’s use case into one app it’s Uber. A common Uber user scenario is someone who takes an Uber to the airport in one country, arrives in another and has poor internet capability for one of several reasons (roaming, poor mobile service, must connect to airport wifi, etc).

thanks for sharing. I'm a former Uber engineer still reading in most Uber-related reddit threads how they could build the app in a weekend

There needs to be a game of Uber HN/Reddit thread BINGO.

Squares include:

* Uber being a simple CRUD app

* Uber having no moat because it can easily be disrupted by those able to build the app over a weekend, because Uber is a simple CRUD app

* Uber continuing to operate solely because VC's currently subsidize every single ride

* Medallion taxis being safer because they're more rigorously inspected.

Free space in the middle: drivers have no clue as to their costs and are only signing up because they're too naive to realize they're losing money.

Just because it's a spot on your bingo card does not mean it's not true.

My experience with all this is that each time I opened the Uber app, I actually did not know what I was going to find. Taking an Uber a few times a month meant finding new UX patterns every time, sometimes features would be there, sometimes they wouldn't. It was very annoying to find that useful features had suddenly disappeared. After feeling that the UX was hostile due to this, and some IRL hostile experiences with drivers with a terrible support follow-up, I just dropped Uber for good. Alternative apps were more expensive but absolutely more stable.

Interesting. I assume other multinational apps run into this problem as well. Is Apple considering support for apps with country-dependent payloads in the future? Otherwise seems like a lot of bloat across all major apps.

It seems like most of these things should be downloaded on demand or at least after the initial install and in background.

Same with dev experience. You shouldn't have to compile things you are not using to dev.

This would give pretty much infinite scaling.

I do think Uber requires tons of engineers, but I disagree that all of this has to be downloaded immediately or bundle size has to be this big.

All of what can be logic wise should be server side. For instance Airport logic seems like one.

Was just about to link to your tweet on this subject from the other day. You saved me the search!

I have to say, this comment is quite ridiculous. Might as well say that Google is simple because it's just an app with one search field.

I never work for Uber but I can easily imagine the following problems:

* Map data is not a solved problem. Tons of inaccurate data especially on non-english speaking countries.

* As someone mentioned about payments, different country requires different payment platform integrations.

* Payment is just one aspect of money, you need to build a treasure-like logic to run a marketplace between drivers and passengers.

* The matchmaking algorithm itself is interesting enough problem. It has to be efficient for the company to make the most amount of money.

* Tons of internal data are produced and required to be understood. This requires ML/data analytic talents.

The items you've listed don't appear to be things that would contribute directly to the size (in MB as installed on the device) of the app. I don't think GP was making a comment about the complexity of the product, just the size of the installed app.

Isn't all of that server-side?

Not as per the long twitter thread.

Well... It should be.

That is a trade off with some pretty deep implications and some subtleties. I doubt it's such an obvious choice.

I've been thinking about pushing more in my client make less work for my servers.

Evaluating map data inaccuracy? More than just a payment UI? Driver/passenger matching? Data analytics? (Doesn't that require information from multiple clients?) On the client?

I mean, you could push some of that to the client, but I'd understand why your app is gigantic and doesn't do app-ly job very well with all the background processing. And I'd be surprised the app hasn't been hacked to get at the analytics data, if not the payments.

And how big is Google's web search bundle?

Most of what you mentioned should be server side.

It seems a little strange to criticize someone for asking a question on a site where most users seem to value curiosity. Not everyone has a background that makes it as easy to imagine problems like these, and while you've given some possibilities, I would also be interested in knowing exactly what accounts for the size.

There are far better ways to ask the question than “What does the app even do, just seems like a map with a few dots” which has a heavy implication on the quality of the team that built it..

You may be reading in to it too much. A lot of people would be surprised that Uber is as large as 30 SNES games.

A negative / unpopular take: like air in a balloon a startup company will grow as large as its capital allows as it is purposefully using this capital to grow faster than natural market forces allow. In other words it has a big engineering team because it raised capital to get a big engineering team. Craigslist is a great example of bucking the trend in this regard.

A more positive take: if you look at the specifics of how many regions Uber is (was) in, and that each region has cities each of which have wildly different regulatory environments, you can imagine the permutations of complexity they had to deal with. Then you add in growing internal business experiments like Eats or mobility (Jump) and it's easy to see how this balloons out of control pretty fast.

I think a combination of these is true. Uber certainly deals with more complexity in mapping than almost anyone. Their scale is massive. That being said: their engineering scale is massive as well. And if you have enough people, they will start adding things just for the sake of being busy. There is certainly someone at Uber writing code that is included in the app bundle that never runs. There are certainly assets being shipped that were relevant in 2016 but have since been forgotten. There is an engineer messing with some low-level powerful API for no real reason other than they find it interesting, and hurting the performance of something or the other by an order of magnitude. For political reasons, there is some non-optimal code shipping somewhere. At extremely large team sizes, it can be very difficult to find and fix these issues. But this is a social, not a technical, problem.

There’s a lot of complexity behind making the user experience as simple as it is. One example: it’s my understanding that there is a lot of effort put into correcting for GPS error in areas like cities so the the little pickup location dot is in the right place. I imagine the entire system for that is a fair amount of offline processing, backend and client code.

From one of his replies on Twitter:

> The app looks simple from the user perspective. But on the global scale the business rules are insanely complicated. Every region has custom rules/regulations. Different products have different workflows. Some regions have cash payments. Every airport has different pick rules...

I was part of the team in Amsterdam on this. Insane days. Had some great engineers we hired in AMS. We were moving the whole source code into a monorepo as well which was another crazy project. I think if we had not had some of the engineers we did at those exact times, we would have not gotten out of it.

Problem here with Swift came down to

- Apple not dogfooding its own tech . Hell they did not even help us or other major companies like AirBnB and LihnkedIn (We formed a workforce on this problem with them which then forced the download limit to be upped by apple)

- Engineers deciding to adopt a language which they thought was great (and turned out not to be at the time) because they did not do the right analysis upfront for such a significant project. 'Wow look at this shiny new cool tool, lets use it' (Oversimpiification, swift was more pleasant to write with and you could be more productive which was no doubt a factor here)

Also, while management at the time didnt care for the remote offices much, I think the AMS team played a critical role in the success here. With the recent layoffs that occurred, I heard that they were fired.

Another fun story, we were trying to bring down our build times on our CI builds which were 1hr+ while battling macstadium and their crappy infrastructure with tens/hundreds of patches incoming each day.

Appreciate your insight, and I'm sorry to hear about the AMS team.

The OP indicates that a rewrite was necessary regardless. Even if Swift was not yet mature, do you think it was more worthwhile than rewriting in ObjC? Given that the goal was to "sustain mobile development at Uber for the next 5 years".

Secondly, I'm not a mobile developer so I don't have much context, and I'm wondering about your outlook on iOS development as a whole. Swift obviously had major problems but has supposedly gotten much better. You mention MacStadium's issues, but Apple build servers seem to be making progress with the recent AWS EC2 Mac reveal.

Is the iOS landscape turning a corner, or do you expect a new generation of bad developer experience?

I was not part of the initial analysis team so cannot really comment if objc was more suitable as it would require to look at objc at that time and do tests etc but I suspect we would have had a more smoother experience. Though the the code at the time was bad and indeed something needed to happen to ensure future scalability of the teams and product/code, but I think there were more sane paths we could have taken. It would have cost the company more time/money and at that time the company was battling Grab, GoJek, didi, Lyft etc so we did not have time to lose. Lots of factors to take in to consideration. No real easy answer

I havent dabbled in mobile developer platform tooling for the last few years now so I cannot comment on the iOS side, all i remember is that back then it was a shitshow :) Mobile is still playing catch up to how backend infra and tooling operates, I suspect its better but probably still got a long way to go.

We could of made it work. I think the one big benefit of the rewrite was how the architecture and nullable types basically gave us one more 9 of crash free rate, which we saw after the release with swift. With ObjC I could forsee how it would of crashed a bit more, because nullables aren't hard enforced in ObjC, but I could also see how we could of stepped around that by strong guarantees in the model & network service generation code.

A complete rewrite of a large project is risky anyway. Doing so in a language in which literally no other project of that size exists yet strikes me as insane.

I love Swift. But I've been adopting it in bits and pieces over the years. It's not just that the language has changed significantly, you also have to get used to the idioms. E.g. my early Swift code often featured Pyramid of Doom patterns, and both language innovations and better familiarity with the idioms have helped a lot with this.

One thing I was surprised not to see was a mention of their Swift 2 to 3 migration, which was terrible in my experience working on apps that were nowhere near the complexity of Uber.

Apple should have never pushed Swift as hard as it did at the time. Between the constant API changes and issues like requiring dynamic framework linkage and copying the whole runtime into every app, it was absolutely not production ready.

It was a lot of work, but we had enough staff to make it someones job for half a year or a quarter, depending on how bad the migration is. With big codebases there are other issues that pop up with every new major Xcode release that we have to migrate, mitigate or fix. For example we are still at xcode 11 because of various bugs in xcode 12. Doing the migration was annoying, but it was not much compared to all the other issues, because the fixes were tractable.

That's why I call idioms idiots. If you have to learn idioms, it's an idiotic thing.

What do you propose as an alternative?

There is no language where full expression is possible without use of idioms that don't have direct translations in every other language.

The Mac CI situation has gotten much worse this year, but Swift has gotten much better. Writing a large app in Swift is still a major undertaking, but there are people doing it these days.

As someone who was there, a rewrite was necessary, and with hindsight we should of done it with objective-c. The original obj-c app was built with under 10 iOS engineers in mind, and now Uber had 100s of mobile engineers work on one app in some form or another.

We didn't do a simple build scalability test until we were well into the project. If we did it would of revealed swift's build problems to us. Our swift experience is what slowed down our kotlin migration significantly. Today Uber android is still a majority Java app.

Even very recently the experience of working in swift compared to the old obj-c code base isn't as good. To this day there are debugger issues, xcode responsiveness issues and a slower build time.

Hiring on the other hand would be harder now, because it's getting hard to find people who know Objective-C and it's surface ugliness scares new people away. Swift is a decent language if you don't have to have a very large code base, so most people rightfully so don't have that much Objective-C, and if you were to work in Obj-C, you might as well go work at Facebook then.

If by OP you mean me, then let’s be clear. I didn’t think the rewrite was necessary.

True story , when Swift was announced I started cheering so loudly in the office I had to be told to calm down.

I hate objective C with a passion. It's like some archaic language of a bygone era forced to run on a cell phone

I hate to break it to you, but almost all the code running on your iPhone right now was written in Objective-C and C and C++.

You can enjoy coffee without grinding your own beans.

I personally find lower level programing to be frustrating and difficult. I've built an amazing career with JavaScript and C# primarily. My hobbyist projects use C# , JS and Flutter.

I can't wrap my head around things like pointers or memory management

post-ARC, Objective-C and Swift aren't very different in terms of memory management. retain and release are handled for you, and you only need to worry about retain cycles. structs in Swift change things a bit, but not that much

With swift & obj-c your coding against the same library APIs, so in many ways it's a very similar level of programming. In many ways swift is far more complicated than Obj-C. With Obj-C iOS apps your not really writing that much C vs. the pseudo-smalltalk attachment to it.

I was an iOS engineer at Airbnb when we were battling binary size increases caused by Swift. I rewrote the watch app twice; once to migrate to watchOS 2 and once to convert it all to ObjC (which saved ~9mb). While Swift does have its problems (and had a lot more back when we worked on this), there are many factors to why the Uber app (and Airbnb) is getting so large.

Airbnb is 75% executables, 9.5% assets and 8% localizations. This is pretty different from Uber which is 60.7% executables, 26% localizations and only 3.8% assets. While the Uber executables are large, ~10mb more than Airbnb (note: this is only the Uber rider app, while Airbnb has host+guest in one app), their localizations seem to be driving a lot of the app’s 300+mb install size.

Looking at how the localizations are laid out in the app bundle, there are thousands of *.strings files in separate bundles, seems to be one for each feature. Many of these files have comments to provide context for the strings. An example from the app is: "Various screens show this as a title in a snackbar message when there is no network connectivity available". Just stripping these comments out of the localization files would save about 17.6mb of the install size. Another side effect of splitting localizations into so many files is there are over 23k duplicated strings throughout these bundles.

While the Swift code size is part of the problem with Uber’s app size, it's not the only way to free up space. There's a lot of work that goes into bringing a large scale iOS app’s size down, and not many tools to help. I left my job at Airbnb recently to work on making it easier to visualize, monitor and reduce app size, please email me (noah [at] emergetools.com) if you might be interested!

How do you know all these facts about the Uber app? Can you inspect the binary?

Parts of the binary on the App Store are encrypted so we can only get limited information from it, but the whole app bundle is available and the rest of the files are not encrypted. We’re building a startup to help monitor, analyze and reduce app size and I was able to quickly grab these statistics from the tool we’re working on, here’s a link to a quick demo: https://www.youtube.com/watch?v=3x-hLxxUNm8

It is also quite easy to decrypt iOS applications using jailbroken devices.

Looking at the bundle now is not going to give you insight into the problems in the Uber app 4 years ago. Also you’re looking at the uncompressed install size, which was not the problem (over the air compressed download size was the limiting factor). Strings files are extremely compressible, encrypted binaries are not (Apple has done some work in subsequent years to make them more compressible but back then they were basically uncompressible).

Absolutely right! I left out some details to try to keep the post concise. 4 years ago at Airbnb we had the watch app problem, but now a Swift watch app wouldn’t be an issue since the latest OSs don’t require embedded Swift standard libraries. The tweet that started this thread cited the 300+mb size of Uber’s app, which looks like the current install size (not thinned), that’s why I used it as the benchmark in my explanation.

Lately Apple has been featuring size more prominently on the App Store, and it appears to be displaying the install size. This is also what users see when they look at which apps are taking up storage on their device in Settings. I think it’s important to reduce both, so users see a good number and downloads are fast/efficient. It is a very welcome change that binary encryption doesn’t negate all the gains from compression anymore. I had noticed it in the numbers but looked around for an official statement and didn’t find any, if you happen to know where this was announced I’d appreciate a link!

Yes, installed size is also an issue. And on iOS tens of thousands of small files will waste a lot of space. One may say that compared to photos and videos that's small potatoes but I disagree. Efficient use of resources, memory, bandwidth, battery, etc. matters.

I was the person that actually managed the binary size task force and proved that it was a business problem in the first place. Not all management didn’t care about AMS :)

The Amsterdam team 100% played a critical role in solving the issue (at least until apple bumped the limit). No way we would have stayed under it without you all.

Clarification: the engineers on the AMS team played a critical role.

What did you guys use on Android? Was it painful to have to maintain both apps?

Great thread.

> So said brilliant engineer in Amsterdam, built an annealing algorithm in the release build to reorder the optimization passes in such a way to as minimize size. This shaved a whooping 11 mbs off the total machine code size and bought us enough runway to keep development going.

> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs

This would scare me too.

> I had privately had the “we need to stop” conversation with my director. He told me that if this project fails he might as well pack his bags. The same was true for his boss all the way up to the VP. There was no way out.

This brings up an interesting point. The author attributes some decisions in the thread to the "sunk cost fallacy". The business costs were sunk, but if the employees and management are too afraid for their job to make the "cut your losses" decision, the costs are not really sunk, at least not in the minds of the right people.

This isn't sunk cost; the exec chain just has different incentives. They're hired for outcomes. It's their job to make it happen, not make it make sense.

> A bunch of people got promoted. We all breathed a sigh of relief. The 90 work weeks stopped for a few weeks.

When you move up, you move on; issues from then on are the next guy's problem. Exec dynamics are often "gather successes to me and push off failures to others" - sounds like some exec won.

>> This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs

> This would scare me too.

FWIW, I made it clear that this was also a great way to break the app :)

Uber also has a habit of solving problems by hiring a manager externally and building a brand new team of all new hires for them.

There's some upsides to that, but basically even though you're a W-2 employee, you're effectively a consultant in that arrangement. Your job is not secure.

Almost every large engineering team has someone with the skills to do terrible, but powerful things like these. You should be scared, not of those people or their work, but the situations in which these people become necessary. Strive to make it so that they are not needed. Too many engineering teams, especially at certain large companies (of which Uber is one), fail to understand this.

> annealing algorithm in the release build to reorder

This is a damn cool use of simulated annealing.

> > This terrified the Swift compiler engineers, they were worried that untested complier pass orders would expose untested bugs

> This would scare me too.

For what it's worth a few (10ish) years ago we did this with the J9 compiler at IBM (technically just the backed that was shared between Java, C++, etc). Same idea with simulated annealing. It did end up finding some really strange bugs, but they were bugs that were present regardless, we just hadn't found the right test case for it. We did end up getting some performance out of it, but not nearly as much as you'd expect. It mostly boiled down to running a few extra constant folding passes.

It sounds almost like it is a kind of fuzz-testing then; I'm interested in what percentage of problems uncovered were problems in the spec or undefined behavior, versus problems introduced by the simulated annealing itself.

I've lost track of the number of "biggest engineering disaster" stories I've read that begin "so we decided to rewrite it..."

The story arc is almost always the same: starts out very promising and moves quickly, then at some point the wheels come off but everyone keeps going.

Anyway, Joel Spolsky nailed it 20 years ago: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

When you write code like shit, the language doesn't matter. What Uber did was write a monstrosity with too many engineers and too many features without considering that was beyond their ability to deliver reliably. The product decision making was also clearly deficient and didn't help.

Rewriting is neither a panacea nor a problem if you completely understand what you started with and what you will wind up with and be brutally honest. Trying to combine a rewrite with major feature changes is extremely hard to pull off (I've done it a couple times with iOS apps, not fun, but ours came out well) and adding a new language at the same time is probably beyond any team's ability. A rewrite for good reason can be done, but not with 100's of engineers and major features changes, all done in a massive hurry; that's a recipe for disaster.

It can be quality code written by good engineers, but there are issues you won't see until you deal with big numbers. The problem here is that Swift seemed to be a cool language that it might work for a "Hello world" app. You won't notice any problem until the thing grows up to a certain point.

For example, I work in embedded and I am proposed with a new "X brand" Chinese Modem to integrate in our devices. I don't recommend it but since it costs less I'm pushed to evaluate it. I am handed 3 samples and they all work fine, so management convinced themselves that Brand X from China is the way to go.

Once in production the modems started to fail at a 10% rate (10 devices over 100 won't work). And that is a problem you only see only when manufacturing a certain quantity of devices.

Now there are delays, an ongoing battle with the manufacturer of modems and a unhappy customer (that's why I didn't recommended it).

We started with Swift 2.X as well but it became Swift 3 before we shipped anything; Swift 5.X now is much more mature and we have no issues. Trying to do too much with a new language plus all the other things was asking for trouble.

Great article, thank you for sharing this. Makes a good case that the right thing to do, from a business sense, is usually not to do the full rewrite. Rather, you should do slow, un-sexy work of refactoring and cleaning up while adding new and maintaining current features.

He's right on, but maybe people don't actually know what to do on those cases

A full rewrite is not a Ship of Theseus thing. You're just going to reverse-engineer your own thing in another thing. Unless it is unusable (quality-wise or maybe the only current version was written in APL for Solaris, who knows) there's no need for that.

Can Swift talk to ObjC? Great. Then do that. Attack the critical areas first

Yes sometimes a whole new architecture is needed, but then maybe you needed to reevaluate your earlier procedures and/or hires.

Couple things always to mind whenever this article is referenced.

Joel's example is Mozilla, but Mozilla ultimately became more successful than Netscape (in terms of userbase with Firefox). Was the problem really the rewrite, or that a browser company was doomed anyway and the rewrite exacerbated it?

Also the article was published just a year before Microsoft pulled off what's maybe the most successful large rewrite and customer migration in software history: Windows/DOS to Windows NT (XP was where it became complete).

I've always wished for Joel's take on NT, especially since he was at Microsoft previously and it happened just after this article was posted. What's his opinion on its success when so many weren't?

I don't think you can consider Windows NT to be a rewrite of DOS. They were different products entirely, and existed in the marketplace concurrently for upwards of a decade.

Microsoft took their entire userbase and successfully shifted them from DOS (and then Windows on DOS) to NT. They did it in a piecemeal fashion with e.g. Windows 95 and WIN32.

What else you would call it?

I would call it "different products entirely".

NT wasn't a rewrite, simply.

It's a completely different system, that happens to support the same API/ABI, backed by long experience at Microsoft at making things backwards compatible (it's probably the biggest source of Microsoft's success).

Microsoft also put ~7 years into preparing the transition piece by piece, with probably the biggest impact point being Windows 2000 that provided significant host of features that compelled big (and loud in complaints) clients over, pulling a lot of software vendors along.

Mozilla eventually beat IE only because Microsoft disbanded the IE team. The years of product stagnation due to the Mozilla/Firefox rewrite didn't matter in the end because AOL acted as a sugar daddy for a while and then IE was also not moving forward. The Spolski advice is for the more typical situation where your competitors are advancing.

The "curmudgeon" ending line in this story rubs me the wrong way. The ObjC people were right, basically, and it seems like the problem was essentially no one in the right place was going to drop Swift at that point.

Seems less like they weren't offering answers, so much as the answer wasn't "but we'll do some stuff and fix Swift!"

Though the 6 library limit and how that wasn't a gigantic red flag on the language is baffling.

Yeah it’s kind of bizarre that after writing everything leading up to that, the author would still frame the ObjC folks in that way. Like, this or that eng. was so brilliant with their hack, but all those obj-c eng. who advocated boring but stable solutions all along were just curmudgeons with no helpful advice.

The problem from the author's perspective is that the Obj-C "curmudgeons" were proving the right solution to save the app (drop Swift and cut losses + go back to a functioning language) but not the careers of the people who had gotten them into this terrible situation.

This perspective is reflexive. The "ObjC curmudgeons" didn't feel like the adventures that the "Swift zealots" were generating helped their careers in any way.

Sounded like a "both sides" analysis, where the assumption is that both sides in a disagreement must share blame.

The Middle Ground Fallacy

It's the developer equivalent of "support the troops or get out of the way!"

Don't ask why, just do or die.

But yeah, I would have been pissed off if I was put in this situation by myopic upper management.

We won't know because all we have is one glib comment from one person, but I took the comment on face value. I've definitely met the class of developer who is mainly interested in pointing out problems and a stick in the mud, and isnt interested in working together on solutions.

Yeah, but what are the odds of Uber having hired a whole raft of that class of dev?

They're fairly common, and any specific environment attracts those who favor that environment.

I think reasonably high!!!!

Wasn't really clear what benefit the rewrite offered anyway other than a new UI, which brought it's own problems and couldn't that have just been written in Obj-C?

I think he said developer productivity.

In my experience, the problem with scenarios like fight the ObjC folks in the story is that the "curmudgeons" don't see the need for anything to change. Their arguments are not "We should use ObjC and do X, Y, and Z to make sure the problems are fixed," but rather usually turn out to be something like "We should use ObjC (and throw out the rewrite entirely; everything was fine before)." Which at best only results in kicking the problem down the street a ways and at worst is not possible.

The emphasis could be on their behavior of 'complaining without providing much in the way of solutions'. Certainly, though obvious now, they could have empirically presented at least one legitimate case as to why Obj-C was a better choice (app size).

Indeed the ObjC people were right (I was one of them). But there is nothing more useless than standing in the corner and saying “I told you so”. That’s the curmudgeon.

Being right isn't enough. You need to be able to convince people of the fact that you're right.

Maybe as a junior engineer, or an intermediate one. But not beyond that. Seniority isn't just being right more often.

I'm reminded of the story of Windows NT and how David Cutler commented at one point after the release that divorces and nervous breakdowns were tracked in the program metrics. I expect that's hyperbole, but there is a real cost to these death marches.

I couldn't find that specific quote, though, and being 25 years back it could be a false memory of who said it (I did read it somewhere, if you were on the team at the time, please speak up!). Here are a couple of illustrations[0][1].

[0] http://read.pudn.com/downloads196/ebook/923902/ShowStopper!-...

[1] http://www.proudlyserving.com/archives/2007/09/dave_cutler_w...

I remember that some engineers on the iPhone project went through divorces as a result of working on it.

> I expect that's hyperbole, but there is a real cost to these death marches.

I used to be a game developer at EA. In the middle of one particularly nightmarish 60-hour-weeks-for-months crunch, a twenty-something artist with no known pre-existing conditions spontaneously died in his sleep.

I went through a pretty intense crunch time at a fortune 10.

We had one person go to the hospital with tachycardia and 3 people who were vomiting. There were only 10 people on the team.

I believe it. During my last stretch of unemployment I started tracking the days I couldn't get out of bed in my spreadsheets.

I find it fascinating that so much engineers (arguably « smart » people) are actively hurting themselves over a freaking job and acting as if it were just a fact of life.

Plenty of people throughout history have martyred themselves for what they believed in. Fascinating for sure but being smart is a completely different question.

I started reading the intro to that Show Stopper book and can't stop. Great stuff, thanks

If I could read a book of these stories I'd stop everything and read it now. Absolutely incredible. I love stories like these, but I also understand these hells are probably reserved for the special world of startups. The interesting takeaway from this for me is the question "Was the rewrite successful?".

As an engineer, I understand I'm a vehicle for success at a startup. Even if I ultimately get burnt out and quit. If I make important changes, that maybe only I could have thought of before I quit, it probably becomes a good investment to hire me.

Uber managed to hire enough of the "right" vehicles for success to get to the point where they shipped a platform rewrite at massive scale. By all high level objective metrics, this is probably success story.

The one metric I think could derail the projects success isn't really documented--what was the human cost to this effort? I bet you looking at a graph of turnover at Uber you could probably identify exactly when this project happened. I'm curious if the engineering turnover from such a massive effort was enough to offset the benefit.

In that case I can recommend the book the phoenix project.

The engineering org had doubled in sized almost every year prior ... Teams began shifting all their focus to bringing their features to the new app ... Apple’s recommend maximum number of libraries in a single binary was 6. We had 92 and counting ... But as the app kept growing. Soon we hit the cellar [sic] download limit (100 mb) for our universal binaries ... All this time the Swift code growth continued ... discovered that our compiled code size was growing a rate of 1.3 mb a week ... But the real problem was the growth curve. It was always eating our winnings back.

This is one of the main reasons why I have no intention of going back to BigTech any time soon. I am sure the author is an extremely capable engineer but he is obviously missing the elephant in the room: that the root problem is app growth and feature creep caused by uncontrolled growth of the eng. workforce (who are all incentivized to ship their code because it's good for promotion). As if app growth was some kind of god-given fate that can't be changed! But instead of fighting the real (organizational, political) issues within the company, they blame it on technical issues (Swift is not mature! 100 MB app size limit is too low! etc.). Incredible.

The larger community benefited from our learnings.

No. The larger community would have benefited if you had written a story about how someone in the org. stood up to management, fought the feature creep, the crazy app growth and doubling eng. each year, and brought the whole project back to sanity. But that didn't happen.

So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.

No disrespect, but no choice / tradeoff in the world would have led to a satisfactory outcome if you have hundreds of engineers burying this app under an avalanche of code day after day. The tools / tech are alright. The company is not.

While I think your tone is a bit more critical than it needs to be, as many very intelligent people have made the same mistake, I think the sentiment is correct. But 'standing up to management' is literally not possible if they don't want it to be. They get to make the decisions if they want to, more or less by definition of 'management'. He did say 'we need to bail' to his boss at one point.

But, essentially, your final paragraph's point is the bottom line. This was not fundamentally a tech problem, it was a too-many-cooks-in-the-kitchen problem. That's a business decision.

What you're ignoring is that some significant percentage of that "bloat" is what leads to more rides and more money. All those tweaks and a/b tests and features that solve problems for specific segments that most users won't ever see, getting shipped week after week, increasing the new user conversion rate by 0.5% here, reducing the rate of canceled trips by 0.3% there, it all adds up over time. Many of the features obviously fail too and don't improve anything they just add bloat, but you can't ever know for sure which features those will be in advance.

In aggregate, the goal is obviously that it leads to the company making more money than was lost by paying all those engineers to move mountains to get around all the issues they hit up against. In the case of Uber, now a $96 billion public company and worth nearly double their 2016 valuation, it's a little ridiculous to call it a failure.

You read this story and see an unmitigated disaster, but it's mostly just a picture of things working the way the company wants them to. The goal is to grow the company, not to have a nice, conceptually clean, tech-debt free codebase just for the sake of it.

It's also interesting to look back at the internal politics of Uber at this time. [1]

There's also a bunch of big articles on VPs quitting during this time. It was peak Uber controversy with the sexual harassment scandals, Waymo stolen IP, TK resigning, Fowler, spying, backdoors, etc. Reading the headlines in the algolia search is like a telenovela.

Reflecting on one very, very strange year at Uber (Susan Fowler) - https://news.ycombinator.com/item?id=13682022

I am an Uber survivor - https://medium.com/@amyvertino/my-name-is-not-amy-i-am-an-ub...

Uber Founder Travis Kalanick Resigns as C.E.O. - https://news.ycombinator.com/item?id=14600873

How Uber Used Secret “Greyball” Tool to Deceive Authorities Worldwide - https://news.ycombinator.com/item?id=13785564

Suicide of an Uber engineer: Widow blames job stress - https://news.ycombinator.com/item?id=14200486

Uber's Design Meltdown - https://news.ycombinator.com/item?id=11109272

Uber employees used the platform to stalk celebrities and their exes - https://news.ycombinator.com/item?id=13163895

[1] https://hn.algolia.com/?dateEnd=1512691200&dateRange=custom&...

It’s interesting to me that you came away from this thinking that I didn’t realize the eng size growth was the problem. I specifically said in the third tweet that hypergrowth was the problem. I even went on to say why it was a problem. The entire point of including the details about the org growth was specifically to draw you to that conclusion.

OK, I understand. I didn't mean to imply that you didn't see the problem of hypergrowth and I apologize if it came across this way. However, in the sequence of tweets it does sound like the conclusion / lesson is to throw ever more heroic engineering at the problem or to get better at eng. planning, as opposed to solving the organizational issues and messed up incentive structures at the root of it all.

I realize this isn't a small ask. I've worked in BigTech myself where the reward / risk ratio of trying to actually solve problems at the root is basically 0 if you're not SVP or higher. Anyway, I enjoyed the tweets nevertheless. Good luck.

It’s interesting that you saw this was a tale of heroics. I view it as a tale of bad choices and organization failure.

Readable version: https://threadreaderapp.com/thread/1336890442768547845.html

Thank you.

Readable and accessible to ex-twitter users.

Weird, that was what I originally posted but someone (dang?) changed it to Twitter.

> So my advice. Everything in Computer Science is a trade off. There is no universally superior language. Whatever you do, understand what the tradeoff are why you are making them. Don’t let it descend into a political war between opinionated factions.

I really like this point.

This is a good point, but it's important to be clear about it. There is no universally superior language. There are a number of competing factors which means that optimizing for one sacrifices others. So instead of a single point in space that maximizes all of them, there is instead a volume bounded by a curve and going farther in one direction requires lower values on other axes.

But, there is a lot of space nowhere near that trade-off optimizing boundary. You can absolutely make languages worse in ways that are strictly negative for all users. For historical reasons, all languages contain some amount of that dead weight. So the absense of a perfect language does not imply that all existing languages are right at the edge of the optimizing boundary.

I think not clearly understanding the tradeoffs was a huge factor here, but I would go one step deeper into that decision-making process.

One thing that could happen when making a decision like ObjC vs Swift is that the decision makers can think they have performed a sober risk/benefit analysis, but failed to dig deep enough to uncover show stoppers. So they may have looked at developer productivity with Swift and took a cursory look at build times for a small pilot, but failed to look deep enough to find some of the crippling shortcomings.

Related to the first point, a lot of the time with a new technology, the show stopper bugs/shortcomings/issues haven't been discovered yet. If you're choosing a battle-tested technology, most of the nasty edge cases and limitations have been discovered at some point. But with something new, there's a lot more uncertainty with what you'll run into once you put it into production.

That risk of surprises is higher with some new tools and technologies than others. But it's a big wildcard when you're betting your billion dollar business on it. Maybe that bet pays off, but you need to do some sober analysis on that risk of unknown unknowns and have some idea of how and when you'd make the decision to cut your losses.

Rewriting apps (esp. as large as Uber) is essentially your plan meeting the enemy. There's many unknown-unknowns simply because technology is complex, and although it's pretty much unprovable, the brisk pace of technology also means you'll have more unknown-unknowns. Only a big industry shift in priorities will change that now.

I also love this point.

It hints at a rule-of-thumb I use: try to wait making an important technical choice until you can explain a pro & con of each option (or, similarly, explain when you'd use each one instead of the other).

It was pretty obvious when Swift 1.0 launched that it should’ve been called Swift 1.0 beta 1. Not a knock against anyone who worked on the project, other than senior management at Apple.

Speaking for myself, I didn’t add any Swift to any codebase I worked on until after 3.0 shipped because it was pretty clear that the language just wasn’t quite ready yet.

I’m genuinely surprised no one ever thought to just email Chris Lattner or invite him out for a drink to say ‘hey, confidentially between you and me we’re thinking about rewriting our app in your new language. Are there any other apps with this order of magnitude of LOC written in Swift yet?’

Large Swift apps are easy to find; companies would invariably use it as a marketing point on their engineering blog. Even if not, people open up apps all the time, any large app that uses Swift would be found and discussed.

Large Swift apps are easy to find

Sure, if they exist. I'm not talking about the world as it is today in December 2020 with Swift 5.3. I'm talking about 2016.

My point was that there was no need to ask Chris Lattner back in 2016: you could just look around the landscape and see that there were not many, as they would be easy to see if they did exist.

Lots of companies don’t speak publicly about their product strategies, like certain former employers of his.

As I mentioned, there is no need for a company to publicly post details of their products for people to figure out what languages they were written in.

Making a huge decision and sticking with it -- blowing through all obstacles through workarounds and hacks -- because otherwise would be deemed to be a failure is the core issue here.

Uber's culture of "letting builders build" is fine, as long as teams have the ability to change things early and quickly. I think top-down decision making and monumental bloating of eng teams made things very difficult over time.

Management had very different incentives while engineers had different motives. The two were not aligned, and TK culture was still very prominent back then (work hard, dont go home early, we are going to be a 1 trillion dollar company etc etc) and Thuan definitely helped him spread that culture. So definitely not great days but I think the engineers took away a lot from their time there. Lots of lessons learned

Yeah, I imagine this pressure from the top had something to do with the "90 hour weeks" cited in the thread...I've never worked in an org as large as Uber, but personally I have made some very bad engineering choices when in crunch mode, especially on the doubling-down-on-bad-decisions front.

When there's no room to come up for air and look at a project from a perspective other than "this is super broken and way behind schedule" my brain just fixates on the micro-steps that need to be done to get it across the finish line (whether or not the finish line is realistically anywhere within sight). It's work being done, but none of the exhausted people working on a project like that have any energy left to consider whether it's the right work.

I started to write a whole big thing here because I found this whole thing pretty disgusting but it's not worth it.

> With only a week left we decided eat the 8 figures and drop support for iOS 8.

I'll say this: If I were an Uber investor I would be pretty angry. And the story just gets worse from there.

This is a story of gross incompetence. It's not an engineering disaster, it's a management disaster. This is classic "inmates running the asylum". ( https://www.goodreads.com/book/show/44098.The_Inmates_Are_Ru... )

This is the story of some hackers who got in good at a big company with lots of money and sturm und drang so they had no adult supervision while they partied while pretending to be actual engineers. (That they were pretending to themselves too doesn't make it any better.)

The writing was on the wall, but rather than own up and take responsibility they double down and keep messing things up for months and years.

> This is when the real brilliant engineers started to shine.

This is when they get pulled from other tasks to clean up your mess! (cf. Broken window fallacy.) Those real brilliant engineers should be doing other better things than dark rituals to appease the Swift compiler gods.

Look, we all make mistakes. There's a blog post from an engineer at a company I used to work for complaining about the pointless middleware I wrote years earlier. (I feel ya bro, I told them at the time that it was pointless.) Here's the thing: I got out of that place when I realized that the folks running it weren't really competent. (And, sure enough, they got pushed out by the investors and now the company is a "Wallstreet darling". So... Yay?)

I've burned enough investor money in my career that I don't think it's funny anymore. (I never really did think it was funny, I don't like waste, but I was young. REDACTED was only my second official job, and in hindsight it was really crazy. I mean, they built a skating half-pipe in the office! At the time, everybody thought that was real cool.)

One wonders if this is common everywhere at this scale. Take Facebook's massively bloated iOS app:

https://www.reddit.com/r/iOSProgramming/comments/6upeu6/how_...

https://news.ycombinator.com/item?id=14903228

And their Android app patching Dalvik at runtime to deal with the huge number of Java methods:

https://www.facebook.com/notes/facebook-engineering/under-th...

And they're not the only ones known for bloated apps, either. Take Amazon's apps for instance:

https://news.ycombinator.com/item?id=14902875

Misinformed take. Android natively supports what facebook did back then, because almost every app hits the dex limit.

Not an Android dev, just going off of how the FB comments there and elsewhere have responded to that article.

I suspect it is common. And mostly inevitable. When you have 10^N engineers working on something, protecting them from each other (and themselves) grows increasingly important. At some point, it becomes your main priority, as without it you have a totally non-functioning system.

Which partly implies that the only solution is to not grow to that scale. Keep a smaller, more skilled team instead. In most cases I think that'd be better, and I think most people (and companies) would agree (if only they could hire those skilled teams reliably)...

... but sometimes throwing more people at a problem really is the best (or only) option, e.g. when you have to deal with a large volume of ridiculous external constraints. For example: my first job was at a company that aggregated loan providers, which entailed consuming hundreds of random PDFs per day and dozens of weird RPC calls. We had dozens if not hundreds of hand-crafted bits of automation to detect when things changed and handle the new format. Many APIs had hand-rolled XML parsers and producers... because the company we were calling would choke if your attributes weren't in a specific order, or if X wasn't within a specific number of bytes from the start of the request, or they returned invalid XML. It was absolutely ridiculous, but there's no way a bunch of banks were going to fix their APIs for us, so there's not much of an option but to do it by hand.

One company I worked for a few years ago, I joined right as a just such a ground-up rewrite of their core system was coming into place. Everyone in orientation was telling me "You'll be spending most of your working life dealing with <new project codename>" but they were actually training me on the existing system. About a month out of orientation there is a company wide. They canceled <new project codename> because it wasn't meeting it's technical specs.

It was also, coincidentally, the company where I most respected the key technical decision-makers, because they had the will to discontinue a product that a few customers already were using, because it wasn't meeting its criteria.

This to me reads of Swift being pushed hard well before it was truely ready. Polynomial linker time? Wow. Ouch. No thank you.

In version 2.x, no less! What was it in 1.x, exponential?

Swift 2 was so, so horrible.

I really loved it and continue to from v3 onwards though.

Linkers must run in polynomial time for certain things; this is a problem independent of Swift.

That is true, but I am pretty sure this is misattributed. At the time we had two major issues which slowed down loading these sorts of dylibs.

One was attaching the code signatures (which gets attributed to the dynamic linker because it is synchronously stalling waiting for the the validation to occur).

The other was that reactive patterns tend to cause A LOT more classes and methods which result in a lot more fixups. When I am debugging apps I can often tell if they use react based one the ration of Objc/Swift metadata to __TEXT in the binary. That ended up causing dyld to stall waiting for page ins (which again are not really the linker but get attributed to it).

IOW, many of those clever patterns that make doing rapid development easier are really just a way to transfer work from the developer to the runtime, and it has a cost. Swift and SwiftUI are actually have some very clever ways to mitigate this, but reactive Swift libraries int he Swift 2.0 timeframe certainly did not.

Apple's botched introduction of Swift is easily responsible for billions of dollars in lost productivity across the industry since 2014.

"Botched introduction, really? Perhaps Apple could have been a bit more upfront about the state of the tooling (which, of course, they never will be) but I think they did a rather good job considering that they moved development out from "total secrecy" to "community-involved" within two years, all the while shipping new features and improvements to the experience at a fairly decent pace.

> “which, of course, they never will be”

That’s the problem. Apple knows by now that their developer community laps up absolutely everything they put out and regularly gets burned. (I’m old enough to have rewritten code for Cocoa Bindings in 2004. It’s the next big thing! And garbage collection in 2008. It will fix everything! And GCD... and...)

For such an enormous change as Swift, they should have switched off the reality distortion field, and gone with a more sober, more academic approach.

What's wrong with GCD?

It's a great library for a lot of things. When originally introduced it was also heavily oversold by Apple as a magic bullet for concurrency.

Here's one account: https://tclementdev.com/posts/what_went_wrong_with_the_libdi...

This page overstates the issues. Almost all the initial GCD design is still good, we just ended up with a lot more smaller low-core and non-SMP processors than expected.

Also most of the performance issues are with excess parallelism (runnable threads), not concurrency (hiding IO latencies).

Aside from ObjC garbage collection, those things are all still good. Bindings enabled you to create things like complex editable tabular UI without writing any code. And the recommendations about how to use GCD efficiently have changed over the years, but it's still the foundational solution to concurrency on Darwin platforms (including being the basis for the new async/await runtime for Swift).

Yes totally botched. We know because have can compare to Kotlin, a project which started around the same time. Google's migration to Kotlin has been progressing very well with no disaster stories like this one. Kotlin was engineered very carefully, developers love it, it scales well and importantly it can be incrementally added to existing codebases because interop is nearly perfect. So the risk to adopt it is far lower. Sounds like Uber got burned with Swift and now aren't really adopting Kotlin which is understandable but IMO a mistake.

I’m afraid it could easily be the biggest mistake made by Apple over the last decade. The bloatedness and slow build time just aren't justified by the shiny, convenient language features.

This assumes a major counter factual where all of the same work was somehow more productive otherwise. Seems dubious.

In some other thread someone who regularly attended meetups hosted by Uber in NY said this (paraphrasing): "You could rely on Uber coming up with the most complicated solution to problems they create for themselves. But the food was good" Pretty much sums up anything I have ever read about uber engineering.

This story at Uber is a fascinating example of the growing pains involved with transitioning to a new language.

LinkedIn adopted Swift back in 1.2. I would love to read some war stories of how things there have progressed over the years.

Source: https://www.youtube.com/watch?v=X9waDi787uo (14:15-15:30, 39:26-40:37)

At an interview with them a few years ago, I was told they had been moving towards rewriting the app in Objective-C.

the difference is people actually need to use the Uber app /s

Many engineering horror stories involve an entire system rewrite. My recommendation is to never rewrite a large system. Literally never. If the people proposing the rewrite aren’t capable of adding the features incrementally, they aren’t capable of rewriting the entire system to “easily” support those new features.

This 100%. From the tweets it sounds like the problem they were trying to solve was to make the app more modular so it would scale better to more devs. There were/are well established patterns for doing so in ObjectiveC. The Swift rewrite was not necessary and probably took a very difficult project and turned it into the death march that the OP describes.

They should have solved the architectural problem incrementally without adding a 1.0 language rewrite to the mix and then slowly rebuilt the app using Swift.

This is so obvious to anyone with experience it seems pretty evident that the decision making process there at the time was deeply flawed.

I shudder at the thought of hundreds or thousands of Uber engineers banging out a rewrite together. I've been through a number of rewrites, and every single time it always meant the same thing: get your resume ready and look for the exit. It's going to be a multi-year shitshow.

I've spent most of my career doing rewrites, each time reluctantly, and each time it was a resounding success.

So easy with the absolutes :-)

Point taken but something tells me we might have different ideas about what it means for a rewrite to succeed.

Hmmm...in one case:

- Performance (which was a major driver) improved 100x-1000x.

- Reliability (other major driver) improved ~100x

- Codebase reduced, features increased, habitability dramatically improved

- Resource usage redcued ~10-20x

https://link.springer.com/chapter/10.1007%2F978-1-4614-9299-...

What would you consider a success?

Binary size should have been treated like a limited resource: set a hard limit in the test suite, and let the engineers who commit code in the organization fight it out how to delete unused code when they create new one for a new feature.

I'm following George Hotz's tinygrad that is a CPU+GPU deep learning framework under 1000 lines of code with great interest where all engineers are trying to shave lines of code while maintaining readable code (it's like a game when you set rules):

https://github.com/geohot/tinygrad

Here's the GPU ops part:

https://github.com/geohot/tinygrad/blob/master/tinygrad/ops_...

Definitely thought this was going to be a story about when their car killed someone: https://www.nytimes.com/2018/03/19/technology/uber-driverles...

I doubt any engineer who worked on that would be even remotely allowed to talk about it.

Were all of these.....tweets originally? Or does the author write in this crazy "few sentences per paragraph then break" style? So hard to read.

nailed it: https://twitter.com/StanTwinB/status/1336890442768547845

Yes, threadreader.app concatenates a thread of tweet replies, and each section is clickable, linking to the original tweet.

Yep, and it's why Twitter as a blogging platform will never take off. I don't understand why people do that. Just find some place to blog and post a link to that in your twitter feed. Gah! It's so frustrating when people abuse a platform like that...

Or write it in word and post a screenshot in your twitter feed :-)

Great thread. Thanks for sharing!

It's interesting to me that the author feels that things were great with Uber in 2016 pre-Trump and that Trump's election was the catalyst for the negative sentiment. Susan Fowler's famous post was early 2017, and maybe _posting_ it was catalyzed by the issues surrounding Trump, but clearly the serious cultural problems she described at Uber had been going on for some time before that.

I'm trying to figure out what to make about a lot of these mis-steps. On the one hand, many of them seem preventable: you could imagine identifying ahead of time that load time and binary size might be risks for the new app and scale-testing these very early in the process to de-risk it. You could imagine testing the new app long enough (and widely enough) to discover the location issues (drivers going to the wrong blocks, etc.). You could imagine foreseeing users being upset about suddenly needing their location data when they're not using the app. Obviously hindsight is 20/20, but is there also a pattern (that we see all the time, especially in Silicon Valley it seems) of not really thinking through the consequences of a lot of decisions, especially where things like user privacy are concerned?

I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same. People saw the issues that were placed on a national stage, then looked inwards and saw similar issues much closer to home, and these things started coming out as people were no longer OK with keeping quiet about it.

For the other half of your comment: I personally think many companies have issues focusing on non-technical problems. From the thread, the issues were clearly that they were writing far too much code, it seems like they were not testing enough, they lacked a person who would tell them how their changes would be perceived by the public. This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others. And you can see the solutions: while highly technical and "awesome", they are obvious band-aids. It's just that it's much easier to slap on technical band-aids than fix the root, non-technical problem.

> I think the election of Trump may have made it possible for these cultural problems to come to light, the same way that (for example) women speaking up about sexual assault can empower others to do the same.

> I personally think many companies have issues focusing on non-technical problems...This is not unique to Uber in any way, these problems show up to various degrees across companies such as Apple, Google, Facebook, Microsoft and others.

I agree with all that. To me, this is about taking responsibility. Our industry has shown a pattern of thinking poorly about non-technical problems while rolling out changes that affect millions of people. That's part of why so many people are angry at the tech industry -- and for good reason.

The thread is thoughtful in a lot of ways, but I get a whiff of not-taking-responsibility through some of it. The first example struck me for whatever reason. I get that people in 2016 didn't necessarily know what was going on or how bad it was. But I would expect that in 2020, we'd look back at 2016 as the time when serious issues beneath the surface were about to explode as a result of choices made at Uber, not some golden time before external forces made life hard for Uber. I don't mean to be too hard on the author or even Uber (and after all, this was a casual tweet thread). I think it's a widespread challenge for our culture in the tech industry.

Yes, I think that many engineers can often have a hard time understanding that problems that occur at their workplace don't exist because the media changed attitudes but because it shone a light at internal issues. I remember talking to someone at Facebook privacy and they were somewhat indignant that the media was making them out for being such a privacy nightmare–ignoring the fact, of course, that the tide was turning against surveillance and data gathering in general and Facebook was just one of the most prominent examples of what everyone hated ;)

Computing is always susceptible to shiny object syndrome. Just because it’s new doesn’t make it better. I wonder if things would be different with a more nuanced approach to Swift adoption, knowing what they knew of its limitations already, and being realistic that the language is not a panacea (none are!)

I know C++ has its warts, but almost every mobile project I've worked on has been C++ with a thin layer of objective c and java interop. IMO it's really not worth it to get so tied up in a single companies tooling even if the developer experience is superficially nicer. C++ has all sorts of awful, but it's familiar awful with mature tooling.

Admittedly the things I've worked on has been games and CAD systems so the internal data models and simulation are a lot more complicated than the UI, but I think that holds for a lot of apps.

The vast majority of apps are UI, CRUD, business logic and other things where native code (as in compiled C/C++) is just overkill and makes things more complicated. The few exceptions seem to be the fields you listed.

> Some of my ex-Twitter colleges called on me to quit such an evil company that would track you like this.

Twitter owns an company called MoPub that broadcasts user location information to the adtech industry.

The biggest engineering disaster Uber had was when it intentionally disabled the automatic safety brakes on their 'self driving' car and killed a person.

I don't know why you're getting downvoted. Making a crappy, slow app is nothing compared to conciously deciding to switch off the built-in autonomous breaking system in lieu of your own(which then doesn't work and kills a person). That's an engineering disaster, not a taxi going to the wrong city block a few times.

Considering that the engineer describes his story as "the biggest engineering disaster I’ve ever had the misfortune of being involved in," I don't think it is relevant to his story that the self-driving pedestrian collision occurred. He isn't claiming--and even from the thread title, I wouldn't assume he is claiming--that this is the biggest engineering disaster ever, or even the biggest engineering disaster Uber faced.

Yeah, I suppose that's fair enough.

They bloated the app so that it wouldn't be slow. The whole twitter thread is about that.

Less a engineering disaster and more a human one.

I was working at Uber ATG when Elaine Herzberg was killed. They kept the details secret from the other employees, citing fears of "too many cooks". I recall being offered grief counseling.

Uber disabled Volvo's automatic emergency braking system, which probably would have saved Elaine, to develop their system on public streets. I left the company and took a pay cut to work on ethical robotics instead.

https://www.washingtonpost.com/local/trafficandcommuting/ped...

Whoa. They disabled Volvo safety system. That’s nuts. My Toyota 2016 Rav has saved me a couple of times with the auto brakes and loud beeps. It was totally worth the 2k extra for the 7 radars and 6 cameras.

I don’t care about L4 self driving. I really care about a car that is resilient to human mistakes and fatigue. Uber failed at that.

"Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience. So to improve location pickup we changed the location permission to collect signal in the background so we could send the drivers to your current location. People freaked out."

This doesn't make much sense. When you open the app to schedule a pickup, and you have location tracking enabled while you use the app, Uber should be apprised as to your current location within seconds. Where you were before you opened the app should not matter.

It would be useful if the author could clarify this.

As one of the people who started a petition to Apple to disallow developers from disabling location tracking only while using the app (i.e. all the time or never, which Uber tried to force at the time), I am partly responsible for the outcome. But from what I can tell, no customers were worse off as a result.

If Uber has data showing otherwise, that would be interesting.

GPS fixes improve over time as the ephemeris gets downloaded and updated so what he claims makes sense. A full ephemeris takes 30 seconds to broadcast from each satellite.

Sounds like a type of kalman filter on the location. Any given reading is likely crap, but combined, you can get fairly accurate.

So, the idea is probably too keep a decent sized history of readings so that your could just add the current to that.

It’s not about collecting info before you open the app. What happens with most people is they open the app, request a ride, then close the app again and do something else until the driver has arrived notification. Location services start out with a wider range and narrow in on you over time, so a longer collection window gives more precision.

My mind is constantly blown away that Uber app needs to be so big.

Does it blow your mind that Google Maps is ~190MB?

Contains more functionality than Uber at a smaller size.

Uber app contains very similar functionality, namely real-time map view that needs to be able to render any part of the globe (or at least serviceable area), a location search engine, the ability to render drivers around you in real-time.

Just like Google Maps lets you select point A-B directions across cars, walking, public transit, Uber gives you multiple products to select (UberX, UberPOOL), and renders the path on the app accordingly.

The Uber app also does a lot of extra computation at the app side to solve some of the business problems that the OP mentioned in the thread. For example (from the Twitter thread):

> Without manual pickup location entry people’s location would just show up as whatever the GPS location was last received. This can be very inaccurate (especially in cities with tall buildings) and drivers would end up on the wrong block. This was a horrible customer experience.

Uber's approach to solving this involves some novel computation that adds to the binary size -> https://eng.uber.com/rethinking-gps/

Uber app also tries to resolve and render points-of-interest, so that you don't have to search for just an address as your destination, you can also search for "Dolores Park". That not only powers the search on the backend, it's also rendered on the frontend so that when the map tile for Dolores Park is rendered, the app knows to drop a tree-shaped icon on top of it labeled "Dolores Park". Uber has been using their own maps for a lot of this, and has an entire Maps org that does in-house map display, routing, and search.

And that's just the before-trip functionality. There's extra stuff that happens during the trip, including showing your current location, being able to share your trip with friends, real-time 911 calling for safety, etc etc.

And this is all just the similarities with Google Maps. Uber does extra stuff on top of that, including:

Onboarding flows for signup/signin, including the 3rd party identity services.

Payments: the app needs to support payment linking flows so that you can add and verify your credit card. Sometimes your credit card declines, and there's a whole arrears flow that needs to be built out to support that. And credit cards aren't the only way to pay for trips in Uber, you also have proprietary systems like Venmo, and PayPal that you can link. And on top of that, there are myriad such proprietary wallets all over the world that Uber has supported since at least 2015, including PayTM (in India) and Airtel Money (India and Africa), just to name a couple examples. And in most of the world outside of the US, you can pay for Uber trips with paper cash, and that's a whole flow on the frontend during onboarding / arrears settlement. This needs to all be packed into the same binary because you don't download a different version of the app when you go to a different country.

They also bundle UberEats into the app for some people, and there are probably hundreds of other random features that are prototyped and hidden behind feature flags, or variable experiences that are selectively rolled out in certain geographies. They all need to be packaged into the same binary for everyone.

Throw in other random product features like their Spotify integration, and it all adds up. Uber isn't just a dumb CRUD app.

You can also see the same in other comparable apps, Lyft's iOS app is 290 MB, Ola is 205.3 MB, Grab is 289.6 MB, GoJek is 236.2 MB, Yandex Go is 145.4 MB, etc etc.

EDIT: there's an even better written summary somewhere up-thread of the sheer amount of business complexity that needs to be baked into the app -> https://news.ycombinator.com/item?id=25376346

Yes.

I don't get it either. What is going on in that app to take up 100Mb?

Statically linked libraries.

The "statically linked" part has little to do with the app size.

I'd love to see the Metrics of How much is Business Logic, UI, and 'Misc/Glue'.

Am I the only one who wants to scream when encountering long-form content on Twitter?

Twitter is such an awful use-case for this. Just write a blog post!

Who's making the version of this post for Netflix?

Fun fact: I was able to access a bunch of employee admin screens in early 2015 because of the client side architecture of the app. I noticed around a year later the architecture had changed and always wondered if my article had a role in the transition

http://nathanmock.com/archives/how-i-accessed-employee-setti...

Wow. The real lesson here is that all this "engineering" and doing stuff the supposedly elegant and better way was a colossal disaster.

There is no cohesion or elegance at those kinds of head counts. There are just too many cooks, too many different perspectives, most toolchains allow you to solve problems in too many different ways.

IMO and contrary to what people say, making good software is not a team sport.

You could try to offset that by some combination of keeping head count low, optimizing for technical homogeneity when hiring and/or using intentionally spartan toolchains (Golang for example) but the odds are still against you.

Also, US business culture is maximalist and land-grabby in nature so no one is going to let you actually apply any of those constraints. The people who write checks don't actually care if the software is "good" because in almost all cases it doesn't have to be.

Seems like the lesson here is to not adopt new technologies on your flagship products (instead of first testing the waters on smaller products).

> There’s a video somewhere in one of our talks of an Uber engineer typing a single line statement in Xcode and then waiting 45 seconds for the letter to appear in the editor slowly, one-by-one.

I'm not sure if you are familiar with Xcode, but this indicates an off-the-charts LOC count. Something isn't adding up -- the user-visible features and screens in the app don't necessitate a codebase this large.

My guess would be the intersection of several compounding factors:

- a product team endlessly pushing special-case features which aren't core to the user experience (the twitter thread talks about the rate of new code being added as if it were a foregone conclusion the app would continue to grow without bound),

- an A/B testing framework which leads to the deployed codesize being much larger than what the typical user actually interacts with (worse if they are lax about culling the vestigial A's and B's),

- and reaching a "thermal runaway" point of too many devs x too many lines of code, where code reuse stops happening. Once you reach the point where it takes less time for a dev to write a given feature in a from-scratch fashion rather than doing some codebase archaeology to find existing code structures which can be reused, you've reached thermal runaway. The Facebook iOS app revelation of "18,000 classes" indicates they were likely suffering from this as well https://news.ycombinator.com/item?id=10066338

Was Swift up to the task? Perhaps not. But does a ride-hailing app really necessitate this level of codebase complexity? Uber's iOS app is now over 300MB...

It’s funny you quote that specific line because that one line stood out to me too. I agree that something extremely fishy was going on that couldn’t have been solely explained by “too many linked libraries”.

However, choosing a more mature technology like Java or C# or even Golang would have certainly alleviated a great deal of stress from their process.

I’ve seen many rewrite stories go well. I’ve seen many go south. This just seems like awful management and not spending time early de-risking things.

That being said it’s a common problem in growth startup. Prioritization is hard. Too many engineers doesn’t mean problems get solved. They need to be able to talk to each other and execute with what’s essential to users.

Growth is a curse and a blessing.

Not a mobile developer -

Was the move to Swift inevitable and they started too early or should they have stayed with - and still be using - Objective-C?

I think it was clear early on that Swift is the future of iOS development. But if you’ve been around long enough, you’ll have observed that it takes about 10 years for a new language and associated tooling to reach “maturity”. What “maturity” means is debatable, but one aspect is that the ecosystem grows to handle the extreme edge cases like very large scale development. And boy was Uber pushing the edges of what was possible on iOS at the time.

So by my way of thinking, they moved at least a couple years too early. (The Swift project started internally in 2010, first public release in 2014, Uber’s rewrite started in 2016.)

If they had stuck with the Swift ban they would certainly be facing a hiring problem today.

I want to hear the Android side of this story!

Android was a much easier story, Still had some binary size and method count issues but we were able to overcome them thanks to another brilliant engineer we had in the team in SF who worked on Android.

The Android apk is 56.9 MB for my device, while the iOS app is 331.3 MB. What accounts for the 6x difference?

Courage.

I assume that the constraints placed on the Android team are much higher, as that market seems like it would be much more sensitive of app sizes.

That is true, but what specifically is the iOS app getting in that extra quarter gigabyte? What couldn't they throw away to get under 200 MB at all?

I honestly have not been provided with a good answer beyond "we just didn't spent resources to do it".

Swift produces far larger binaries

APK is now more a delivery package format than a real executable format now.

On recent Android devices, the APK wil be recompiled to machine code on the device after a small number of runs (to gather code statistics).

So you are comparing an apple and a robot.

IPA is also a delivery package format. The big difference I am aware of is that IPA encrypts binaries, which will make them less compressible.

Bytecode is very compact compared to native machine code. Android converts on the fly on the device. That's at least a part of it.

https://twitter.com/ZacSweers/status/1337112019116453892?s=2...

(disclaimer: this is me)

Gone are the days where Twitter body limit was understood as the reason to post short tweets instead of long tales...

Works well in ThreadReader though: https://threadreaderapp.com/thread/1336890442768547845.html

I know right. I was even more disheartened when someone who I thought might be mature enough and understand the modern internet also used Twitter instead of their own blog to write a series of tweets as a blog, Cory Doctorow.

I refuse to go to Twitter to read such stuff (threadreader is not a justification, it's a crappy workaround) and fully endorse the communication of the idea that it sucks.

> As a consequence of all these problems, there began to be a growing movement across all levels of the org that was rallying around the idea of “rewriting the app from scratch” The general sentiment was that the architecture was slowing us down, starting over would be faster.

I've read plenty of these stories where a rewrite was a total disaster. (Most famously, Netscape.) I've seen and/or lived through some too. (One where a major distributed system was completely changing its storage model, and it was thought refactoring was impossible. In hindsight, it was achievable and would have saved _years_.)

Anyone have examples of regretting a refactoring to the same extent?

The uber app was never fast and snappy. The uber app crashes regularly, shows fake cars flying across the map, misaligns your drivers car with the black line of the car will take, has spinning cars and countless other bugs.

He ends his "twitter thread" explaining the results of them never properly testing their new UI & UX. Horrible. Another billion dollar company just throwing expensive engineers at narrow, single issue problems. No proper engineering strategy, user centered design or connected thinking visible. Just an overengineered mess having hundreds of megabytes of bloat. I rant, but could my company or me do it better? No.

What's this "cellular limit"? Is it some apple specific oddity?

The App Store doesn’t allow downloading apps over a certain size unless connected to wifi.

They changed it with iOS 13. It will warn you now, and you can still download over cellular if you want.

https://www.wandera.com/ios-13-app-download-cap-removed/

Note that some of the dropping platforms advice doesn’t really help here, or only helps coincidentally, because the App Store serves you a device-specific app binary.

Yep, it was to prevent you from accidentally blowing through your cellular data cap. They no longer prevent the download, just warn you that the app is over 200MB.

Easy to bash on Uber. Partly a fault of Apple releasing swift and calling it 1.0. It’s clear swift didn’t scale for Uber and it should have been in beta. Apple could have made a more rock solid product.

Why not release the Swift app version as an additional app, e.g. "Uber Next" while keeping the original version alive until all problems are fixed?

This story screams of so many self-made issues.

Because Apple would not have approved it.

They'd want to keep the existing install base.

Former Uber employee. I can corroborate much of what they say (not having been on those sorts of teams). The depiction of the company culture is accurate.

I wonder why there wasn't a push to move a lot of the logic onto servers? That's obviously an oversimplified solution but presumably there should be some way to take stuff like localization, payments, etc. and push them onto a server application?

This isn't a rhetorical question. I imagine there's a well thought out reason why that wasn't considered.

Here's the Threader (i.e. Twitlonger) thread for anyone who wants to read the whole thing: https://threader.app/thread/1336890442768547845

I wonder if you tried to do the same thing today, but with different tools, let's say Flutter, how it would turn out?

If you think how the Uber app looks and works, it should be a major relief doing it in Flutter, I don't see any major breakpoints.

Is this where RIBs came from? My company adopted that and ran into smaller versions of all the same problems. We moved on to something leaner after a couple of months. Lots of interesting ideas in RIBs though.

Yes

On top of the discussed complexity and featuritis, I suspect over-engineering, NIH syndrom and "architecture" must have played a role. Misunderstanding, misapplying and blaming MVC to end up with twice the code in 3 or more times more classes.

He talks about having swift compiler engineers in team. Was swift not opensource back then, so they could optimize the compiler for size instead of reordering optimization passes using simulated annealing?

Are webview apps still possible on IOS? Given that the Uber app has to work online anyway, it seems like a lot of this headache could have been avoided by keeping most of the logic on the server.

Yes, they are. The experience would be quite poor for many things, however.

https://threadreaderapp.com/thread/1336890442768547845.html

> “biggest engineering disaster”

I just supposed that this was a story about Uber Elevate ElectricVTOL (eCRM) design.[0,1]

[0] https://twitter.com/TheWoracle/status/995107989886353408

[1] https://github.com/OpenVSP/OpenVSP/tree/main/examples/vspaer...

The Otto due diligence has to be up there as well.

I understand that the fast pace development of the first app versions was also related to have to be faster than uber's rivals.

anyone else think it's ridiculous that this is in a giant tweet thread? :\

oh gosh thanks https://news.ycombinator.com/item?id=25375799

The biggest disaster about this story is having to read it 2 lines at a time.

Too broken up, didn't read.

Posting stories as Twitter threads personally for me make me loose all respect for you as an engineer. I mean what in the hell is the abysmal way of reading? How could any person allow themselves to post something like this?

As with Swift vs. ObjC, this may be an instance of "fashion over function".

I mean, I’ve seen people post things on Twitter that would make me not hire them[1], but it was usually the actual words and not the way they’re posted.

[1] anyone remember that weird thread about Beau Brummell inventing toxic masculinity?

Reply on news.ycombinator.com