Hacker News Reader

Comment by sha_r_roh | Hacker News Reader

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

View on HN

At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off.

There is a lot of room in between, where cost estimates are more realistic.

Plenty of things can be prototyped over a weekend, but many will require months and even years to get production-ready, feature-complete, and useful, especially at scale.

This answer pretty much sums a lot of my experience. Of course when the guy somehow pulls this off in 2 weeks it is seen as an easy side project with proof that it is, haha

This is why incentives favor the heavy bloated enterprise approach: if it looks expensive, people feel like they got something good for their money.

The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.

Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict.

https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.

> you could get yourself 25 top-notch engineers without AI PhD

Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher.

Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number.

And that's just the cost of labor, you haven't spun up a single machine and or sent a single byte across a wire.

Work from home doesn't mean that home has to be in the US.

> Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number.

Economies of scale help a bit with this for larger companies, so it's probably not quite double for Uber, but yeah, not too far off as a general rule of thumb. Probably a 75% increase on the employee facing total comp to get fairly close to the company's actual cost for the employee.

"and have another mil left over for metal" was the part accounting for hardware, infrastructure, etc.

And you can fudge the employee salary a mil or two either way, but the point is that spending that much on a team to build something isn't infeasible or even unreasonable.

It doesn't sound like they needed to implement a new DB system for this.

This is using existing features of Docstore which is Uber's own DynamoDB (sharded MySQL) which they seem to be using for almost everything.

Is accounting really a core part of Uber's business? They're a transportation company not a bank. I kind of question the premise really

Uber is a technology company that tracks 'rides' between drivers that are contractors and customers, and accounts for taking money from one and giving it to another. I wouldn't just call it a core part, I'd go so far as it say it is the intrinsic essence of their business. They're not a bank, but they're not running a branch with tellers taking cash and running ATMs, either.

They are in the transportation market serving transportation needs for a transportation-seeking customer base. How they accomplish that is obviously interesting, but their attempts to move laterally haven’t been amazing from what I can tell (though I don’t follow them closely).

They are structured and run like a tech company but imo they don’t produce a tech product.

> They are structured and run like a tech company but imo they don’t produce a tech product.

This kind of comment could kill their stock :P

Uber said they were going to produce self-driving cars, so technically they are a failed tech company, which is now just a company. I am surprised they didn't get crushed by a competitor that didn't waste their capital on worthless stuff. So much for capitalism working. Uber should have gone bankrupt.

There is just no way that their database is actually technologically novel. Uber just doesn't have the expertise.

> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance

I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

> I'm guessing they know a lot about their costs, and you know very little.

I'm curious what makes you believe the OP doesn't know about cost? They might be director-level at a large tech company with 20+ years experience for all you know...

> There's little value in insulting the team members like this.

I'd argue it's not insulting to question a claim (i.e. 'we saved $6MM') that is offered with little explanation.

Regardless of position at some other company it will tell you precisely 0 about this specific situation.

That was not a nice reply for a non-insult. Do you have anything to add maybe?

> That was not a nice reply for a non-insult.

It's an insult if you dismissively explain basic things to the folks working on the project.

It’s not insulting to speculate in a conversational way around the errors we very very commonly see

If you read the article the system was a layer on top of DynamoDB they updated it to use internal product Docstore which required adding a feature to Docstore. So it's not as involved as people make it out to be. Also records are immutable which makes a lot of things way easier.

Off the self software doesn't make sense for a company that is planning on lasting a long time. These solutions are all designed for multiple use cases. That means that there is complexity and inefficiencies that are not required for your particular problem. If you were to just focus on your problem wouldn't you just end up at an ASIC as the most optimal solution? Reason most software doesn't is 1) people like to re-invent the wheel 2) As you go start going lower level the less qualified people you can find.

I'd be curious as well to see a more complete cost-benefit analysis, and I'd be especially interested in labor cost.

We don't know how much time and head count Uber committed to this project, but I would be impressed if they were able to pull this off with fewer than 6-8 people. We can use that to get a very rough lower-bound cost estimate.

For example, AWS internally uses a rule of thumb where each developer should generate about $1MM ARR (annual recurring revenue). So, if you have 20 head count, your service should bring in about $20MM annually. If Uber pulled this off with a team of ~6 engineers, by AWS logic, they should about break even.

Another rule of thumb I sometimes see applied is 2x developer salary. So for example, let's assume a 7-person team of 2xSDE1, 3xSDE2, 1xSDE3, and 1xSTAFF, then according to levels.fyi that would be a total annual salary of $2.3MM. Double that, and you get $4.6MM/year to justify that team annual cost footprint, which is still less than $6MM.

Of course, this is assuming a small increase in headcount to operate this new, custom data store, and does not factor in a potentially significant development and migration cost.

So unless my math is completely off, it sounds to me like the cost of development, migration, and ownership is not that far off from the cost of the status quo (i.e. DynamoDb).

If the savings are 6 million per year, then in later years it should pay off since the development is a one time cost.

The cost doesn't suddenly drop to zero once development is done. Typically a system of this complexity and scale requires constant maintenance. You'll need someone to be on-call (pager duty) to respond to alarms, you'll need to fix bugs, improve efficiency, apply security patches, tune alarms and metrics, etc.

In my experience you probably need a small team (6-8 people) to maintain something like this. Maybe you can consolidate some things (e.g. if your system has low on-call pressure, you may be able to merge rotations with other teams, etc.) but it doesn't go down to zero.

If you follow the various links on the Uber site, you see that they have multiple applications sitting on the same database. see https://www.uber.com/blog/schemaless-sql-database/ . It's not just 1 design of a database, with 1 application on top...

Not an engineer, but something like this takes 6-8 people working on only this for a full year?

That has been my experience, yes. You need one full-time manager, one full-time on-call/pager duty (usually a rotation), and then 4-6 people doing feature development, bug fixes, and operational stuff (e.g. applying security patches, tuning alarms, upgrading instance types, tuning auto-scaling groups, etc. etc.).

Maybe you can do it a bit cheaper, e.g. with 4-6 people, but my point is that there's an on-going cost of ownership that any custom-built solution tends to incur.

Amortizing that cost over many customers is essentially the entire business model of AWS :)

You're assuming that the team only works on this product. It is possible they are owners of a lot more than just 1 db.

> Payments would not likely be a long-term bet as well

How so? It’s a pretty ubiquitous problem…

Reply on news.ycombinator.com