Hacker News Reader

Comment by xyst | Hacker News Reader

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time.

Usually to aim for a significant promotion.

“Designed and built homegrown system to save $Xm! Give me promo, bro?”

Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

View on HN

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics.

If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly expensive. 1.7PB of Parquet files in on-prem or collocated object storage, even replicated a zillion times, is quite cheap. And quite a few companies and open-source projects are currently competing aggressively to provide awesome tools for querying that data.

The hot data would fit on basically anything — the choice should be about robustness and barely even consider cost per TiB. Datomic got written up recently and seems credible for this type of application. FoundationDB is bulletproof. Postgres could probably handle it without breaking a sweat, although active/active replication isn’t free. Heck, writes straight to a warehouse with a cache in front to help with reads seems credible — Uber rides rarely go for longer than a couple hours, and back-of-the-envelope math suggests that the total data rate is maybe 50GB/hour. An entire day of data for an entire country would fit on a single very ordinary commodity server, and the live data for the entire world would fit on one mildly beefy server. The indexes involved sound straightforward.

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes.

Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements.

This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

- Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient.

- Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data?

- Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record.

So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.

>This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)

Capture can happen days after successful authorization.

Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.

> It’s done even in banking

What are you basing this on, banks have less real time constraints when it comes to certain data but thats rare. You will almost never see a bank, processor, etc. archive their system of record data.

"then developers would have multiple systems and namespaces to deal with in order to query on transactions."

Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.

Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.

This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all

In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).

I heard from some X-Uber people that you could call Uber a database company as much as you could call it a transportation company. Something like 80+ databases invented there in one form or another.

Promotion-driven development. I suppose better than blog post driven development, but marginally so.

> I heard from some X-Uber people that you could call Uber a database company

Someone once called Airbus planes "Sun servers with wings". All businesses today have an IT department and those build around the idea of real-time scheduling done by computers are compute/storage businesses first. That's why Uber was able to launch Uber Eats or e-bikes. I am not surprised they have a bunch of busy devs building cool stuff. Not all of it is ego-driven. I liked their logger for Go last time I used it.

nih-syndrome is a real thing. I'm not saying they don't make cool stuff but the engineering culture that prioritizes creating a new database over using something that already exists, then promoting the people who develop the (70th) new DB over the people who were focused on actually delivering value for the customer, it's not my choice in working environment.

The architecture where all the magic lives in the database layer and the services treat the DB like this magical thing that synchronizes across multiple DCs for you, and takes care of every complicated part of engineering a distributed system is a "now you have two problems" situation. It's a massive complicated system that will fail in novel ways, and you don't have a large community of other users that you can fall back on for expertise.

> you don't have a large community of other users that you can fall back on for expertise.

That is one of the most important points of consideration when I am being asked to asses new tech. If the project is not actively supported by the community, I don't recommend it. I just can't expose the client to risk just because there's a new shiny thing.

> I suppose better than blog post driven development, but marginally so.

I find DB development quite interesting, and I find Uber's core product quite not interesting (from an engineering perspective).

So as an outsider without any financial stake in the company, please keep writing about databases!

I'm not sure if this is true of modern Google. Certainly there are a ton of custom things around, but it also seems like in the last 5 years there has been a huge push towards not re-inventing more databases/etc, and normalising everything onto a few core technologies. A good example is Spanner, many products have been moved off other home-grown systems onto Spanner, basically to reduce complexity and cost. Sure Spanner is a Google technology, but at least it's one, instead of having all sorts of different esoteric systems, and Spanner is just one example, this is happening in a number of different domains.

https://news.ycombinator.com/item?id=38300425#38322311

> Uber is famous for NIH syndrome

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1]

If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings.

It very well might be the case that "We spent $25m of engineering resources to save $6m annually".

[0] https://www.levels.fyi/companies/uber/salaries/software-engi...

[1] https://www.levels.fyi/companies/uber/salaries/software-engi...

Bitter story time:

I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings).

My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.

The organization still got the end result, though (to play devil's advocate). That sounds like a win for the company. They got the cost savings, plus they redirected attention back to a project that was higher priority than saving $10m 3 months earlier than they could have.

It may be a win for the company in some categories, but a loss in others.

Sounds like a fantastic way to maneuver and get credit for someone else's proactive engenuity

Of course you didn't. You used your time to promote yourself instead of doing what you were asked to do instead. That could have cost a promotion for your manager who could have promoted you.

We were migrating to Kubernetes from fixed hosts and had to buy the capacity. I just spent a day looking at our utilization and made a better decision about what instance type to use.

I don’t know you are downvoted. Aligning the personal interest vector with the companies interest vector is a huge problem that is usually underrepresented in NH comments.

Usually we only complain about the short sighting of the CEOs that prioritize short term stock gains over long term prosperity, but that also is just a specialized case of the success vector misalignment

This is a poor way to view salaried work.

If OP is meeting deliverables and did this on the side, this is a great example of a story that would make me inclined to hire.

I think maybe my point was misunderstood.

I was trying to say that the employee tried to do what’s good for the company. However that backfired as his personal, and his manager’s, success vector are not aligned with the company’s. Hence although he did something that benefited the company it hurt his and his manager’s performance (exactly because they are not aligned with that of the company’s).

Hope it clears things up.

The problem is OP didn't meet deliverables and the team was behind schedule.

Helpful in the right light, but equally not helpful in another.

“Team behind schedule” usually means that some manager promised an arbitrary date based on an imaginary timeline.

Every team needs slack. Bad managers hate slack. Good managers hide slack.

$25m spend to save $6m annually is a return on investment of 4 (and a bit) years to break even, then it's positive afterwards. So as long as the project runs for more than 4 years, they'd be ahead.

The only problem is that the database code might need maintenance, specs change or situations change, and the cost is _not_ 25m, but way more per year than anticipated. So much so that after a few years, a new batch of engineers start eyeing writing a new database system to purportedly save money...

Right, but that's $25m in R&D investment. Much better than $6m in cost of good/services delivered! Former is great innovation and will be ignored by investors because it's just a fixed cost on the way to becoming profitable. The latter is going to appear in the marginal cost of services calculation.

> we spent $25m of engineering resources to save $6m annually

This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

That math assumes the on-going maintenance of LedgerStore is no higher than that of using DynamoDB was.

I suspect there's now a team of technologists maintaining, securing, and operating LedgerStore.

I suspect they already had a team maintaining/debugging/operating dynamodb

Hosted services doesn't reduce maintenance costs to zero

They have teams maintaining/debugging/operating their use/consumption of dynamodb; they don't have teams maintaining/debugging/operating/securing dynamodb itself.

With LedgerDB, they now have to do both.

Optimizing to address high percentile issues (latency/error/correctness) on someone else’s code follows a bimodal pattern: it’s easy until it is impossible. Doing it on your own code is progressively more difficult and the complexity may cause you to give up, but you normally don’t hit the same wall.

Question is: Where else could they have spent those $25m? Could they have built something with more value?

Opportunity cost is a cost, too

if at uber's scale they still have enough opportunities of that scale for every one of their thousands of engineers i need to buy some of their stock

I mean, ideally you are still employing those people into the future. Plus was there other opportunities to drive value that would've been better spent?

Ideally for the people, but not a requirement. I doubt they won't be conducting more layoffs either.

That’s what I was thinking and fully loaded cost at least 35% more than their salary.

Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it.

Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.

Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.

Yes but since revenue is growing at over 70% due to squeezing out the drivers, there's more money to spend on fighting Amazon over the DynamoDB contract https://www.forbes.com/sites/lensherman/2023/01/16/ubers-new...

Reply on news.ycombinator.com