Hacker News Reader

Comment by ThinkBeat | Hacker News Reader

When you decide to just write your own database to store stuff in, because whatever is out there just doesn't fit your use case. you have most likely failed.

You can then either write a new database that fits the use case, or take a hard look at the use case and change it to fit existing database systems.

Now as a programmer I LOVE the idea of writing a database or an operating system, it would be super leet.

When writing a new database system, you will make a lot of mistakes and a lot of bugs that you will no doubt encounter at inconvenient times after a lot of debugging and profiling.

This is built on top of MySQL which has been battle tested (still not where I would put any critical data) Technically that you're not writing the engine but with so much functionality in two layers above MySQL you are in essence building an engine for the engine.

View on HN

You’re implying that you should never invent new tools. But every tool was a new invention upon its creation. Also, all software is an “engine” for an “engine”.

I think this depends on the context. Let’s say Uber had a database problem, they hired a bunch of database people and they came to the conclusion “we just have to write our own… fine. I think that’s actually perfectly legit. What I see much more commonly though is people who don’t really know too much about databases have problems and rather than go off and hire/consult with experts, they just decide to build their own. That’s when you get into real trouble. The dirty truth is that whilst yes, Uber faces challenges due its scale, it also faces few real limits on building superfluous shit because of their success.

Yeah this smells like promotion oriented architecture. I have a hard time believing that a logistics company that built DynamoDB and uses it extensively can make it work but uber cannot.

The 6m in savings does not properly account for things like ramping up new hires on some custom database, maintenance (what happens in 5 years when whatever language you wrote it in needs to be upgraded to a new version, or some dependency), and a host of other things.

Yes the cloud is expensive, but the entire point of it is that you are offloading all of that underlying maintenance/feature work to a team that only does that all day every day and is very good at it.

I understand that this Ledgerstore thing is just some tables and materialized views in a Docstore database which they already use for everything else.

This seems to be an in-house DynamoDB (sharded MySQL with Raft) which has been developed a long time ago, see: https://www.uber.com/en-US/blog/schemaless-sql-database/

But maybe I'm totally wrong here because they also use or used Google Spanner: https://www.uber.com/en-US/blog/building-ubers-fulfillment-p...

Perhaps but I think most of us have never worked on a problem of Uber's scale. The behaviors of things change spectacularly above a certain level; it's not just the same thing times N.

The implicit assumption of "not invented here syndrome" is that it has been invented elsewhere already, which isn't always true.

whats missing from the reuse picture for things like operating systems, compilers (somwhat), and databases is systems that use a 'lego' model of components that can be put together into solutions rather than deployable systems (which can clearly be built on top).

this provdies room for doing real customization without having to build all the things from ground zero, and is likely to be more robust than trying to attach functionality fully on the outside like was done here.

in the database world it would be really nice to implement caching and sharding extensions inside the transactional envelope

That all makes sense. On the other hand, next gen workhorses must often arise from people opting to do something new like this, rather than make do with the current gen workhorse.

Agreed. There’s no reason to have so many indices.

I think the issue is their engineering culture. They have always had far too many engineers. They are clearly over complicating a such a simple product.

Promotion and Resume Driven Development.

Reply on news.ycombinator.com