Hacker News Reader

Could someone give their perspective on why a lot of these companies (big tech) develop their own databases? I don’t want to sound dismissive I am generally curious. What sets this db apart from other streaming services? It must cost a lot to develop something like this?

Nonetheless cool product, thanks for sharing and making it open source.

View on HN

Big tech companies operate at the edges of the problem space. As a theoretical example, if you need to ingest and process 10TB of numeric data every day, you can probably find an existing DBMS that you can tune to do this. But what if you foresee your volumes growing to 100TB? No one has needed that before.

Patching an existing DBMS means your changes have to mesh well with all its other features. And since you have a lot of talent on payroll already, it's probably easier to write a DBMS that solves your narrow use case, but solves it 100%.

The first question they should be asking is, how much of this can I discard immediately, and how much can I discard after storing it in summarised form, then get technical.

Reply on news.ycombinator.com