Hacker News Reader

> PjM/PM time to organize initial build

This sounds what big companies or a disorganized company would need. For an efficient enough company, a project like this needs just one or two dedicated engineers.

In fact, I can't imagine why this project needs a PM at all. The database is used by engineers and is built by engineers. Engineers should be their own PMs. It's like we need a PM for a programming language, but no, the compiler writer must be the language designer and must use the the language. Those who do not use a product or do not have in-depth knowledge in the the domain should not be the PM of the product.

>For an efficient enough company, a project like this needs just one or two dedicated engineers.

Maybe for a research project or a hobby project, but not for a real, high performance database to be used in a business-critical application.

FTA:

"Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief."

>Engineers should be their own PMs.

For small projects, sure (your "one or two dedicated engineers"). But once you start tackling projects that require larger teams, or even teams of teams, you need someone to track and prioritize the work remaining and the work in progress (as well as the corresponding budgets for personnel, services, and other resources). Similar to the way a sole proprietor can do their own accounting, but a multi-million dollar business probably should have an accountant.

As an aside, I wonder if this might be a use case for a bitmap db engine like Featurebase (https://www.featurebase.com/).

> what we’ve built is just a cursor streaming a binary file feed with a very limited set of functionality - but then again, it’s the exact functionality we need and we didn’t lose any features.

The trick is that they didn't need a database that provides "Atomicity, Consistency, Isolation, and Durability (ACID)". By only implementing what they need they were able to keep the project small.

It's like people are scared of doing anything without making it into some huge multi hundred developer effort. They've written a super simple append only document store. It's not rocket science. It's not a general purpose arbitrary SQL database.

If it's really true that they only needed a "cursor streaming binary file feed with a very limited set of functionality", then why claim in the headline they wrote their own database and begin the article with the observation that doing ACID databases is really hard?

Maybe they really built only what they needed. If so, good on them, but ditch the clickbait title. On the other hand, a lot of people believe that they've built only what they need, only to discover in a week or a month or a year that they underestimated their own requirements. Then they gradually re-learn all the lessons that led to the development of real databases, to their eventual sorrow.

OTOH, the Bitcask paper can be implemented in about a week for a write-ahead log that’ll give you pretty strong guarantees.

This.

Everything we are teaching people to do is centered around solving the problems with already built services, preferably from one of the big 3 cloud vendors.

We are churning out “qualified” engineers who don’t know how anything works and we pay them to configure a few AWS services.

> a project like this needs just one or two dedicated engineers.

So that is at least 20k a month, for fairly cheap engineers.

I doubt he meant « two engineers full time », I suspect it’s more like a role of maintaining / improving when necessary.

1-2 days work is 20k a month? This place hiring?? Even Google wanted me to work minimum 3 days a month...

> In fact, I can't imagine why this project needs a PM at all. The database is used by engineers and is built by engineers. Engineers should be their own PMs.

What about when two different projects have two different requirements they need supported by the database. Which one is implemented first? What about if there is only engineering capacity to implement one?

I don’t think a database is the place for “just send a PR for adding your required feature and ping the team that owns it” kind of development. It requires research, planning, architecture review, testing, etc. It’s not a hobby project, it’s a critical tool for the business.

> Which one is implemented first?

One of them. This is true whether you have a person named "PM" or not. It's just a matter of who picks.

> What about if there is only engineering capacity to implement one?

How does naming some guy "PM" solve the issue? The team just picks one of the features.

I feel like the word "database" is throwing people off because they're comparing it with something like MySql/Postgres, when this seems slightly more complex than a k/v store stored to a file, with some other indexing, where data integrity is a low priority. That shouldn't take too much time and should be fairly isolated on the tech side so little involvement from product/project managers.

A k/v store typically is really fast at looking up the value based on the key. So there are usually some pretty advanced indexes involved.

or a simple b-tree...

My simple btrees have had bugs in them. :)

(Though to be fair, if I actually wanted to put this in prod it probably wouldn't take too long to fuzz it and fix the kinks.)

https://github.com/eatonphil/btree-rs

80/20

For a k/v store, a hash table will take you very, very far.

> The cost of 120,000/year for Aurora seems like it would be less than the cost of development/organization time for the custom database

Only if they planned on hiring someone just to develop this new database and if they switch to Aurora they’d let them go immediately. If the said developer was already costing them $250k to maintain and develop the application and work on top of Aurora cost seems like a good way to save $100k/year.

There’s the opportunity cost of whatever else they could have been paying that developer to work on.

Agreed, a developer that can pull this off is pretty good, if maybe distracted by shiny objects, what could they do working on the actual product instead of this technological terror?

True. Also, to your point, one could argue that if that developer leaves, they'd have an easier time hiring anyone with Aurora experience as opposed to someone to learn and maintain the custom database.

But at the same time, Aurora costs could also scale with usage. It may cost $120k one year, $180k next year, $500k the year after. If the database they have now is well designed after it's already built it may not need active development every year but adding a feature here and there. Also, switching back to Aurora could also be an opportunity cost "we should have written our own thing and could have saved millions ...".

Well considering the cost is lower than Aurora isn’t the opportunity cost in favor of the home built situation.

That’s only true if Aurora is the most valuable thing that developer could be working on.

Well presumably they need only 1/3 of a developer to do this and they intend to scale up 10x in the next 5 years.

$60,000 per year in-house vs $1,200,000 per year aurora. No brainer really.

Also worth mentioning that it's 150x faster.

Its $120,000 a year for aurora, not $1,200,000.

What has changed though, is that we’ve replaced our $10k/month Aurora instances with a $200/month Elastic Block Storage (EBS) volume.

Note 'instances' eg plural, versus a singular EBS. There is some ambiguity here, I'm not sure where the 10x came from, but it seems plausible.

if they were able to replace aurora instances with a glorified kvp store, then they bought the wrong tool in the first place.

i saved hundreds of dollars per month by switching from an audi a4 to riding a home-built bike for my 1.5 mile commute to work

To be fair, one hardware mysql server for 5k, outperforms a dozen auroea instances.

It really bugs me how everyone has drank the coolaide. Cloud is stupid expensive, but of course this is cloud vs cloud.

RDS also outperforms Aurora. My 12 year old Dell R620s outperform both for certain types of queries (admittedly they have some fast NVMe over insanely fast Mellanox).

SANs add latency, who knew? What did surprise me during testing is that Aurora seems to have added latency even if the entire dataset could easily fit into buffers, with zero disk reads required.

If you want actually fast cloud DBaaS, the only way is with instances that have local NVMe for caching, like AWS ??gd.db instances.

wholly unimpressed with the aurora line but my mgmt has swallowed it hook, line, and sinker

Mine thinks we’re going to be able to shift a massive, resource-hungry PG BDR mesh onto Aurora. I don’t buy it, purely from a performance capability standpoint. The only reason the shit queries we have run at all is through sheer performance.

I think for this kind of thing their needs are so simple and well-suited to a bespoke implementation that it probably paid for itself in less than 4 months. This doesn't seem like a db implementation that's going to need dedicated maintenance.

They're operationally using a funny spelling of SQLite and I don't imagine anyone arguing that such a thing needs constant attention.

Don’t forget this is a largely one time cost vs Aurora, which scales cost with usage.

Also they said their current volume is around 13k/second. They’ve built the new platform for 30k/sec per node. This should last them a long time with minimal maintenance.

Using Aurora is also not free in terms of cost of development. Developers need to be trained on it's implementation, features, constraints, performance implications, keeping up with API changes, etc

The problem is all those people you listed would still exist plus the 120k bill to Amazon.

They may or may not be doing other things depending on the company size and state.

You could drop the PM, engineers writing for engineers don’t need a PM.

You will likely hire similarly costed engineers to maintain the database stack anyways.

You basically hit all the talking points big cloud has brainwashed people into thinking into being true, but every day we see stories of a handful of engineers doing something we are told can’t be done and saving millions in cloud cost.

It’s so painful to watch. Software Engineering became a thing because you could hire a engineer solve your problem, and big businesses stepped in and told you that your problem was something else and gone out of its way to stifle innovation by settling industry standards on how to do things that only guarantee you use cloud services.

Any company that wants to own its destiny knows to stay away from lock-in.

I would imagine, as someone with no special insight into goings-on at Hivekit, that the answer is intended scale.

They mention 13.5k simultaneous connections. The US has 4.2 million tractors alone, just the US, just tractors. If they get 10% of those tractors on the network that's a 30x to their data storage needs. So multiply that across the entire planet, and all the use cases they hope to serve.

Investing time early on so that they can store 50x data-per-dollar is almost certainly time well spent.

Presumably those tractors wouldn't be connecting directly to the db though. Not sure why they dont just go the standard iot events route and store data in a data lake and propagate into an analytics db/warehouse from there. Add a layer to make recent events available immediately.

S3 is relatively cheap.

I came here to ask the same question.

If this db requires 1 full-time developer then the cost would immediately be not worth it (assuming salary + benefits > $120k/yr)

As you say, without details it’s hard to know if this was a good idea.

Shouldn’t we up our standard developer cost for inflation?

That barely qualifies for the median mortgage in the US.

I believe the 120k number was in reference to the OP’s Aurora spend.

What makes you think a standard developer can afford a mortgage?

I find that highly unlikely. Maybe in specific markets but not US wide.

The median home price is under 400K, so a 120k salary is not really stretched.

Now, median in the Seattle metro, or in San Francisco, sure. But 120k in, say, St Louis is still going to get you an intermediate dev, no problem, and they can afford a house by themselves too. There are 4 bedroom houses in my neighborhood for 300K.

I actually disagree with you here. There are costs above and beyond the engineer's effect on the balance sheet. There's the partial salary of management to manage them, plus asking them to document their work and train others so that the database won't have a bus factor of 1. So in well-run engineering departments, there's no such thing as paying for a "single" engineering salary. You have teams; a team maintains the system and it has a pre-existing workload.

A large part of the value of popular platforms is precisely that they are not bespoke. You can hire engineers with MySQL/Postgres experience. You cannot hire engineers who already have experience with your bespoke systems.

They said 'instances' (plural), so that number should be at least $240k

> PjM/PM

What do you need them for?

To stop the engineers from spending too much time optimizing the database and to focus on customers

So now every change takes 10x as long because you need to explain it to somebody who's bonus depends on it not being done? I love how the solution to too much management is always more management.