Hacker News Reader

Comment by cheradenine_uk | Hacker News Reader

I know, right?

It's also perfectly possible to split those things up, and to have them share the same database.

This causes microservice extremists' heads to pop off..

It's possible, but also a really poor idea. The whole point of microservices is to separate concerns. Reintegrating those concerns through the database is an excellent way to run into problems. Changes to the database schema will alter how all the services that touch it work, and now you have to manage schema migrations across multiple services and probably multiple teams.

So much nicer to expose a documented and versioned API (and/or published event stream), so that the consumers of data your service manages have some flexibility at which moment to migrate to the new schema.

We do this at work with multiple entry points. New engineers sometimes don’t even know it exists until the 6th month when they eventually run into shenanigans.

Hehe "shenanigans" ;-)

The truth is, most things can be made to work - though there are various trade-offs ("shenanigans") involved.

e.g: In my current project, there's a bit that loads large-ish files, parses/converts the data, and loads them (order of tens of thousands of rwos) into SQL tables.

For reasons that best escape me - since after the files are parsed the entire dataset is _literally_ in memory, instead of doing a bulk sql update there- and then, thousands of messages are enqueued to be consumed by a practically unlimited number of lambda functions that then... bomb out as the SQL database hits it's connection limit and keels over.

I guess those shenanigans have better buzzword compliance!

Sounds like one of my favorite shenanigans when the stakeholders tell you it needs to handle billions, so you engineer it that way. Reality: it needs to handle thousands. Sounds like you got the reverse shenanigans.

Except that there's only one database, you it wouldn't even billions. In fact, I suspect it will handle significantly less load than a batched model will. I've been able to get 100x speedups by batching SQL queries.

I miss the days of just one database sometimes…

Batching is the way to go, 100%. Remove the network as much as you possibly can.

You should really put a connection manager in between your lambda functions and database, like HAProxy or AWS RDS Proxy.

(still, that doesn't change the fact the code is badly designed)

Reply on news.ycombinator.com