Hacker News Reader

Comment by erulabs | Hacker News Reader

Gotta love how far MySQL + k/v store can get ya. I have no experience with zippyDB, but MySQL and ScyllaDB certainly feels like having an M1 Abrams and a B2 Spirit in my back pocket. I’m constantly pushing the “relationships from MySQL, data from Cassandra” narrative. It sounds like Facebook has a similar approach - although this post doesn’t make it clear where message data actually resides.

MySQL, ScyllaDB, and RabbitMQ. A stack made in pragmatist heaven by devops angels. Now to find some developers who don’t hate it ;)

View on HN

Can you share your take on why MySQL and not Postgres?

For me it's mostly about known-knowns. I've been responsible for pretty large/company-critical MySQL clusters and have managed to sleep soundly at night. I tend to be the "database guy" at every startup, and those startups tended to use MySQL.

I have to say, I do love PostGIS. But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time. I can and have debugged MySQL replication on zero hours of sleep. It's entirely possible Postgres 16 is the better choice for new projects today, but I haven't had the time to try it out in production yet.

> But for "this data absolutely cannot be lost or its game-over", I'd go MySQL every time.

Man...times have seriously changed if that's the case. Good for MySQL to improve enough to get that recommendation.

It amazes me how much better MySQL has been in this regard for at least a decade, and it's also amazing that it's still not that well-known today. Back in 2015 I worked at a fast-growing unicorn that had badly implemented basically everything because they started with a tiny ops team of grads and developers. Very little was being monitored, there were only a handful of metrics being graphed (mostly network stuff in Cacti). Our DB issues were all caused by stupid stuff :

* undetected hard disk in array fails

* battery in array controller fails

* disk fills up

* dubious backups, with no point-in-time recovery

* extremely poorly written SQL queries

* poory configured MySQL (in oh-so-many ways)

The top three (at least) would lastly cause replication lag, which would eventually trigger an alert. ... And yet we never lost a cluster. (And we far a lot of them!)

My team sweated blood improving processes and tooling, and then I spent a 6 month stint on database clusters (switching to GTID based replication and rewriting the ops config code so that they were all consistently configured and monitored).

Occasionally we'd get a new senior hire insist that PostgreSQL was a necessity, so we'd stand back and let them produce a proof of concept that stood up to the types of failures our MySQL clusters dealt with regularly, without waking oncall up at night. And it was always a bit of a joke by comparison.

I haven't worked in big tech but I imagine their tooling is really good for this particular stack. Internal expertise too for running into big scale edge cases.

From having talked to Facebook engineers (I suspect Instagram/Threads is a very different beast for historical reasons), Facebook is only built on "PHP and MySQL" in the sense that the statement is technically correct but not helpful.

The way Facebook apparently uses MySQL is not "MySQL + k/v store" but "MySQL is the k/v store". Forget SQL, they treat it as a key-value lookup and it's good enough for that. This also explains a lot of oddities coming out of FB engineering open source that at times look like they're trying to roll their own database on top of ... well, a database.

I wouldn't go to Facebook for architecture advice unless you have both the scale of Facebook and the technological baggage of Facebook. I'm not sure how transferrable this writeup about Threads is either given that Threads is so strongly tied to Instagram (although EU legislation seems to have motivated Meta at least to no longer require users to delete their Instagram account in order to delete their Threads profile).

> look like they're trying to roll their own database on top of ... well, a database.

That’s exactly what fb is doing. MySQL isn’t used as a k/v store per se but as the storage layer of the Tao graph database (which is also using memcache. So fb builds a database on top of 2 databases, technically). Source: https://engineering.fb.com/2013/06/25/core-infra/tao-the-pow...

This is not correct. Facebook has a large number of distinct MySQL tiers, which serve different purposes and have wildly different workloads. None of them are traditional "key-value lookup" workloads, not even the core user database (entities/associations); this k/v claim is often repeated here, but has never been an accurate representation of the architecture. The TAO graph access pattern absolutely requires range scans over indexes, which is not compatible with a k/v access pattern.

source: I'm a former member of one of Facebook's MySQL teams, and among other things I worked heavily on the managed database-as-a-service tier. This was like an internal-only RDS, used by thousands of engineers for many different purposes, most of which involved SQL.

Thank you for sharing your experience and expertise.

I'm a big fan of MySQL's scaling properties and some of the tooling (Vitess).

Do you see any emergent general purpose RDBMS achieving similar popularity of MySQL and/or PostgreSQL in the near future?

I heard good things about Clickhouse but it's a different tool (column based) from my understanding.

I am not a good judge of popularity, given that I still post a lot about MySQL on HN in the year 2024 :)

In all seriousness though, as a bootstrapped founder I'm not usually able to devote time to tinkering with emergent databases, sorry!

From what I've heard, Clickhouse is excellent, but it is OLAP focused rather than OLTP.

Ahh I see now, you've founded https://github.com/skeema/skeema which is great!

Keep it up!

>given that I still post a lot about MySQL on HN in the year 2024 :)

Not nearly enough So please do that more often :)

Seriously I dont want HN to be one sided about RMDBS and I constantly submit a lot of MySQL related content and release notes. And there is still a whole world using MySQL for a lot of different reasons.

Especially now Oracle has finally updated MySQL release schedule.

>The TAO graph access pattern

Do they still use TAO?

Just wondering if they will ever open source TAO.

I can only go by what FB people have told me in person and that was "we use MySQL pretty much just as a k/v store". I think all the people I talked to worked on the frontend rather than MySQL directly though, so if you worked with MySQL directly that seems more authoritative.

Facebook actually moved off of Cassandra well over a decade ago. Not sure about other Meta properties though like Instagram, but the original article here doesn't mention it anywhere.

Reply on news.ycombinator.com