Hacker News Reader

Comment by sparkman55 | Hacker News Reader

From the article:

> the replicas were out of sync with the master

Most replicated relational database environments are single-master environments; this mirrors the read-oriented traffic of many database use cases.

Most multi-master solutions require you to leave parts of SQL and ACID behind - the advantage is significantly better write scalability and write availability.

Mysql replication has been around for ages; it seems to blow up spectacularly on occasion. Postgres replication is relatively new, and I'm wondering if it also has similar issues. Anecdotal evidence suggests that it has not: I have never been awakened by panicked developers/operators because of postgres replication, but mysql replication has caused more than one sleepless nights.

View on HN

> Mysql replication has been around for ages; it seems to blow up spectacularly on occasion.

For us "old school" guys who have been programming for awhile, this is truly eyebrow-raising. Hearing that DB replication "seems to blow up spectacularly on occasion"...there's something wrong with this picture. A tool like DB replication should be at a much higher level of reliability than application code. It also confirms a lot of the grousing I've heard about the engineering hubris of MySQL over the last decade.

Also reminds me of Alan Kay's quip about what makes programming, "not quite a field."

I want to rush to MySQL's defense here, but I can't. If we forget for a moment about the history of it or the engineering challenges specific to MySQL and so on, what we have is an infrastructure application that one-way copies data to a remote instance of the same application, with almost no error handling and very little consistency checks, and that halts the data copy in the event of an error, without notifying anybody that there's a problem.

MySQL replication feels like a hack, not the sort of thing that people can use as part of a reliable infrastructure.

My MySQL replication wishlist would be:

1. bidirectional communication protocol so that slaves can ask the master for a fresh copy of some particular data in the event of an error;

2. built-in notifications for anything that might make an alcoholic out of a sober sysadmin;

3. periodic idle-time consistency checking (master: "I have X tables with Z definitions and N rows each"; slave: "something is wrong with my copy of Y, I need rows 1 - 100").

I have a couple of projects in my pipeline right now that are being held up entirely by the feeling that MySQL is not yet reliable enough and I need to build better monitoring and automated error-handling systems first.

I suspect your statement could be templated:

MySQL <X> feels like a hack, not the sort of thing that people can use as part of a reliable infrastructure.

What you're describing is essentially any flavor of Galara Cluster for MySQL. Unfortunately, it comes with its own tradeoffs.

However I do see something a bit... off... about your 2nd concern. Why would you want a database to handle its own monitoring? I'd personally rather just set up a set of MySQL monitoring plugins into Nagios. That way all of my monitoring is one place.

Consistency checking across terabytes of data on multiple servers is a hard problem. It would be great if it could be solved, but I'm not holding my breath for them (well, any DB vendor for that matter) to get it right.

I've had issues with Nagios in the past and am not its biggest fan. Centralized monitoring is nice, but infrastructure-critical software shouldn't require people to add on monitoring applications IMO.

Do you have examples of other critical software incorporating their own monitoring solutions, such as Apache, PostgreSQL, even Linux?

How would you monitor memory, disk and load usage?

Sure - Nagios has its issues, but there are quite a few alternatives that can do the same thing, all of which can interface with your database for monitoring.

"old school" guys really shouldn't be raising eyebrows at MySQL blowing up in dumbass ways; they should be quietly congratulating themselves - yet again - on abandoning that inept piece of idiocy years ago.

Anecdotal data point: No problems. It's been rock stable for a couple of years now. Even though we have 22 databases on our master, the streaming replication overhead is practically zero, and the latency is usually in the order of milliseconds.

We did have a weird replication failure at one point when running pg_repack. It's a tool that reclusters tables by duplicating them (essentially writing a new table sequentially) and then renaming the duplicate back to the original table name. This caused the slave to suddenly try to access a table that had not been created on its end yet. But it's possible that it was due to misconfiguration; we are currently investigating it so we can collect enough to data to maybe submit a bug report.

Postgres-XC looks interesting, but I have not heard of anyone using it in production yet. Would love to hear if anyone has used it for anything serious.

Reply on news.ycombinator.com