WAL-G – fast archival and restoration for PostgreSQL

I sort of commissioned the project at Citus, under the auspices of figuring out how much memory copies were costing WAL-E. The answer was: a lot. Our main goal was to touch as many risk points as possible in the design of WAL-E replacement, with emphasis on performance. It was a prototype to that end, and I rather planned to rewrite it, borrowing heavily as I saw fit: the fate of all prototypes. Various priorities got in the way of that.

It was also an opportunity to build competency in management and definition of an intern-sized project. Katie did a very good job, surpassing, by far, the amount of information I thought we could gather during her tenure.

However, unexpectedly, and for a while now, Yandex staff have really worked on the code base a rather lot, bringing it beyond merely practical. They seem to use it under similar conditions WAL-E was designed: for en-masse deployment.

That said, though, I don't think it has the end-user polish that WAL-E had (insomuch as it did) at its peak of maintenance attention. I would consider it acceptable for a programmer to use, but don't expect a sealed project. It might be suitable for those running a large operation and with willingness to get into the implementation.

You can see some plots here. https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-.... Since that time, the errata about parallel recovery has been lifted, courtesy of Andrey.

If your DB is under few TB, you, probably, can go with WAL-E. We have thousands of clusters with total amount around 1PB of data. WAL-G is simple to set up, but current CLI experience is far from perfect.

Our goal is to make the most performant PostgreSQL backup system for cloud deployments. WAL-G is not just fast compression tool: we parallelize serial archive\restore interface and provide very cheap delta-backups. In PostgreSQL, you usually have PITR through WAL. If you have rare backups, your restore time is slow: WAL is applied serially. With WAL-G you can have delta-backups often, they are applied in parallel and much faster than WAL. This is important for us, because we have a bunch of distributed datacenters and from time to time we need to repair HA clusters from backups as fast as we can.

Best regards, Andrey.

WAL-G only supports AWS, so a lot of us are still using WAL-E. If anyone is looking for a reasonably simple Go project to contribute to, have a look at adding Azure, OpenStack or GCE support to help bring it up to parity.

I switched the production backups for ctadvisor from WAL-E to WAL-G largely because of the whole the workflow I blogged about here:

https://www.lolware.net/2017/02/02/continuous-backup-tests-w...

I'm not a python person and playing with pip the various deps were painful.

It's not the developer's fault - I write plenty of Ruby and I expect non-rubyists would have the same issues with some of my code.

But having a Go binary make all these problems go away is a dream.

We are using Wal-G in production and testing it every day by copying/restoring prod db to staging. The CLI is spartan, but it works and it's fast.

Andrey is also responsive if you encounter any issue and went to a lot of trouble to fix a hard to reproduce recovery issue we encountered.