I just really like gitlab. I am not personally affected by the outage, but as far as collosul fuckups go, I think they will do their level best to fix the issue, be transparent about it and do what they can to make it right.
We'll get a really good measure of the company by how this turns out.
Is anybody using a procedure to automatically perform restores and check that they match production data? I guess it's not easy for complex systems and non trivial amounts of data. Do you want to share the details and give us an idea of the costs? Thanks.
I've setup a postgresql backup before that would use pgbasebackup and get the latest xlog segments then I would start up postgres and run select 1 against it. That simply wasn't enough as I still got corrupted indexes but at least data was there. If I were to do it again I'd probably run a whole production test suite against it. The cost is extra hardware and time as backup boxes aren't made for performance for a 1Tb+ database it would take more than 24 hours.