Tough night dude. I'll buy you a drink or three if you're ever in Sydney...
Alas, poor Yorick!
Sorry for the rough night Yorick. This could happen to all of us but of course it happens to the person that is working the hardest. <3
I'm going to add to your message box unnecessarily, but I want to say I love GitLab and it's a shining example of a transparent company. I still have ambitions to work there someday, and this event is hopefully a net gain in the end, in that everyone here and there learns about backups.
Thanks for the transparency. Doesn't always feel good to have missteps aired in public, but it makes us all a little better as a community to be clear about where mistakes can be made.
I've been there myself, it was at the start of my career and almost ended it. I know how incredibly emotional this kind of thing can be. Just understand you aren't the first, you wont be the last and shit happens. If you are ever in Philly I'll buy you a beer if you drink, and a dinner if you don't.
Was just trying to push earlier today and found out about the issue. Sorry man! Drinks on me in Montevideo, Uruguay. This stuff happens, more than most of us are willing to accept so, here is for your transparency and you know, fix it, learn it and on you go!
Infrastructure ops here. Happens to the best of us. Just have to get back on the horse.
All I could think as I read the postmortem was "I hope YP is doing okay". We've all been there, and I hope you're doing okay!
I'll pour one out for you next time I go out.
Unlucky mate. Even monkeys fall out of trees. Good luck with the fixing.
Dude, we've all been there. You are neither the first nor the last. It's never a single person. One day the technical side of our little startup collectively went to a conference (back then it was only two engineers plus a technical leader) while our server DoS'd itself by broken mail processing... we had a rough night in a Belgium hotel figuring out who attacked the site to realize it was ourselves. The 10k block of missing image IDs always stood as a reference not to leave even a low traffic site unmonitored. It happens.
Hey man, don't beat yourself up over this. It's shitty but you found some flaws in the process, in the setup, and y'all can make things better because of that.
Been there, done that (like most of us that deal with any kinds of infra). Hang in there and best of luck on the recovery efforts!
Yorick! Thank you for the transparency, I know how tough incidents like these can be. Stop the bleeding, figure out how to handle this better in the future, but most of all, take care!
Hey - thanks for GitLab, and my condolences on your (data) loss.
It happens to everyone eventually, so don't feel too bad. But at least now you'll have a much more robust backup process.
We've all been there! Feel better man, it could be a lot worse :)
Go sleep man, shit happens
Always welcome for a drink when you are in Osaka/Tokyo.
You have drinks of your choice in Stockholm too. Man hug.