Hacker News Reader

Comment by amarshall | Hacker News Reader

The developer added more details in the discussion thread:

> We have all of our available code backed up on our SVN server. But the programmer that wrote the lobby code back in 2014 wrote some of the code directly to the lobby server. So we don't have these, and since this programmer isn't here anymore, and probably wouldn't remember what he wrote even if he were, the entire thing doesn't work unless we rewrite. (src: [1])

> Unfortunately, it really is due to incompetence on our part. We really didn't know that our server programmer was writing some of the code directly to the server until we went looking for it after the server deleted our code. We thought we'd just pop the backed-up code directly onto the new server and that'd be the end of it. (src: [2])

[1] https://steamcommunity.com/app/237870/eventcomments/16330403...

[2] https://steamcommunity.com/app/237870/eventcomments/16330403...

View on HN

The missing code in question is from 2014? No one in the last 5 years suggested keeping a yearly snapshot, or making a backup prior to each major release? That's not an "Oopsy, I'm new to this field" error.

After skimming some of the Steam reviews, the cynic in me thinks they didn't want to bother supporting the game anymore after the one that followed (My Time at Portia) took off, so they made up a story that would get them more sympathy than an abrupt announcement that the MP portion would be discontinued. I mean, there's no direct evidence of this, but you'd think that in 5 years... hmm...

Looking at a Unity forum thread from 2015 that the U-Link dev responded in, it seems the writing was already on the wall, which should have triggered a reassessment of their game's use of that component. It's also mentioned that ULink support ended in a reply from 2016 and it released as open source. (https://forum.unity.com/threads/is-ulink-dead-update-devresp...)

That's as far as I care to look into it, but something feels off.

That really sounds like they haven't been backing up their server, and that they think of source control as the backup. Both of these things are pretty bad.

This is common. Most places use their revision control system as backup for their source code and take backups of their database. This makes sense because the source code holds no state. Why would you back up the server? What would you do when you had 100 servers? You’d take backups of the running disk state of all of those? To what end?

Right, it's not a problem with technique. There are lots of storage management paradigms whose backup and restore strategies don't involve filesystem copies of the running servers.

The problem here was that there seemed to be no backup and restore strategy at all.

That said, though: if you're a tiny group like this and don't have the time to invest in a gold plated storage design, "Just Back Up All The Servers" is a pretty reasonable choice to make.

Precisely to this end :)

in that role I would typically back up the deploy directory (/opt/foo) and the /etc/ and parts of /var/ every day to Glacier.

Of course this is all pre-docker and pre-k8s but this is 2013 we are talking about

Well, I think we have an answer to "what end" now.

Yes, if you have a deployment strategy where you can reproducibly deploy from your source code repository, then you don't need to back up the servers. But until you've tested that that works, and I mean really tested it, you need server backups.

Why would you back up the server?

In case of things like this. Also it's usually quicker to recover a server using an image than it would be to provision from source in the case of, say, a hardware failure.

What would you do when you had 100 servers?

Something more scalable. There's no reason why you should pick an approach to solving a problem and stick with it forever. If the situation changes you do something more appropriate. That doesn't mean the small scale approach is wrong when the scale is small.

I mean, the better way of doing this is just keep your application code stateless and make frequent backups of your data stores. If the developers knew they had server code that wasn't checked into SVN, your solution makes more sense, but they just had no idea. Even in corporate, if I need to set up a new box for a service, I don't backup an image from another box and deploy it to the new one - I just pull my Docker image and run that.

Yes, but part of keeping your application code stateless is that you need to actually do things to keep it that way. If you haven't done those things, it's easier to set up Tarsnap in a cron job than it is to make sure your deployment process is stateless.

If you have a service you haven’t built in 5 years, then I’d say you have a lot more important problems than server backups.

Why would you back up the server?

Because the rule number one of the programmers club, is backup everything, everywhere, all the time, if you don't remember do it again, and if you're sure you have enough backups do it again anyway. I even do backup of my backups everyday (in pendrives, in cds, in ancient scrolls, etc)

In the age of treating servers (or containers) like cattle instead of pets, the "Back up everything" mantra has fallen by the wayside. In order to get away with selective backups you have to know exactly where long-term state is stored and you need to have the infrastructure in place to manage re-provisioning everything and restoring snapshots. It's not something you can tack on later. Iterate, test, integrate, document, audit, review. It ends up being much more complicated than periodic wholesale snapshots on a server.

There's a certain elegance and assurance you get from this that has been lost with the times, akin to how monolithic server software with all functionality natively available in the code has gone away in favor of microservices. Now you have message queues, k/v stores, caches, search engines as a microservices that are tacked on to the core services and rarely fully understood by the engineering team and containing more functionality than the codebase ever really utilizes. Ends up being more complicated in manage in a lot of ways. I think the emergence of microservices is one of the driving forces behind selective state backups, because you can never back up the entire state at once, everything is too spread out. You're not going to back up the running state of the k8s node, or whatever

It's more "make everything restorable" rather than "back up everything."

If the servers are still treated as pets, then they need to be backed up. Outside of code there would be configuration.

Keep in mind that once the servers are treated as cattle then individual server backups are typically no longer needed. The developer in this case would likely not be in this situation then.

Shit like this happens more often than one might think. My last job we had one magic disk image that worked and anything else we tried to handroll didn't. (The app itself was written in an ancient version of Django.)

While working on other things I discovered modifications written directly to cached versions of the Django instance itself, which were not committed to any source control. He simply edited the instance of Django in the disk image, which was then replicated to our various deployments.

The worst part was the mods this guy made were perfectly capable of running within our project, from code in source control. Why he took the approach he did was a complete mystery to us.

My last job had similar -- our internal tools were a buggy mess, to the point where customer support spent upwards of 75% of their time fighting the software instead of supporting customers - but the CEO decided that the dev team would not fix any bugs that aren't DIRECTLY affecting end-users, nor would the dev team "waste time" reviewing patches from frustrated tech-literate support staff fixing internal bugs for themselves.

However, the support staff still had root logins to all our servers, so they followed the letter of the law by not having their patches go through the code review / source control / gradual rollout process...

This is actually a pretty reasonable way to run things In The Cloud™. If your servers are transient and provisioned on demand, then backing them up is largely meaningless. Source control is the backup.

But you're supposed to prove it's the backup. It's part of the deal.

As one writer said early in the unraveling of Agile, it's like you were told that if you eat your vegetables you can have desert, and now we just want desert but don't want to eat our vegetables.

> That really sounds like they haven't been backing up their server

Nor rolling it and both testing this procedure and putting people off mutating the server.

My next question is "why didn't the QA team find this?"

And, "Do you have a QA team," he said, expecting the answer "no".

My next comment as a QA: "I told you so"

God, the many times I had told them so months before difficult situations arrived, just to be swiftly ignored by business because some devs didn't want to bother, and wanted to get to more shiny and fancy and interesting problems.

the old "can we use the back-up to really bring everything back up?" question.

in 2002 or so i pushed the sync button an a sony palm pilot (yes, they existed). first button push: whole crm database, strip out non supported fields.

second push: push database back to crm, 'null' all unsuported fields.

after it became clear what happened, to the IT and discuss how to bring the data back via crm backup which we paid alot of monye for (monthly, this was before cheap cloud storage).

.... 1 month later 30 people started refilling copy&paste the crm from their mails, adress books, ...

tl;dr: always test your back ups.

Reply on news.ycombinator.com