Hacker News Reader

Comment by chrisleydon | Hacker News Reader

We have more backups than you can shake a stick at. The issues lays with the server being compromised and being forced to shut down. The old API and apps relied on a fixed IP address. It would have taken us months to get TinyGrab v1 back up to scratch, or a few weeks to perfect 2.0. No data was lost, what would you choose?

Chris Leydon TinyGrab Founder and Project Manager chris@tinygrab.com

View on HN

Now I'm even more confused. What was stopping you redeploying the server side app onto a new box and changing the hardcoded IP over to the new machine?

Exactly. As a lot of other comments have mentioned, using URLs would have been a better idea, but its irrelevant now. Assigning the same IP to a different machine would have been a perfectly acceptable (and most straightforward) solution. I fail to see why they didn't choose to do that.

We're all interested here in learning from tinygrab's mistakes, not criticize them after the fact. Why patronize us with that "would have taken _months_" quote instead of giving some technical details about what really the problem was?

Thanks for the explanation, but I still don't understand the problem. You re-image the server, fix whatever holes they exploited and re-install from a backup.

Applications should be designed for failure. Assume this will happen every week and plan for it. If not someone cracking your system then a hardware failure, or your ISP going bankrupt or any number of other disasters.

Fixed ip addresses? I apologise if I fail to understand why that should be necessary without assuming that someone is cutting corners. Surely such values should be configurable allowing you to easily move the site elsewhere?

I certainly think moving to 2.0 was a bad move as a broken service can be worse than a missing service (although that can depend on how "critical" the service is for people). Re-establishing 1.0 would be the sensible thing to do and if that is impossible then it sounds like you've been doing something wrong somewhere.

Could you explain the decision process behind using fixed IP addresses? It's not my area of expertise, but surely using a URL would provide you just as much control with vastly more flexibility.

It was an early and foolish decision we made back in the dearly days of TinyGrab. The original system wasn't built, or even created, to handle more than a handful of people using the service. Thanks to some great PR though we scaled to ver 400,000+ users almost over night. The code base was horrible, buggy and 2.0 was being written to fix all of that. It was supposed to be a smooth transition over, but unfortunately someone got there first and forced us to launch the new system.

We are seeing the downside of the "get version 1.0 out in front of users as quickly as possible" mentality.

Fine, get the first version in front of users sooner rather than later. But if you are not in a position to recover from an intrusion, disk failure, fire in the server room, etc. you are not ready to have users. You really need to address all those concerns before the first line of code is written, as without it all your investment in development is at risk.

Were there backups of the source code? The article seems to indicate code corruption on the application hosts, which would also indicate a VCS restore would have reverted the systems to the state prior to the attack.

OTOH backups that cannot be used to restore service are by definition, not backups.

But .. why can't you `grep|sed` your source code for the ip address? I'm confused.

I suspect it's hard-coded into all the clients -- thus they'd have to get everyone to download a new version of the client (which they are now doing, but straight to version 2.0).

Although it should not have been hard to get a new server up at the same IP.

This still doesn't make any sense. You have source code and the ip address. Why can you not configure a new server?

Reply on news.ycombinator.com