Letting go of your data - or how I learned to love the MongoHQ and Heroku | Hacker News Reader

Letting go of your data - or how I learned to love the MongoHQ and Heroku

kent.posterous.com

15 pointskentf7 comments

I run my own servers. It is not that difficult and while it can take up time, it is also a very valuable experience. The biggest benefit: I learned a lot about server administration, stuff that gives me an edge when doing development. I also learned a lot about operations. Most system admins that I encounter are happy to discuss server arrangements with me, rather than scuffing at "another silly developer who doesn't know mysqld from grep".

I also know the exact limitations of my hosting platform: I know where and when things are backed up. I know who has access to the users' data. I know what kind of redundancy is built-in.

If you want to focus on "building a kick ass application" get yourself a cheap VPS, install Debian or Ubuntu on it, and start coding. Once you get to the point where you are paranoid about your VPS provider disappearing or having downtime, set up a hot standby with a different provider and automatic failover.

Unfortunately, the periods when you need extra power because your 'kick ass application' has had some excellent publicity will enviably occur in the middle of the night, or when something else is on your plate forcing you to divert technical resources when you can spare them the least.

Infrastructure issues kick you when you are down.

'The cloud' is still in its infancy, but the goal is to let you sleep soundly at night. Its getting there, but has some distance still to go.

In my experience, scalability issues are often rooted in either not understanding your infrastructure, or not testing it. My WordPress blog survived a pretty heavy Reddit effect with no caching enabled and "only" 6 PHP processes. Similarly, my other applications seem to scale fairly well on VPSs with under 1GB of RAM. Knowing the infrastructure causes me to profile and optimize frequently.

My other issue with the cloud is the vendor lock-in. Google App Engine is pretty cool, but who else provides Big Table support? Amazon's datastore is similarly unique although AWS lets you run MySQL at least. Lastly, I found that it is cheaper to run your own VPSs. The AWS is at least twice as expensive for the same amount of RAM.

You can test all you want, but spikes in demand can still take you down. I dont care how big your VPS is, there will be a point that it can't take anymore, and in real life, that point will probably be sooner than your testing will predict.

I repeat, infrastructure problems will kick you when you least need it.

You have a point about lock in, but there are many cloud providers that just supply hosted versions of open source software (amazon RDS, mongo HQ) so you can still use the cloud, but have the ability to take your data away with you.

This is how I ended up with lots of data in a difficult to use AppEngine datastore.

Proceed with caution.

"There is no cloud, just other people's hard drives" -- unknown

"Letting go of your users data". :)

Reply on news.ycombinator.com