Advice on moving to the cloud? | Hacker News Reader

Advice on moving to the cloud?

getclicky.com

18 pointsniels17 comments

I'll save everyone some reading. The advice was:

Don't do it.

While I agree that in their particular circumstances that it's probably the correct answer, I don't think it's right for everyone. For one thing, their system isn't tolerant enough of downtime. If one of their databases goes offline, they've got some manual fixing to do to get it back up and going. Ugh. (I took this to mean that if you unplug the network cable, they have manual fixing to do. They might have meant power loss... And that's a little more understandable, but still not something you really want in the cloud.)

I was referring to power loss. Although in the cloud, it's not actual "power loss" that's the issue, but the end result is the same - the database is shut down uncleanly and must be dealt with before coming back online.

You'll have to design your stuff to tolerate failure; use things like Chef for quick deployments so that provisioning new servers isn't a huge pain in the ass.

Further, the biggest thing to keep in mind is that your resources are shared. This is not too big of a deal for some applications, but for a database server, this can spell certain death (I/O doesn't virtualize too well; you can mitigate some of this by deploying a lot of slave machines for your reads.)

I think more interesting than the question is the fact that the questions was asked at all, by a company to the customers.

I'm a customer of getclickys and in most cases openness like this would be refreshing and it is but there is an element of 'flying by the seat of our pants' that could come across with this post.

Yes, someone on our blog raised a similar point.

The reason we are asking for advice is because we have zero experience running an app in a cloud environment. Well, we did build our own custom CDN using DynDNS and vps.net, but that's just a small part of our service. I'm scared of the cloud for running everything, so just asking for anyone with experience to tell us what they think of the idea.

I have very very limited experience with EC2, a single small instance, for over a month, but I think that gives me more credibility when I say this, don't use EC2 unless you designed your whole app to run on servers you expect to catch fire and explode at any time, because my single little instance died within a month, then twice it, well, did something I've never seen happen before, blow out a device driver? So I couldn't ssh in, all I got was "PTY allocation request failed on channel 0", so now I'm on my fourth instance. I couldn't imagine the carnage if I had been running tens or hundreds of instances.

Thanks. That is exactly my fear. Most of the commenters on our blog told us this probably isn't a good idea so we're very unlikely to do it at this point.

I'M JUST SO DAMN SICK OF DEALING WITH PHYSICAL SERVERS. :)

There's also the option of splitting your app between EC2 and your own servers, so you can run the more ephemeral stuff like web servers and caching in the cloud and the bedrock stuff like databases on your own quality hardware.

Then you'll have less physical servers to deal with :)

My concern about that is communication between web and DB servers. Some reports need hundreds of queries to generate - wouldn't the lag severely slow this down?

Most of our servers are DB anyways. We have 2 load balancers, 6 web servers, and ~40 DB servers.

Yes, it would, but you might be better served to rethink your app design to return multiple result sets, and/or do most of the relevant joining/filtering on the database, even if your DB is <1ms away from your web farm.

We run a good sized e-commerce site (Alexa top 1000; barely) and our ratio of DB queries per dynamic page is right around 2.5:1. You might not be able to get your query-per-page ratio into that range, but hundreds is certainly too many, IMO. Even "dozens" would be cause for concern, IMO. Moving your webfarm to a different datacenter than your DB is going to shine a spotlight on that problem, but even without that spotlight, you still have a problem.

PS: Even with pushing that amount of work to the DB, we have almost the inverted ratio of web servers to DB servers as you report.

Well, most of those queries are just to look up the "name" for a given ID. And of course we use memcached to cache almost everything. But it still requires a bunch of round trip requests to either the database or memcached.

Before anyone asks "why don't you just use a join?" it's because we have tons of different types of data stored in a couple of summary tables. It's 100x faster to just grab all the IDs and values, and then grab the names one at a time (most of which will always be in memcached) than to do it as a join. Don't believe me? We used to do it that way and the performance sucked.

Most of our servers are DB anyways. We have 2 load balancers, 6 web servers, and ~40 DB servers.

Wow, that's not what I was expecting.

I have no idea if those 40 DBs are just mirrors sharing the load of some heavy queries or you are sharding the data in some way. If they are all just replicated maybe move most of them up into the cloud where they talk to your web app. But I don't know how much data you have to replicate up into the cloud or how long that would take.

We're monitoring traffic for 260,000 web sites with over 200 million daily page views. We have almost 10TB of data stored. That requires a lot of servers. Yes, it's sharded. There are no slave servers.

It sounds like you need more in terms of actual application architecture & design - whether or not to offsite some of that capacity to AWS is purely a cost measure in the end.

You will still ned to deal with systems and sysadmin tasks - and those can always be contracted out. AWS does NOT save you from that.

I have a very similar volume site (game analytics) ... I've always been afraid of Amazon because of the bandwidth and request based billing. I just use VPSs to scale when I have to ... it helps that my hosting guy is on my MSN Messenger.

Edit: Also your site looks terrible in Opera ... the CSS or JS or something isn't loading which is breaking everything.

FWIW: I'm not seeing any such phenom. My Opera (10.62) is scarcely discernible from Safari - both of which look very attractive.

Oooh I know what it was. One of the tech blogs had some annoying shit happening when you right clicked URLs that made the right click menu not show for me (slo0w connection here) so I blocked the subdomain.

Reply on news.ycombinator.com