Moz Dumps Amazon Web Services, Citing Expense and 'Lacking' Service | Hacker News Reader

Moz Dumps Amazon Web Services, Citing Expense and 'Lacking' Service

xconomy.com

150 pointsrbudd127 comments

For what many orgs need to do, a myriad of lesser-known VPS providers would be just fine, and be a fraction of the cost. But most people I've known don't investigate or educate themselves much - it's been a herd mentality towards AWS, irrespective of needs or costs (because often, it's not them paying - it's "the company").

Perhaps as a freelance consultant right now I take a sharper look at the costs involved - if a client has, say, $50k to budget on a project, I really don't want $10k of that to go to hosting if it doesn't need to - I'd rather it be available for my services directly (obviously) or to give us more play for marketing, testing, whatever.

I've known a couple of small to medium sized companies where they were using AWS rather strategically as part of their services - dynamically spinning up test environments for running parallel testing, off-peak data crunching, etc - but even in those cases, there weren't gigantic savings vs other options.

What bugs me most is the false dichotomy I get when I talk about this with friends; their choice is presented as either use AWS or "hire a full-time sysadmin, buy multiple server-grade boxes and manage all this in a rented data center". I've been told Amazon is saving people "easily" $100k/year by not having to hire staff and buy expensive hardware. Hrmm... not all hardware is expensive, and instead of hiring a sysadmin, you've now got multiple developers spending time learning the AWS API, and building critical business processes around AWS, tying you to Amazon for the long haul, all while they could be focusing more on line-of-business stuff that you hired them for in the first place.

I don't doubt that in some cases there's robust cost savings, but it's never struck me as a terribly cost-efficient way of doing any web stuff. In some cases it may be the only way to deal with certain problems, but in those cases cost probably isn't a consideration anyway.

> if a client has, say, $50k to budget on a project, I really don't want $10k of that to go to hosting if it doesn't need to

On a $50k project, how are you even dropping $10k on AWS? I've found a medium instance (~$100/mo) handles with ease 100k+ visitors/mo without breaking a sweat.

depends on the work load, data size, computation needs, etc. measuring by "visitors per month" makes no sense.

I think it's reasonable to assume that d23 knows that. The question is "how", and the second sentence provides context for d23's knowledge and expectations, which mgkimsal might use in a followup answer.

I too find it difficult to think of a $40K project which is both likely to spend $10K at Amazon, and which can acquire equivalent resources (hardware, power, space, admin, etc.) for significantly under the same cost.

Not impossible, but real-world examples would be nice.

Then there's the AWS massive disadvantage in data traffic cost. Small VPS providers (or large ones for that matter) typically include 4-5 Tb of data, and their compute is no more expensive (or cheaper) than amazon. And half the customers want large pngs (because jpg at 2% "reduces quality"). I'd say only to people with 45" monitors at 5cm to their screen or less. It also produces 50% data savings.

That's the biggest mistake one can make : host a traffic intensive app (even lots of graphics) on AWS.

If anyone is in that situation and remotely budget conscious, you can do this trivially:

1) Order a server with lots of bandwidth from a provider such as Hetzner, 100tb or Leaseweb (or possibly OVH, though I haven't used them personally).

Low performance hardware is fine, though you'll either want enough ram to fit your frequently used images, or an SSD.

2) Install nginx as a reverse caching proxy. There's an example configuration in the nginx docs. Requests to /blah/something.png on your nginx server are sourced from www.yoursite.com/blah/something.png, but cached for the period you specify.

3) In your DNS, set static1.yoursite.com, static2 etc to point to the newly created server. This is for browser parallel downloads.

4) In your html, point your static assets randomly amongst the newly created domains (best if this is done automatically).

In this way, your newly created box soaks up the load of the static assets traffic, but it has no additional cognitive overhead. You don't change your development model at all.

If you have a switch in your code to source the images from the original domain, you can go back to hitting AWS instantly should the box have problems, so the risk is very low.

You can get a 10 gigabit unmetered server from the Leaseweb volume network for 600 euros/mo. The bandwidth is lower quality than AWS, but if you have high utilization you can pump more than a petabyte from the server a month, which would cost approximately $60k from EC2.

Or simply use a CDN like e.g. CloudFlare.

Sure... the above is how to get a ton of bandwidth on the cheap. CloudFlare (and other CDNs) aren't the way to do this, as described below:

http://phoboslab.org/log/2013/02/how-much-traffic-is-too-muc...

>> At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.

> The Enterprise plan comes at $3000/mo.

Hurricane Electric transit FTW.

Bad language/platform/library choices. People go for what's hip rather than what's efficient.

> I've found a medium instance (~$100/mo) handles with ease 100k+ visitors/mo without breaking a sweat.

This is not really intended as a direct reply to you, but I think that price point illustrates mgkimsals point well: I have a Django-based project with about the same amount of visitors per month on an old rented vserver that costs about 7€($10)/month. Yes, you have to set up nginx once, but as mgkimsal mentioned setting up your project for using AWS is not done without any work either. (And of course the load heavily depends on the type of project, so perhaps my visitors could mean only a fraction of the computational load of the ones in your example.)

> Yes, you have to set up nginx once ...

This is the kind of thing that I've just started documenting at http://tweaked.io/

Great Idea! May I suggest a tutorial for setting up uwsgi in a robust way, e.g. including a proper startup script in /etc/init.d/ or something similar?

$50k/$10k is possibly a bit extreme - say $80k/$8k perhaps.

Typically when my clients do a project, they're budgeting out for the year, so that's $8k/year, not per month.

AWS would be cheaper with reserved instances, but then (IIRC) you're prepaying in advance for something you may not need. Even if it's cheaper, it's sort of defeating the purpose of "i can scale on demand". (which, yes you can still do, but you're back to 'on demand' pricing).

As someone else pointed out, some of where AWS gets you is the storage and traffic. I've got a setup right now where I need to have at peak 2-3 servers to handle traffic, as there's a moderate amount of computation going on - it's not all the time, nor every day, but when I need it, I need it. And it's processing documents (photos, etc) and we need to store them. Then show them back to the user. With on demand pricing, and the amount of traffic, this can be ~$600+ per month (*12 = ~$7000/year).

Could I optimize the code more to use less CPU, and perhaps require 1 less server? Possibly. Could I optimize some other stuff? Possibly, but that will cost more time, in order to save money with AWS.

Reserved instances would probably be the best, but if we ended up needing more later, it's harder to go back 6 months later and say "hey, we need more $ for X", as the budget was already done and processed for project Y.

I've got some projects like this at digitalocean, and one on month-to-month dedicated hardware, and the pricing is about 50%, mostly because of the built-in storage that AWS line items separately.

Can AWS do more functionally? Sure. Do we need it? Usually not, and paying for that infrastructure when it doesn't match the core needs isn't usually an option. Elastic IPs would be nice, but we could get much of that utility at linode if need be, and still pay less.

Reserves instances long-term don't make much p&l sense, it makes more sense to rent dedicated hw where most of the other costs are cheaper.

Agreed, which is why I don't do it or recommend it. I had a client look in to it once, and it was a toss up between renting dedicated servers vs reserved instances, and they went dedicated.

> AWS would be cheaper with reserved instances, but then (IIRC) you're prepaying in advance for something you may not need. Even if it's cheaper, it's sort of defeating the purpose of "i can scale on demand". (which, yes you can still do, but you're back to 'on demand' pricing).

That's why you mix. We know our base needs, these are reserved instances 100% utilization. When something comes up that drives a lot of traffic the on demand instances kick in automatically. If you have somewhat "regular" spikey traffic you can buy reserved instances with less utilization and still save vs raw on-demand pricing.

I'll be the first to say we could do this all cheaper with some other VPS and a bunch more labor, but we are using a _ton_ of AWS services. My personal/moonlighting customers I use DO where a single nginx/sql droplet can handle everything with ease for small load sites

Well -- when you are spending 6+ million a year on AWS, a smaller VPS provider might not work out because of scale.

This absolutely hits the nail on the head.

The dumb part is that the next option down, renting dedicated hardware, seems to be effectively identical with regard to server administration overhead. The only difference is that the time to provision is longer (4 hours with our host), and billing monthly instead of hourly.

In either case, you need to know how you are setting up new machines, monitoring them and deploying your application to them.

The part that really kills me is this though. Using AWS is supposed to be good for start-ups because it allows you to get started quickly and think about building your product rather than focusing on technology that someone else could handle. The problem is, how you are going to scale your technology on AWS is a problem you need to think about very early.

Even the lowest end dedicated hardware is so fast that you can have millions of daily active users before you even need to move beyond a single box. It's amazing how many things you don't need to think about when you only have a single box.

I've done both. In my previous startup (circa 2005) we started at The Planet (are they still around?) and within 18 months we were colocating physical servers at a nearby Level 3.

I'm sure it's gotten better but when we colocated I had to project what type of traffic and storage (photos) we'd need. That led us to lease a few servers from HP. We were able to reuse hardware we owned already for reporting type tasks. But I got 1TB storage servers when we were using about 1/5 of that.

In my current startup (photos, again) we were on AWS from the beginning. I think we could save money now if we wanted. But when we first started out all we needed were 2 micro instances and S3.

I'd say we've increased capacity by 8x since then.

But here's the kicker. Since we started on AWS we were able to keep our costs fairly low at the beginning and as we grew we never had to make any long term projections. We simply added capacity (literally) on a weekly basis.

Once you start to buy into the AWS ecosystem you begin to reap some serious benefits. You simply cannot think of AWS as just EC2 instances. Of course, the trade-off by going all in is some level of lock-in (though this can easily be minimized).

Having done both I am happy to pay AWS more for the ecosystem.

We're using EC2, ELB, SNS, ElasticTranscoder, RDS, S3, Route53, CloudSearch, CloudFront & Glacier.

ThePlanet was purchased by Softlayer, who was then purchased by IBM last summer. So they're now IBM.

But here's the kicker. Since we started on AWS we were able to keep our costs fairly low at the beginning and as we grew we never had to make any long term projections. We simply added capacity (literally) on a weekly basis.

The exact same thing can be said for renting dedicated servers. You can add new ones at any time and it's fairly quick. This is exactly how we scaled up as well. Adding new servers day to day as required.

Yes, it takes 4 hours rather than 4 minutes but unless you are dynamically scaling up and down throughout the day in direct response to load, you are not using the versatility that you are paying for with AWS. Few people are.

Once you start to buy into the AWS ecosystem you begin to reap some serious benefits. You simply cannot think of AWS as just EC2 instances. Of course, the trade-off by going all in is some level of lock-in (though this can easily be minimized).

And there is nothing stopping you using these other cloud services without EC2. If you look around at other vendors, you might even be surprised at the prices you can get.

For example, we worked out at one point that in our case CloudFront would cost over over $50,000 per month for what we were paying about $4000 per month with a different CDN.

It's entirely feasible that things are much better than they were a decade ago. I sure hope they are :).

Perhaps it's slightly more expensive to start with AWS, cheaper as you scale (WRT man hours) and gets (prohibitively) more expensive once you hit a certain scale.

I don't doubt there are plenty of reasons to be off AWS. Lots of evidence exists to support that. And vice versa.

It's amazing how many things you don't need to think about when you only have a single box.

This. On the mobile app/social network thingy I'm working on now, I reduced it down to a single process. I've got one thing to watch, and on a single box, I expect—like you said—to scale to millions of users.

One that not many people seem to know about but which interests me significantly is http://www.instacompute.com/.

It's run by Tata, which is pretty massive plus according to http://cloudharmony.com/ it's got pretty good performance, especially considering the price.

I guess problem for most of you (it seems pretty US centered around here) may be that they don't have a presence in the United States (they're good in Europe, Asia and India though).

Anyone have any experiences with them?

The clue is in your answer: you didn't name a single alternative.

I think that at this point EC2 should almost be described as a scam, except for cases where people are going to rely on several of the AWS services like DynamoDB that has little to no competition, and are also relying on having very low latency connections to those other AWS services from their EC2 services. I think scam applies just because the cost differential is so much and only exists because people don't know about other services.

But it is weird that people don't know about Linode or Digital Ocean (which is incredibly cheap compared to AWS and actually works, which the micro EC2 instances often stop working if you put a load on them). Even Rackspace would be much better option.

So even if you are just thinking of very popular companies, there are well known reliable options out there that are vastly better values for most things than EC2. Then there are probably dozens or even hundreds of small companies that would probably provided even better value and service.

To be clear, we haven't dumped AWS for short running stateless processing. We dynamically spin up spot instances as needed for all our stateless processing using AWS and other cloud services. At our current size our AWS bill will not be 7 million but closer to 500k-1 million a year. Not exactly pocket change. Some of the cost savings we are realizing is due to working with AWS as well on best practices.

For our longer stateful processing or apps that need to be available 24/7 with no variability in load we have purchased our own hardware (a process that has been going on for over 18 months). Owning the equipment plus the data center will run us approx 1.2 million including growth to build a hot back up.

It should be noted, staffing cost was not a factor. We must have staff to manage 1000s of servers at AWS or at our own data centers. The biggest factor was paying for compute on boxes that crashed and yielded nothing we could use to move our business forward. Well, I take that back, we got really good at check points and rollbacks. Other than that, not much.

No matter how you slice it, AWS and other cloud services are a great service for the right types of processing, and applications.

Skinner Moz CTO

Thanks for the additional information. It was difficult to decipher your problems from the original article. I'm guessing the "no variability in load" processes are where AWS was the most unnecessary.

Yes! 95% of our services were running at AWS. Paying AWS for running our website is not a good use of money. Many cheaper ways to host a website.

Paying for 70-100 hours of compute and having the server crash in the middle of calculating your predictive analytics not exactly all that bright either. So, we bought our own gear.

AWS works and worked with us pretty damn closely as we pulled apps out of the cloud.

I'm curious to know if you considered other hosting providers as an alternative to building out your own hardware/datacenters? I think it's reasonably well known that AWS is around 3 times more expensive than alternatives where you pay by-the-month instead of by-the-minute. Was a Rackspace/Linode/whoever implementation costed against a buy-your-own-boxes solution, and if so, is there anything you could share about why you chose the way you did?

(Oh, and thanks for the information you've already shared - even if you can't answer my curiosity here…)

We use other cloud services as well. Rackspace and Nimbix are a couple of them. We even looked at Azure figuring no contention for boxes but we don't have any MSFT in our stack (sorry to my whole neighborhood of MSFT employees). We never depend on just one and have a detailed cost breakdown before we decided to buy or move a service. The ROI needs to be there.

As an aside, AWS has everyone beat when it comes to regions however. We can be close to our customers in Europe, US and so on.

To date, no one is spinning up cloud fronts and services in more areas than AWS. It will take the MSFT , IBMs and the like to move the global cloud along. MSFT just needs to realize not everyone wants Sharepoint, SQLServer and .net.

Which service you use is really situational. Plenty of good ones out there. AWS is just one!

I'm very curious about your ROI analysis for colo vs. a service like Rackspace's "Managed Colocation" [1] where a hosting provider runs the DC and provides the hardware and your folks manage the OS layer and up.

I run technical operations for a company with a fraction of your footprint (but growing quickly). We're at the point where we are growing out of the RAX public cloud but by my calculations, the decision to run our own private cloud in colocation vs. lease one from a service provider is (financially) a wash.

From a practicality standpoint, the scales tip towards leasing bare metal from a provider. I'm curious to hear your experiences with colo. How many folks do you have working in your colocation facilities doing hardware maintenance? What about network engineering? I presume you also keep a sizable stock of spares?

1: http://www.rackspace.com/managed_hosting/managed_colocation/

All good questions. The answer is pretty long winded. In short, we love our bare metal and you are not going to be able to rip it out of our cold dead hands. It is stable, fast and cost efficient.

Now for the long winded:

1. We do have a decent amount of spares but not a ton. Our contracts require replacement parts within hours to a day. Some items like F5 gear we have two and no spare. They just replace their gear in hours.

2. Each data center has 24/7 support that can do some minor tasks.

3. Yes, Networking is a pain and you need the right people to do it. It is not cheap either! Luckily our VP of Tech Ops is a networking guru.You mess up networking and you are hosed. Our first networking guy wasn't exactly Tops! So, we know first hand.

Having said all of that, for us there are economies of scale. It is the case if we want to test different machines, databases or any other combination at scale it could cost us several hundred thousand dollars just to run the tests. Yep, we have dropped over 100k for testing at scale. Its simply not sustainable and an irresponsible way to spend investors cash. Also, when you add in multiple environments for dev, test, staging and integration you can quickly see we consume a lot of boxes. So, many in fact, a lot of colocation/cloud services will not work with us unless we plop down large amounts of cash. Let's also factor in AWS wants large up front spend for reserved instances. Thus, if you need the capital to get the amount of compute you need to run your business it is not very hard just to call Dell, Cisco, Nimbix, Equinix or any other vendor and negotiate our own deals. If any of these companies can get half our spend a year they are willing to at least talk.

AWS tends to pick locations based on cost rather than their proximity to major Internet hubs, so the number of physical locations they actually have for most of their services is deceptive. They only have 9 available regions, and their latency is going to be much higher than most other providers that are located right by major Internet hubs instead in the middle of nowhere. We're a company (dedicated hosting provider) with only a team of 5 people, and we already have colo in 7 locations, which has them handily beat in North America and matched in Europe. There are also several VPS providers and resellers who simply rent dedicated servers from multiple companies like us with more locations than AWS. I'm looking at the website of one of our clients now who has 20 meaningful locations. Akamai absolutely destroys Cloud Front and everyone else in terms of presence, and a number of other CDNs have them handily beat. Neither AWS nor Cloud Front are anywhere near being leaders in terms of meaningful geographical (i.e. network) presence.

Not surprised to see this, although I am surprised more companies aren't following suit. We abandoned AWS a year and a half ago and cut our server / infrastructure costs by 90% in the process.

Care to share more details? Which services were you using? Most everything reserved? Were you taking advantage of auto scaling / was your traffic very elastic? Do you have as much redundancy in your new host / do you have redundancy requirements?

We were using EC2 and RDS (we still use S3 quite a bit).

Nothing reserved because we never wanted to be tied to the platform for long, although we ended up staying on longer than I wanted to due to difficulties in migrating away.

Yes, I realize we could have saved a lot with longer contracts. But the costs would still have been much higher than what we're paying now.

RDS crashed (completely) for us 2 years in a row, on exactly July 4th. We never got a straight answer from Amazon and their support system is a nightmare. Performance was always an issue.

Our main servers are now hosted on Hetzner (surprisingly).

We've had issues here and there with individual machine performance but nothing catastrophic and their customer service has been good.

For redundancy, we replicate back to the US onto Rackspace (and small EC2 instances for additional redundancy) servers and can switch over completely if need be.

We were worried that latency (because servers are in Germany) would be an issue but it hasn't. Also, in addition to US customers, we have a lot of customers in Japan so it doesn't seem to make much difference.

It's probably not for everybody and it's certainly not "cool" but it's worked very well for us.

Edit: our needs are relatively inelastic, which obviously makes AWS a poor choice to begin with. That was something we learned a little later in life as a startup.

who did you migrate to?

Yes, more specifics would be nice.

Yup. Folks like Netflix and Zynga are showing off how much money they are willing to spend for more potential agility. They could save a ton for predictable capacity by deploying dedicated boxes + openstack. That's where the savings are obvious, because AWS is like living at a hotel.

You're assuming Netflix and Zynga are paying the prices that are published on the webpage. I'm fairly confidant that that isn't the case.

Zynga are actually "well known" for having a hybrid model and running the steady state games on their private cloud - http://wikibon.org/wiki/v/Zynga_Cloud_Case_Study:_The_journe...

Public cloud is not for everyone. If you have a large number of machines that have to run all the time and are being taxed, then public cloud services do not make sense. If you are doing complex machine learning, large scale web crawling/scrapping, or generating large quantities of data (images/video/etc.) then services like AWS might not be a fit. They are incredibly awesome though for certain types of workload, but for others it is a very expensive option that isn't economical.

Personally I think the best is some mix of both public and private, so you have the scalability when you really spike but have cheaper more capable machines handling baseline and things like analysis or storage workloads.

It's for people not making enough money (or not having the clue) to hire systems talent.

If you read the Moz report, they appear to have gotten a new CEO and are now building out a datacenter presence. They spent 7MM+/yr on AWS in 2013, there is no way that won't be viciously slashed by renting racks.

> It's for people not making enough money (or not having the clue) to hire systems talent.

Yeah, this isn't really surprising. As soon as the convenience/expertise 'tax' exceeds hiring your own, it makes sense to make the jump.

Are people surprised that big enough organizations have their own accountants or lawyers?

> It's for people not making enough money (or not having the clue) to hire systems talent.

That may be true compared to doing your own racks, but not compared to renting managed servers at a monthly basis.

Every time I price out AWS vs alternatives, I end up with the same: If you need a server for more than about 8 hours a day, renting a managed server at a monthly basis tends to come out substantially cheaper. And that doesn't increase sysadmin workload; in fact many hosting providers now offers APIs for provisioning monthly billed services too, just generally with lead times in hours instead of minutes/seconds, so it takes really bursty traffic before taking the hassle of auto-scaling with cloud instances becomes worthwhile.

That could hire 18 staff w/ solid chops in Silicon Valley and still have plenty for gear, software and vendors.

What I find most perplexing is that people don't really understand that the point of Amazon is elasticity to make capacity planning easier for high growth situations.

That elasticity only makes financial sense when you use it properly. Like, if you keep your entire infrastructure turned on all the time, and cranked up to 11, but you aren't using reserved instances, then you are missing the point and you might as well just build/rent your own datacenter space and run your own servers (or pay someone like Rackspace to do it).

If you are using AWS, then you need to make use of reserved instances and the elasticity to spin down things that are being underutilized. If your infrastructure is too fragile to safely handle spinning up and down instances, then you probably are missing the cloud or aren't taking full advantage of what it has to offer.

I'm not saying Moz wasn't doing all of these things, I'm just saying that if you are spending $7 million a year, but aren't doing the appropriate capacity planning, then you are making a mistake.

At the very least, the exercise of doing real capacity planning by moving into physical data centers probably had a lot to do with the cost savings as well.

What you're saying may apply to simple SaaS companies who might build a two-tier application that can be easily scaled up and down by adding front-end nodes and r/o database nodes, but it doesn't always apply to data analysis companies like Moz, who may use large ElasticSearch clusters that don't autoscale. I run technical operations for such a company. The vast majority of our infrastructure remains static, simply because the underlying technologies don't allow for autoscale. The best place for a company like ours is in a private cloud, be it at Rackspace, SoftLayer, or in colocation.

Right, and that makes total sense. I haven't ran the numbers, but I know that if you have some amount of fixed capacity required, you would absolutely want to do reserved instances on all of those and consolidate as much as you can where it makes sense.

However, if the numbers don't work out on reserved instances and you have a pretty steady demand, going with your own infrastructure makes total sense.

My employer spends a small fortune with AWS, and we make use of many of the services. There's no way our small team could have built our product on any other platform.

Still, as my colleague observed, it must be the only business in the world you can spend $1m a year on and get only basic support, with web-only tickets that take hours or days to get a resolved.

Why aren't you spending even the $50 support tier if not enough for one of the higher tiers if you're dropping that much in usage costs? I don't understand why anyone expects free support when every little thing in AWS is charged for as a utility.

Sorry, I didn't phrase that well - it wasn't really supposed to be a criticism of AWS, but rather (what I thought to be) an interesting observation about how their model differs from most services.

To be fair to AWS we don't need support often; we really only have problems when everyone else does (major outages) and we can follow the status of that online, we don't need to call someone to be told "we're working on it". Also our account manager is responsive and pro-active in helping us out, and they've put us on free Business Support trials a couple of times, but because we need it so infrequently there's really no reason for us to fork out for it.

Support contracts are a form of backup.

Except with AWS you can start a support contract in about a minute, via the website, and turn it off after a week when you're done.

Well I guess I don't understand GP's point, then.

The paid support tier is also atrocious and close to useless. AWS routinely denies the existence of major flaws for months or years, until they eventually admit it, often by just documenting it instead of fixing it.

They're horrifically expensive, and low quality.

It's really, really sad that so many startups have been convinced that it's "cheap" when it's almost comically expensive and lousy.

I don't mean to claim otherwise, but I'd be interested to see what instances led to your observation.

As a simple example, if you have a moderate number of hosts in a single security group (e.g. 200+ hosts), you can expect to have intermittent communication problems between the nodes. They used to deny this was a problem.

here's a source, since you're attacking and disbelieving everybody who doesn't love AWS: http://searchcloudcomputing.techtarget.com/news/2240203992/N...

Asking politely phrased questions is not an "attack". I just looked through the last 30 days of his comments, and I don't see anything attack-like there.

It may or may not be the same issue, but from what I have been able to gather a modification of a security group is basically a "delete, recreate, repopulate" operation. All of our intermittent network issues could be reasonably tied back to SG modifications.

That's a different issue. This one is instances that are launched into an sg that is then left unmodified. If there are a moderate to large number of instances in that sg, intermittent network connectivity problems will ensue.

The issue you mention (where you have to treat sg's as being immutable if you want them to work reliably) is another problem with the sg's.

I suggest trying to asking for help with network problems. One of my admins spent a month going back and forth only to be told "nothing to be done, it's fine."

It really is truly terrible support.

I've found the opposite - the support techs I've talked to have been technically proficient, and willing to spin up VMs and test things out for me.

It's not because it's cheap, it's because you can avoid capacity planning and blowing capital (for which you would need an ROI). Once you are big enough that you hire your own accounts, lawyers & ops, then you can make a capital investment into your hosting infrastructure get a return on it. Too early and that was money that could have been better invested into growth opportunities.

> There's no way our small team could have built our product on any other platform.

did you actually try, or are you just rationalizing your enormous AWS spend?

Did anyone else think "Moz" was short for Mozilla?

Yes

Not I, but I'm part of their demographic -- http://d2v4zi8pl64nxt.cloudfront.net/2013-in-review/52e9b643...

Weird. I recognize only half the companies on that chart, and I'm not sure what Adobe or salesforce have to with marketing.

Adobe are huge in the enterprise analytics world. I would argue they are kind of shitty at it but they are huge all the same.

so you're telling us you don't know what content creation or CRM has to do with marketing?

> I'm not sure what Adobe or salesforce have to do with marketing

They're some of the largest players in marketing and analytics, especially since Adobe's $1.8B acquisition of Omniture in '09.

http://www.adobe.com/solutions/digital-marketing.html

http://www.salesforce.com/marketing-cloud/overview/

Something's weird about that graphic. Do 17% only of people really recognize the brand Microsoft?

Or (more likely) is it that only 17% of people realized they do "marketing technology"?

I'm guessing it's the latter, and as this was put out by Marketing Land, it was a poll of their readers. I don't necessarily associate MS with marketing, but they probably mean Bing as a marketing channel.

At the first glance, yes, I then quickly realized it was about the SEO company. I think it has a lot to do with being a developer and using the css prefix -moz for mozilla based browsers.

I did at first, and I am a Mozilla engineer. Mozilla makes heavy use of AWS for running portions of our continuous integration system; the cost is significant:

http://oduinn.com/blog/2013/12/13/the-financial-cost-of-a-ch...

It's pretty staggering that every checkin and try run costs 30 bucks.

I bet one of the leading factors is the number of sites blocking the AWS servers. As a scraping service, the fact that Amazon makes all their public IPs known (https://forums.aws.amazon.com/ann.jspa?annID=1701) is really inconvenient for anyone "crawling" the web. Rackspace also makes their list available, albeit incomplete (http://www.rackspace.com/knowledge_center/article/cloud-site...).

As more websites setup default blocks for all accesses from AWS services, the Moz exit was inevitable.

We've actually never crawled out of Amazon (mostly because it was expensive to do so), so the crawl blocking stuff is unrelated, at least for us. As Sarah noted, it's really been costs, service, and support issues.

I'd redact my suggestion then if I could. Thanks for answering so directly and promptly! (and great seeing you on HN)

Thanks dude! Totally fair question & assumption BTW :-)

What type of hosting infrastructure works best for crawling?

> Rackspace also makes their list available, albeit incomplete (http://www.rackspace.com/knowledge_center/article/cloud-site...).

FWIW, this is specific to Cloud Sites, which is like one-click Wordpress/Drupal deployments.

You can just use BGP to find out all that information easily enough for any network with an AS number (i.e. any who speaks BGP). Just take a look here: http://bgp.he.net or a plethora of looking glasses like http://lg.level3.net or publicly available route servers like telnet://route-views.routeviews.org

there are dozens, if not hundreds of smaller TRUE managed hosting providers popping up to service these kinds of accounts. that's what my startup does. we routinely steal large $5k-50k/month accounts from AWS and rackspace and softlayer (IBM) and the like. they just do a shitty job for an exorbitant price, plain and simple.

folks... NOBODY gives a fuck about you at those companies, and you better believe it. not the account manager, not the engineer working on your 5 day old ticket, not the sales guy who probably isn't even paid a decent commission, and certainly not the executive management who is pulling down 300k+/year annual salaries plus huge stock grants and bonuses (check out the SEC filings!)

a well managed cluster of computing equipment running baremetal operating systems, hypervisors of your choice, and maybe something like openstack or docker is both profitable for the vendor and a MASSIVE price reduction for the customer.

if you know what you're doing, you're spending $10k/month for $100k/month worth of equivalent amazon services. yes, you heard me right. it is an order of magnitude of cost savings. once you scale past $10k/month, using amazon for all your needs is an INCREDIBLY STUPID business decision.

the genius of AWS, of course, is that they disguise all their marketing as technical spec sheets and deliberately design their collateral and sales process to appeal to the "engineer" types with acronyms, numbers, and highly specific jargon.

it's quite clever. but not clever enough to fool everyone. and those that are fooled, learn eventually that maybe paying 300% more than you should isn't that smart.

I agree. AWS has got to the be worst scam ever. But they're clever, I admit with all their jargon. They give you a pricing calculator but have so many caveats that it's so hard to compute the true price of what you're buying (not to mention the nickel and diming. a cent per I/O cycle or something? what does that even MEAN?)

Disclaimer: Been using OVH for the past year. Sure, there's downtime (like 1 hour every couple of months), but it's so cheap I can afford a lot of redundancy.

I'm interested in knowing more since we do development and system integration for client in this kind of budget, and we didn't find a decent hosting provider in years of trying. Could you name some of those managed providers, including the one you work for?

i put my email in my profile. i don't promote my company by name on HN, but send over your requirements and we can take a look.

thanks.

Is there a way to contact you?

my profile has my email address now.

To save everyone a click: it appears he's removed his email from his profile. (Too much interest?)

I'm curious about what is behind Moz's statement "The business impact is profound. We’re spending less and have improved reliability and efficiency."

I don't have the experience with large AWS setups to comment about the cost implications, but improved reliability and efficiency strikes me as questionable - AWS's reliability has improved markedly over the last 18 months to the point I cannot remember when their last major outage was.

I'm keen to hear other peoples' thoughts on this..

There may not have been a recent total outage of the kind that hits the news, but I would bet Moz are referring to degraded service (slow CPU/disk/network) that causes their application to fail (or their error rate to rise).

I'm not very familiar with Moz, but it looks like they have an analytics product, which is probably particularly sensitive to slowdowns.

Sounds like they are more at a size that makes sense to have an on site setup vs. a cloud based one.

The cloud market is maturing now and a hybrid private/public model I think is going to become more of the norm, for companies that are large enough that this makes sense.

It is cheaper to run your own hardware when you are big enough, and then use AWS for failover / load spikes / cool tech, etc.

The scale where doing your own thing makes financial sense can be much lower than many think.

About two thousand dollars a month gets half a rack with a high-speed, highly-reliable connection in co-location. The lease payments on several extremely high performance, flash storage rack servers would be a couple thousand dollars a month. Throw on some shared nothing redundant virtualization, and you have the equivalent of an army of 4xmegaxlargemongos.

AWS has a place (such as being the redundant offsite backup for that scenario, or for extremely variable service requirements), but endlessly on here we see people disbelieving alternatives make sense.

I own 15 OVH servers (32 GB RAM, 2 TB HD, quad-core), and the cost monthly is less than $1000. What can that buy more in Amazon? Maybe a huge instance? But I surely can't run thousands of distributed crawlers and store a distributed search index.

You can get some decent beef with the new C3 instances, but you have to pay a good chunk of the year upfront (called a reservation) to get the sane rates. If you do that, the prices are OK.

But you don't come to AWS just for EC2 instances. You're there to combine their portfolio of services in various ways. Because of this, it'd probably be wrong to just compare $1000 on AWS vs what you are paying now.

Moz is an analytics startup for marketers. It essentially crawls the internet to create a graph of links and reports that in their user interface. They also calculate a score based on the authority of links similar but not the same as Google's Pagerank. They also do social tracking.

Technical details:

I don't know exactly how they do it but my guess is that they are I/O bound on the crawling side, then CPU bound on the parsing and processing side. I'm assuming they use different machines for those tasks.

On the crawling side, the index with 60-70 billion URLS was using 80 cc2.8xlarge machines with a backup on 200 c1.xlarge machines. (http://moz.com/blog/one-step-back-two-steps-forward)

(DISCLAIMER: I'm in the same industry. I am a developer with working for an indirect competitor. I know a few engineers and non-engineers who work at Moz.)

Rand replied that they don't use AWS for crawling.

I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.

A big thing you fight with services like AWS is a lack of transparency - everything's very opaque, and so when you have something like an IO degradation on a given node, you've got very little to go off of, and you have to try and decipher if it's that you're seeing neighbor pressure on that node, if you're on degraded hardware, or if you're just abusing the hardware inappropriately.

Moving to your own hardware lets you actually reason about what's going on in a controlled environment, and at the scale that Moz is working at allows us to get hardware that's robust, well-supported, and reliable enough to have a pretty serious net positive overall.

Cloud premise is an ability to fire expensive sysadmins and let cloud babysit your needs. Amazon was the first in the world to create the cloud but it's still a bare-metal cloud today. "Lacking service" ? Of course - they didn't promise any service whatsoever to begin with.

In other words you pay more to use Amazon cloud + you still have to hire and pay sysadmins to babysit the whole pile of Amazon cloud mess.

That's what MSFT is perfectly positioned to solve with Azure + their excellent set of mature tools and services.

MSFT came late to the cloud game (as Ballmer was mostly doing in all other games for the last 10 yrs) - but they are getting this enterprise play right step by step.

(I was part of a preferred AWS solution provider shop that demonstrated AWS' enterpriseness.)

Pair (US), ByteMark (UK) and Linode (global) are top choices for budget stuff. ByteMark has setup VMware boxes with freenas storage for us in the past. (For us being sysadmins, budget dedicated is way cheaper than AWS.). Pair and Bytemark support are top notch.

For enterprisey stuff self-service/remote hands/support... OpSource, SoftLayer and RackSpace.

Also the killer cost in a datacenter is power followed closely by labor, so reducing needs by using the simplest / COTS solution can be much cheaper. Servers tend to sprawl, not decrease... So cull viciously.

At some point, I'm going to guess that even moderately large AWS configurations would be better off getting a high capacity business FiOS line (or whatever the next tier is), get a couple racks full of home rolled high-end systems and staff 2 or 3 guys full time to run everything. The systems will be faster, probably about as reliable, everything will be cheaper even with extra people on payroll.

I'd love to see some numbers on this though.

Me too. I suspect there's medium and longterm costs that probably skew my "gut feel" of how much cheaper that'd be. I wonder what the real costs associated with feeding a roomful of local servers replacement disks/ram/powersupplies are, and what a realistic replacement schedule for the hardware is?

Sure AWS is expensive for 24/7/365 applications, but I suspect there's some "in between ground" where Amazon's economies of scale make it less expensive to just pay them to deal with owning and maintaining the hardware (and ensuring you write your applications to be able to deal with cloud instances vanishing underneath you randomly).

If you want a worldwide site though, AWS gets you in 10 locations around the world. This would take even an experienced admin a month for just the admin stuff to get going. An inexperienced one ... half a year ?

(but it's still better to have a webserver on AWS, that simply 302s to your "real" services, or only provides a skeleton web page, though 302'ing is better, without it you'll quickly run into difficulties with same origin policy. 302 redirects will also change much faster than dns updates).

Where do you just throw a couple racks of computers? Closet? Sounds great, now let's deal with cooling them and the maintenance around that. Then there is power. You think your standard office had all the necessary circuits to drive a couple racks full of servers?

There is so much more to running a server farm than just buying a couple racks of computers and upgrading the FiOS line.

Yes, to all of that. Apparently you weren't around 15 years ago when startups pretty much had to do that if they wanted servers and an internet presence. I lived through a startup during that era and that exactly what we did. We had our fancy ISDN line and could server our web site to the world at a blistering 128K baud.

We bought a $300 air conditioner from Home Depot, cut a whole in the server room door with a sawzall and installed the air conditioner. Ok technically we only used the air conditioner in that configuration on the weekends when there was a human in the room and the main building air was off (we couldn't override it) but I like that story.

You're not going to get 5 nines this way, but you would be surprised at how much you can do, even when there exactly zero dedicated sysadmins.

Having helped put together at least half a dozen such rooms over the year (not an expert, but I can cobble together hardware into a 1u and 2u case as well as anybody and then screw them into a rack) I'm frankly surprised by the lack of both can-do attitude and basic know how here on HN. There's a weird notion here that additional lines can't be run for more bandwidth, and electricians don't exist who can come run more circuits to give you enough amperage for a few racks full of systems. Does anybody do anything for themselves anymore? This stuff isn't rocket science and it's not that hard to do on your own.

And it's surprisingly less hardware than most people think. Stack Overflow runs off of two machines. A few pieces of decent networking equipment and some fans and they're probably not even consuming half a rack.

Turns out running their own hardware is exactly what they ended up doing and even with staff and other costs, runs a fraction of AWS. https://news.ycombinator.com/item?id=7155165

I know it is possible, and have also done it. I just get tired of people saying you can match what AWS provides for less money in house by buying some servers and upgrading your FiOS line. It is disingenuous and simply not true.

In reality the sweet spot is in the planning. If talking about only compute, colo your 24/7 servers and spin up extra capacity when needed in the cloud provider of your choice (what Moz appears to be moving towards).

Something we make heavy use of is Glacier and I don't know of a good alternative on that front.

Well you can't match what AWS provides. Almost nobody strictly can. But it's over-engineered and more than what most businesses could ever possibly need or want. Some questions, does your English only web startup really need redundancy in 30+ countries? Do you really need five 9 reliability or can you get by just fine with four 9s? etc.

For what 95% of the people here on HN need, 2 or 3 decent servers running off of a business FiOS line will be more than adequate for even their wildest ambitions. 99% of the remainder could make do with a rack or two of equipment, a generator, dedicated a/c and some loving electrician work. And even then if they need quick surge capacity expansion, they can still bring in AWS to augment their home-rolled stuff.

If keeping your stuff in-house isn't sexy enough, colo'ing it is still cheaper than AWS in most cases.

There's probably a fraction of 1% of the startups here that actually need what AWS offers. But the prevailing attitude here is that everybody's appointment reminder photo sharing web app is going to require AWS when they'd be perfectly well serviced, and be far more responsible with investor money, with something far less capable. The problem, as I'm seeing in the comments here, is that basic industry know-how (like how to assembly a reliable high-performance server from parts) seems to be largely absent in the industry.

I sincerely hope you never deploy a system then. Not trying to attack you, but I don't want anyone to get the wrong idea. Deploying servers is not as easy as you make it out to be. If you have the dough to hire 2-3 people to run your servers, you can afford to do it right.

I would never roll with a FiOS line to run my services, for one I doubt their TOS supports it. I'm not sure anything but the smallest of startups could pull that off, and even then they'd be better served spinning up a few droplets on DO.

If you have a 'moderately large' AWS configuration, you need a corresponding 'moderately large' DC deployment with plans for redundant network links and redundant power. I can't express the importance of that: anything you run that you care about needs to be in a controlled, monitored environment. Not a home-rolled franken-rack powered by a single non-symmetric line without an SLA. Not to mention proper fire suppression.

FiOS business lines explicitly exist to allow companies to run their own servers and most good companies that self-host build their own hardware. It's not a "franken-rack" (clever term), it's pretty common industry practice (google even builds their own server hardware). That's why you hire 2 or 3 guys who know what they are doing. Hell, I barely know what I'm doing and I've probably helped cobble together half a dozen such setups over the years.

In some industries, designating some office space for your "server room" and building it out to data center specs is pretty common and surprisingly not all that expensive. I'm surprised by both that lack of can-do attitude here on HN as well as the basic know-how about it. I guess the demographic here skews towards young-hip-cloud-framework-I-don't-know-how-to-do-anything-for-myself-everything. But it wasn't all that long ago that buying some empty 1u and 2u cases, and sources parts from the local hardware shop, installing them in racks that you bolted to the floor of your server room, that you fitted out with redundant backbones, industrial a/c and enough amperage to blow out the transformer down the street was what people actually did and it's not that hard. Hell, one of the points of RAID was to enable companies trying to save a buck on such setups to exist in the face of very expensive enterprise class hardware disk options.

(edit) and this is exactly what they did https://news.ycombinator.com/item?id=7155165

There's always a compelling case for that until your line breaks or your power fails. I've seen data centres fail too, but they do have some redundancy for their systems, more than you could reasonably spec for your own premises.

I've helped do it at least a half dozen times. It's not really that hard and you'll make up the costs pretty quickly if you really need all of that.

To save myself chastising yet another HN'er about this my last comment

https://news.ycombinator.com/item?id=7156521

A real datacenter is almost certainly Better Than You at reliability. Consider:

- Redundant fiber connections to multiple independent ISPs

- Battery backup and generator failover

- Redundant HVAC

- Waterless fire suppression

- 24x7 staff presence

- Security systems and personnel

These are all economies of scale. If you're just running a couple of racks you're either throwing away money by paying for all this or making yourself irresponsibly vulnerable to things that can and will go wrong.

You just described that last 3 server rooms I ever worked in. This stuff isn't all that hard. I'm really surprised at the lack of knowhow here on HN. If people don't have businesses in the right kind of office buildings you can always just colo your own hardware and pay power/bandwidth and still save money over AWS.

> If you're just running a couple of racks you're either throwing away money by paying for all this..

You're paying for all of it anyways. Especially if you colocate, and then you're paying for their costs plus whatever they charge on top of that. Money you pay into somebody else's profits, that you can self-service is literally throwing away money.

It sounds like the OP built their own hardware, but colos it all in a datacenter. https://news.ycombinator.com/item?id=7155165

The issue is primarily cost. This stuff costs more per server at small scale. Maybe some colocation hosts are running such a large margin that they cost savings is thrown away, but others aren't.

Colocation is a great solution (in fact, the solution) to the scale issue.

While AWS can be a good solution depending on the needs and the costs/earnings ratio (for example there are companies netting tens of millions every year that have relatively limited computational requirements: if they spend 200k/year on EC2 is not the end of the world), I bet that part of its popularity is due to the following interesting feedback: at the end of the day, the sysops/devs picking EC2 are not paying for the bill. At the same time they are not going to be fired if the service is down, as "eveybody is down right now". This creates an environment where EC2 is not just a popular pick, but also praised by most, even when actually for the specific company needs other solutions could be way more cost effective or even provide a way better environment.

AWS is best for PoC, bursting capacity, batch jobs and getting around corporate policies.

It makes total sense for a small team (whether a startup or in a large organization) to be on AWS rather than going the DIY route by renting VPS out of some datacenter and managing everything from patching to migration to troubleshooting. What most fail to realize is the fact that a complete AWS package comes at a much cheaper cost than all of the above tasks combined. It just requires one to use some smart infrastructure planning in order to keep the bills low by tuning most of the configurable services which AWS provides.

AWS doesn't automatically upgrade your OS image or anything like that. If you want to do that you have to do it yourself. And AWS isn't going to troubleshoot shit unless you pay a huge amount for support which is hard to even find the option for that.

However, on Ubuntu server at least there are literally about two commands to run for upgrading the OS.

Their m1.small is still a pretty good deal if you're in need of root access to an OS image, and your capacity needs are very small. Anyone aware of anything comparable?

DigitalOcean is fabulous. It's very nice hardware and not very expensive. No AWS services, but if all you need is root access to an OS image, I can't recommend it enough.

If Moz is a privately held company, why do they publicly release their financial data?

This link would probably help answer that: http://www.feld.com/wp/archives/2012/05/seomoz-tagfee-and-me...

Reply on news.ycombinator.com