Hacker News Reader

Comment by bane | Hacker News Reader

At some point, I'm going to guess that even moderately large AWS configurations would be better off getting a high capacity business FiOS line (or whatever the next tier is), get a couple racks full of home rolled high-end systems and staff 2 or 3 guys full time to run everything. The systems will be faster, probably about as reliable, everything will be cheaper even with extra people on payroll.

I'd love to see some numbers on this though.

View on HN

I sincerely hope you never deploy a system then. Not trying to attack you, but I don't want anyone to get the wrong idea. Deploying servers is not as easy as you make it out to be. If you have the dough to hire 2-3 people to run your servers, you can afford to do it right.

I would never roll with a FiOS line to run my services, for one I doubt their TOS supports it. I'm not sure anything but the smallest of startups could pull that off, and even then they'd be better served spinning up a few droplets on DO.

If you have a 'moderately large' AWS configuration, you need a corresponding 'moderately large' DC deployment with plans for redundant network links and redundant power. I can't express the importance of that: anything you run that you care about needs to be in a controlled, monitored environment. Not a home-rolled franken-rack powered by a single non-symmetric line without an SLA. Not to mention proper fire suppression.

FiOS business lines explicitly exist to allow companies to run their own servers and most good companies that self-host build their own hardware. It's not a "franken-rack" (clever term), it's pretty common industry practice (google even builds their own server hardware). That's why you hire 2 or 3 guys who know what they are doing. Hell, I barely know what I'm doing and I've probably helped cobble together half a dozen such setups over the years.

In some industries, designating some office space for your "server room" and building it out to data center specs is pretty common and surprisingly not all that expensive. I'm surprised by both that lack of can-do attitude here on HN as well as the basic know-how about it. I guess the demographic here skews towards young-hip-cloud-framework-I-don't-know-how-to-do-anything-for-myself-everything. But it wasn't all that long ago that buying some empty 1u and 2u cases, and sources parts from the local hardware shop, installing them in racks that you bolted to the floor of your server room, that you fitted out with redundant backbones, industrial a/c and enough amperage to blow out the transformer down the street was what people actually did and it's not that hard. Hell, one of the points of RAID was to enable companies trying to save a buck on such setups to exist in the face of very expensive enterprise class hardware disk options.

(edit) and this is exactly what they did https://news.ycombinator.com/item?id=7155165

Me too. I suspect there's medium and longterm costs that probably skew my "gut feel" of how much cheaper that'd be. I wonder what the real costs associated with feeding a roomful of local servers replacement disks/ram/powersupplies are, and what a realistic replacement schedule for the hardware is?

Sure AWS is expensive for 24/7/365 applications, but I suspect there's some "in between ground" where Amazon's economies of scale make it less expensive to just pay them to deal with owning and maintaining the hardware (and ensuring you write your applications to be able to deal with cloud instances vanishing underneath you randomly).

If you want a worldwide site though, AWS gets you in 10 locations around the world. This would take even an experienced admin a month for just the admin stuff to get going. An inexperienced one ... half a year ?

(but it's still better to have a webserver on AWS, that simply 302s to your "real" services, or only provides a skeleton web page, though 302'ing is better, without it you'll quickly run into difficulties with same origin policy. 302 redirects will also change much faster than dns updates).

A real datacenter is almost certainly Better Than You at reliability. Consider:

- Redundant fiber connections to multiple independent ISPs

- Battery backup and generator failover

- Redundant HVAC

- Waterless fire suppression

- 24x7 staff presence

- Security systems and personnel

These are all economies of scale. If you're just running a couple of racks you're either throwing away money by paying for all this or making yourself irresponsibly vulnerable to things that can and will go wrong.

You just described that last 3 server rooms I ever worked in. This stuff isn't all that hard. I'm really surprised at the lack of knowhow here on HN. If people don't have businesses in the right kind of office buildings you can always just colo your own hardware and pay power/bandwidth and still save money over AWS.

> If you're just running a couple of racks you're either throwing away money by paying for all this..

You're paying for all of it anyways. Especially if you colocate, and then you're paying for their costs plus whatever they charge on top of that. Money you pay into somebody else's profits, that you can self-service is literally throwing away money.

It sounds like the OP built their own hardware, but colos it all in a datacenter. https://news.ycombinator.com/item?id=7155165

The issue is primarily cost. This stuff costs more per server at small scale. Maybe some colocation hosts are running such a large margin that they cost savings is thrown away, but others aren't.

Colocation is a great solution (in fact, the solution) to the scale issue.

Where do you just throw a couple racks of computers? Closet? Sounds great, now let's deal with cooling them and the maintenance around that. Then there is power. You think your standard office had all the necessary circuits to drive a couple racks full of servers?

There is so much more to running a server farm than just buying a couple racks of computers and upgrading the FiOS line.

Yes, to all of that. Apparently you weren't around 15 years ago when startups pretty much had to do that if they wanted servers and an internet presence. I lived through a startup during that era and that exactly what we did. We had our fancy ISDN line and could server our web site to the world at a blistering 128K baud.

We bought a $300 air conditioner from Home Depot, cut a whole in the server room door with a sawzall and installed the air conditioner. Ok technically we only used the air conditioner in that configuration on the weekends when there was a human in the room and the main building air was off (we couldn't override it) but I like that story.

You're not going to get 5 nines this way, but you would be surprised at how much you can do, even when there exactly zero dedicated sysadmins.

Having helped put together at least half a dozen such rooms over the year (not an expert, but I can cobble together hardware into a 1u and 2u case as well as anybody and then screw them into a rack) I'm frankly surprised by the lack of both can-do attitude and basic know how here on HN. There's a weird notion here that additional lines can't be run for more bandwidth, and electricians don't exist who can come run more circuits to give you enough amperage for a few racks full of systems. Does anybody do anything for themselves anymore? This stuff isn't rocket science and it's not that hard to do on your own.

And it's surprisingly less hardware than most people think. Stack Overflow runs off of two machines. A few pieces of decent networking equipment and some fans and they're probably not even consuming half a rack.

Turns out running their own hardware is exactly what they ended up doing and even with staff and other costs, runs a fraction of AWS. https://news.ycombinator.com/item?id=7155165

I know it is possible, and have also done it. I just get tired of people saying you can match what AWS provides for less money in house by buying some servers and upgrading your FiOS line. It is disingenuous and simply not true.

In reality the sweet spot is in the planning. If talking about only compute, colo your 24/7 servers and spin up extra capacity when needed in the cloud provider of your choice (what Moz appears to be moving towards).

Something we make heavy use of is Glacier and I don't know of a good alternative on that front.

Well you can't match what AWS provides. Almost nobody strictly can. But it's over-engineered and more than what most businesses could ever possibly need or want. Some questions, does your English only web startup really need redundancy in 30+ countries? Do you really need five 9 reliability or can you get by just fine with four 9s? etc.

For what 95% of the people here on HN need, 2 or 3 decent servers running off of a business FiOS line will be more than adequate for even their wildest ambitions. 99% of the remainder could make do with a rack or two of equipment, a generator, dedicated a/c and some loving electrician work. And even then if they need quick surge capacity expansion, they can still bring in AWS to augment their home-rolled stuff.

If keeping your stuff in-house isn't sexy enough, colo'ing it is still cheaper than AWS in most cases.

There's probably a fraction of 1% of the startups here that actually need what AWS offers. But the prevailing attitude here is that everybody's appointment reminder photo sharing web app is going to require AWS when they'd be perfectly well serviced, and be far more responsible with investor money, with something far less capable. The problem, as I'm seeing in the comments here, is that basic industry know-how (like how to assembly a reliable high-performance server from parts) seems to be largely absent in the industry.

There's always a compelling case for that until your line breaks or your power fails. I've seen data centres fail too, but they do have some redundancy for their systems, more than you could reasonably spec for your own premises.

I've helped do it at least a half dozen times. It's not really that hard and you'll make up the costs pretty quickly if you really need all of that.

To save myself chastising yet another HN'er about this my last comment

https://news.ycombinator.com/item?id=7156521

Reply on news.ycombinator.com