Building Servers for Fun and Prof... OK, Maybe Just for Fun
codinghorror.com
codinghorror.com
The correct comparison is his server vs a single EC2 High-Memory Double Extra Large instance with a 3 year heavy-utilization reservation. This instance costs $3100 upfront plus $0.14/hour. The total 3 year cost for this server on AWS would be 3100 + (.14 * 24 * 365 * 3) = 6779.2, or about $188.31 per month.
Sure, its more expensive. But AWS provides an insane of value on top of the server. Like instantly being able to provision additional capacity. I wouldn't be at all surprised if, on a full-loaded cost basis, it is extremely competitive with building his server. Heck, the employee salary expense of building your own server will easily drive the cost of the server well beyond the $3100 up front amazon fee.
I love building hardware too (never had a computer I didn't build except for laptops). But my mind boggles at AWS value proposition
I'm not sure how liquid it is yet. You could also equally go dumpster diving for unused virtual instances, say, if some startup goes under and liquidates.
When you reserve an instance, you are committing to a higher upfront in order to get a lower hourly for the reservation period. The low/medium/high utilization is sort of a knob that allows you to further control this upfront vs. hourly cost. With high utilization you have the highest upfront, but also the lowest hourly cost. If you are planning to run the server 24x7, this will also give you the lowest total cost.
And of course Amazon hosting is more expensive. It is more flexible; you can spin up instances at your whim. You pay for that. It would be better to compare it with standard dedicated server hosting.
When a hard drive dies under vendor support, one guy from the vendor drives out there to replace it. Half the time they don't test it after it's replaced unless you request it. Sometimes the parts are duds. Sometimes they bring the wrong part, or it doesn't fix whatever was broken on the server.
If it's the datacenter's remote hands, it can be hours, and on rare occasions days, before somebody starts working on your issue or even answers the phone or e-mail. Same issues come, and you have to have your own spare hardware for the remote hands to use.
It's highly variant based on the datacenter and vendor(s) and other issues. There's no guarantee that outsourcing will be reliable. You have to find the best one possible and build a good relationship.
Just in case, but that does come up from time to time and if it's many hours away (or worst of all an airplane ride) you will get hours/days to fix things as you noted.
For more complicated problems, our main colo was about 45 mins away from almost all the SAs. We used remote hands for colos that were an airplane ride away, and had varying (read: sometimes really shitty) results.
Looking at he.net at the moment, I see they have a deal for $1/Mbps. Presumably someone like Jeff Atwood can get twice as good a deal as that, so he'd pay around $500/month for bandwidth for those servers. Going by the cheat sheet (https://blog.cloudvertical.com/2012/10/aws-cost-cheat-sheet-...), that is within a factor of 2 of a yearly-reserved h1.4xlarge ($2263/month * 0.47 savings = $1199). It's almost equal to the three-year reserved machine ($2263 * 0.30 = $678).
Edit: He probably only needs 1 Gbps for all 4 machines, driving his bandwidth costs down by a factor of 4, but we could start to take power/space/cooling costs into account at that point.
So is he getting a better deal than that? I'd love to know where.
I'm assuming Amazon are a cut above that.
thatamurgy's Pinboard links lead to this hosting provider: http://prgmr.com/san-jose-co-location.html The machines in the original post being ~250W, they'd need the $362.50/month plan. But it's only at 10Mbps, which would only work if the application has low network requirements. Maciej appears to have gotten a much better deal, "For $206, I got 10U, 4 amps, and 100 Mbps connectivity", perhaps in exchange for promoting it through his blog. Everything's easier when you're a celebrity!
Fortunately bandwidth is one thing that has gotten substantially cheaper over time:
http://www.codinghorror.com/blog/2007/02/the-economics-of-ba...
Power is also a big cost as well. To quote the prgmr colo: "Please note; power, generally speaking, is a bigger deal than rack units. I'm more likely to let you slide on an extra rack unit than on extra watts; watts cost me real money."
Prices are from someone like L3, qwest, ntt, telephonica. Assuming youre buying 10s or 100s of gb. Don't forget to factor in a couple routers, a bunch of 10gig optics, line cards, maybe some local loops, PMs legal & accounting staff, a bunch of NEs to run it.
Really? Honest question.
But looking at general rates, I would say no.
Linux and BSD are hellaciously complex and prone to very strange behavior once you start taking them into high-performance land.
If you're OK settling with, "can read syslog and look stuff up on the web", those guys are cheap.
Being a good Linux sysadmin might be fundamentally harder than being a good EC2 sysadmin; I'm not honestly familiar enough with EC2 to know.
I would point out though that Amazon has a vested interest in making EC2 less hard, so I would be surprised if the general opinion was that EC2 administration was just as hard as Linux administration.
For junior or mid level SE I've seen about 10% of candidates meet the hiring bar. SDEs I've had less involvement, but I'd hazard 20% get an offer where I've been.
The other problem is the pipeline. Colleges keep Pumping out SDE candidates from comp sci and maths. These junior SDEs will also have a reasonably consistent skill set and experience. There are (effectively) no colleges or curriculum creating new SEs. The incoming candidates also vary widely in their depth and breadth of relevant knowledge. Pretty sure that contributes to the higher negative signal.
- The five stages of hosting: http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/
- Building servers: http://blog.pinboard.in/2012/05/a_cloud_of_my_own/
- Going colo: http://blog.pinboard.in/2012/06/going_colo/
A bunch of people are probably going to respond to Jeff's article by saying things like, "But VPS hosting means you get other people to deal with problems for you", but in reality all that means is that you're at maciej's "monastery" or "dorm room" stage of hosting, and your needs haven't yet driven you to get "the apartment".
"""* What is it with these aggro facility names? Rather than Hurricane Electric or Raging Wire, I would much prefer to host with "Calm Multihomed Oasis" or "Granite Mountain" or " Cooling Breezes Pure Sine Wave Mount Bitmore". """
This is perhaps the single most glossed over topic in the entire article. If I am a 1-5 person shop, maintaining a web app, virtualized hosting pays huge dividends in that I don't even notice if a hard drive or motherboard takes a dump. There are additional costs that come with the benefit of being abstracted away from hardware failure or geographic problems (building fire, power out, etc), and that's something that every business has to evaluate for itself.
(And yes, you can heartbeat so you have two cheap physical HAProxy machines, too. This gets into sub-blade territory, where the 1U server is internally two or more complete low-ish power servers with independent power supplies, etc. )
That's the whole premise behind FOSS, you don't need to worry about all the licenses, and the hardware is so cheap it is effectively free and getting freer by the second, so you throw a lot of cheap hardware at the problem.
But agreed, "lot" in this case means at least two so one can fail and you don't need to care.
Ultimately, I think it comes down to priority instead of possibility. If your company lives and dies on having reliable servers, you should probably roll your own. But if servers are 'just' a technical detail to your overall business model, then a cloud solution can be well worth the additional cost
So yeah, you gotta think about it. A lot of the time, public cloud is the correct solution; however, you should have a solid understanding of what you need to do to run reliably in that cloud, how to build redundancy in the cloud you pick, when you might need to move to a different solution, and how to make those processes easier.
http://arstechnica.com/business/2011/09/google-devops-and-di... is a bit overdramatized but the final section is a great summary of real stuff you need to think about on EC2, or any other provider. Again, I know you're not minimizing these issues, but some people certainly do.
i now have a wicked setup, Xenserver Cloud, SAN, all Highly Available, don't have to worry about bandwidth overcharges, and much much faster .
I thought about putting four 512 GB SSDs drives in a Raid 10, which would give me striping performance levels without losing the mirror, but that seemed like a bit too much overkill. These servers have 4 front drive bays, only 2 are filled with the SSD mirror, so we could decide to drop in 2 more drives and rebuild the array if we need even more I/O perf.
Building your own servers is dirt cheap, but there's always the risk of hardware breaking, and you can't exactly scale up very quickly.
But the middle ground, rented dedicated servers, can be a perfect compromise for a lot of companies. Scaling up and adding servers might take a day instead of instantly, and it costs more than owning servers, but you get all the benefits of managed hosting, broken hardware simply isn't something you need to worry about anymore. If a server goes wonky, just scrap it and rent a new one.
Running a porn site or image/video sharing service? You're going to go through a ton of bandwidth and that might end up being your bottleneck way more than performance.
This sums up every comment here. Every web app is a unique snowflake and what works for you (EC2/Dedicated, SSD, RAM bandwidth) is just as unique.
Much more important than building your own server, is fully understanding your requirements, and deploying the appropriate solution. But it is still wicked fun to build a server.
> But not the kind that go in your home. No, that'd be as nuts as the now-discontinued Windows Home Server product.
Its all about how you build them. Obviously a 1U with 40mm fans running fast could make a lot of noise.
Quiet, ECC RAM, and decent speed for most home-related tasks (backup, etc.)
Seriously a great product and value if you're looking for a home server.
I could see putting my own server into a colo if it were like a storage locker - carry/cart your server into the building, mount it in the rack you've rented, plug into the provided power and network connections, and lock the security door.
There are options where you can install servers in cages. (Some industries/companies require very strict physical access control.) This is more for show than because it meaningfully increases your security against the attack "your colocation provider is secretly The Adversary."
It does help protect you against the "another colo customer" attack, whether an actual attack or accident.
That's why I used a RAID mirror, so we'd have to lose two drives in one server before it mattered, e.g. the server is down.
Ideally the RAID array would run degraded (unmirrored) until you can get remote hands or yourself down there to drop in a new drive.
They are front-accessible drives in caddies so changing them is relatively easy, no need to re-rack the server.
That translates to literally the exact same 3 year cost as building the server.
[1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...
I still use our company servers that are co-located but having things at home on the same network where you are developing is very compelling for me.
Its very quiet and cool (latest i7 processor).
But if you are hosting your own production boxes, I usually buy from HP or Dell and know that I rarely need to worry about those machines. Would hardly ever build a box to put into production unless it was something that was rather more specialized.
Provisioning physical machines at the same convenience level as ec2 would be awesome.
I ended up doing what Jeff did. Bought my own server and hosted it at HE for $75/month. It's xeon 5650 with 48GB Ram + 1TB disk for $2k. Assuming the machine will last 3 years, it's a $131/month. That's way cheaper than the closest that softlayer offers (https://www.softlayer.com/Sales/orderServer/41/2087/)
Most of the times, EC2 is really about convenience and not cost.
Ubuntu Orchestra: https://launchpad.net/orchestra
Or build your own around cobbler & puppet/chef, but you're sort of reinventing the wheel at that point. Still, sometimes that's fun.
Sadly I live a 5 hour drive from the nearest co-location facility, so I'm forced to rent a dedicated server.
The problem with "hardware is cheap and programmers are expensive" is that your hardware will fail when you least expect it and have programmers sitting idle. Hardware is cheap, so have someone else assemble it and rack it.
If you don't need it up all of the time, this is a great way to get a lot of performance -- IO and memory in particular.
Jeff is glossing over a lot.
If you're already massively invested in hardware, in terms of people, processes and hardware, then you could argue that cloud architecture is less valuable in the general case, otherwise, it's usually no contest.
I've stopped saying that I've never ever seen any static-related problems and I'm using the silly bracelet connected to something metal these days.
That said I always advise touching something metal on the exterior case before touching anything in a computer, and that's how I have always done it.
The semiconductor industry (i.e., the companies like Intel that made the parts on your circuit boards) spends many millions, if not billions, of $ a year on static-protective mitigations. They're pretty smart folks and also very cost-conscious. They would not be spending all this money if it were pure voodoo.
However, parts mounted on a circuit board are much more resilient than loose chips. I, too, use the "touching metal" method but am very careful about it.
>The instance types included in the Web Application customer sample are 2 small (for the front end), and 1 large (for the database).
Nope, the instance types are 2 small for web, 2 small for app, and 2 large for DB. That's fully double what he's claiming it is. And he's ignoring the 4 300GB EBS volumes that are in that $1400/month as well as the load balancer and 120GB of bandwidth. And that is entirely on-demand instances, if you are comparing to a colo setup, you should be using the much cheaper reserved instances.
You're right that it is x2 for each EC2 server which I didn't notice until later, but that doesn't change the economics or performance story very much.