Going Colo
blog.pinboard.in
blog.pinboard.in
I'm hoping the Hurricane Electric facility he mentions is Fremont 2. The Fremont 1 facility has been plagued with power and packet loss issues over the last year (ask your friends with Linodes in Fremont all about it ;)
It also seems a shame to put both of the deployments so close together. Proximity to your machines is a non-issue with a decent colo provider: his example of driving to Vegas every time a drive fails assumes that the provider would be unable or unwilling to sell him 20 minutes of remote hands. Spend a little extra money up front for out-of-band access to your infrastructure and leverage the provider for the physical stuff!
Edit: "for" -> "from"
Never mind. Who cares.
https://news.ycombinator.com/item?id=2984932
Basically, he was too mean in a comment thread. The submissions on HN typically run media hot (X is doomed, Y is for idiots, Z is braindead) while the comments run media cold. He got the two mixed up, and was banned.
This /sounds/ like a good idea until the reality of trying to get someone random to fix an actual hardware problem over the phone sets in.
It only takes one idiot pulling the wrong drive once.
(to be clear, spend the money on oob management, too. Serial consoles are essential if you want to have a hope of knowing why that server rebooted last night. And you need to be able to monitor your power draw; it really doesn't cost that much more to spring for the PDU that can monitor and switch the outlets.)
The problem is power is even more expensive there than in SF.
Connectivity wise, they are pretty good. Sometimes may be a little more DDoS prone though.
I've been in several Sacramento datacenters and currently have a half rack in in Webion in Rancho Cordova. Ragingwire is freakin expensive.
The pricing with prgmr.com looked pretty good though. SJ gets better bandwidth pricing, and probably about just as crappy on power.
If looking for a serious deployment, generally have to look outside CA. Great connectivity, but power is what will kill you.
I'm trying to move my own stuff in the direction of santa clara.
Of course, at the low end? I mean, if you are the $200/month customer that is thinking about buying from me? that sort of thing probably makes less difference.
The funny thing about he.net is that if you work out the cost per watt, they are more expensive than most places after negotiation. (on the other hand, he.net gives you the price off the top; most other places go for 30-50% over what the real price is.)
It's pretty funny; here in santa clara, people charge about twice (per amp) for 208v as 120v. thus, I end up just getting a whole bunch of 20a 120v circuits. It's silly, but eh, watts are what I actually want.
(208v is slightly more efficient; but an amp of 208v is, well, 208 watts, and two amps of 120v is 240 watts... 208v is not /that/ much more efficient.)
HE just seems at weird economies of scale. Pretty much no cooling, 70-80% of the warehouse empty, and $1/Mbit (they lower now?). If Fremont 2 gets more populated, will it inherit the same power issues? Part of me doesn't want to know.
I've heard good things about SVTIX though... haven't toured or had anything in there, but it seems to have a good rep.
they'd only sell me 1 15a 120v circuit, which is 12a usable; 1.44Kw. Something like $500/month before bandwidth, $347 per Kw.
Compare that to $600/month for 20a 120v at svtix (I'm going to charge you $650, and throw in some bandwidth if you buy from me. If you go direct with the guy that runs the place, maybe you can do better, maybe not, but you'll definitely spend more time negotiating with him than with me.) that's 16a usable, 1.92Kw. $339 or so per Kw retail, including 10Mbps bandwidth.
(I'm not saying my prices are great; I think my prices are a reasonable no haggle deal. compare this to the quote from BAIS in santa clara for 11.5kw.. 6x20a 120v circuits in a single 47u rack... 96a usable, for two grand, that's $174 per Kw. Of course, at that point you are paying extra for higher density servers, though, and the up front price was rather higher.)
The power at Svtix has been solid for me, and the owner of the data center keeps it way colder than it needs to be. Which, I guess, is good marketing? but it costs money; 70-80F isn't going to kill a modern server.
I think that he.net fremont is a fine temperature; 80 degree temps aren't going to hurt your servers. (much above that and you worry) but the power at fremont 1, at least, has been horrible. I have two legacy racks there that I'm trying to get out of, and man, they've gone down three times in the last two years, which is horrible by the standards of such things. Once, the power surge when the power came back toasted one of my servers.
It was miserable. One of those "this is why you should get dedicated servers rather than co-locating your own stuff" experiences. I screwed up the configuration on my PDU, so my stuff didn't come online after the first outage; I was paged and had to leave a party. I fix it, and then the next day (early, if I remember correctly, or maybe it just seemed early) the power went out again, and this time it fried one of my servers, so I had to shuffle in, zombie-like, and swap it out with a spare.
I must need sleep, since just realized I calculated kW/$. :)
With my math corrected though, getting about $338/Kw.
Ping - Power - Pipe : Ask everything you need. If a datacenter is unwilling to share or discuss who they are, what they do and what they offer then walk away. Colocation is a buyer beware environment.
Here's a VERY ROUGH list/braindump of questions that AT THE VERY LEAST you need to know before placing equipment with the colocation provider/datacenter.
Tier I,II,III, IV datacenter? Who owns the datacenter? Investors? What's your ASN number(s)? Carrier neutral facility? How many connectivity providers are present in the data center? Does data center run BGP over these providers? IP Transit providers - Bandwidth Pricing? Overage pricing? Straight $mbps/month? Who manages your network? Ip allotment? Assignment procedure? High density for micro vms(Heroku style)? 200 VMs per host meaning 200IPs per vm. Swip/rwhois updates on assignments? Cross-connects? Cross connect pricing? Charges? Local loop(last mile connectivity) charges? BGP support? Is our traffic on a private/separate v-lan from your other customers?
Infrastructure? -Redundancy in detail -Brands -Networking -Connection -Power. Maintenance programs?
A/B feeds from two separator breakers or breaker panels or better yet two Utility Power Service Provider if they are truly A/B feeds. A/B Diesel-Generators? Additional fuel storage on site? Is a PDU included? If so, how many outlets and is it accessible from the web or telnet? What type of hand off/drops to your rack will you be provided (Ethernet)? Are there raised floors? Are there any fire suppression systems in place? Physical/Video Security? Biometrics? Screening of employees?
Insurance requirements to colocate equipment with you?
On-site visit requirements? 24x7 staffing? Remote Hands - Cost and Abilities? What's included? Shipping/Receiving of hardware?
Sas II 70 or SSAE 16 certification? Audit available for review? -Common practices -Staffing -Security -Emergencies - Escalations -Contacts Outage alerts? Notifications through Twitter or similar offsite status page? RFOs for past outages? What's been your corrective action?
Power density per rack? 80% rule or true metered? Can you rack the equipment for us? Who provides cabling? Types of rails for servers? Billing-95th or other? Rack size? Partitioning available? 4u/10u/Quarter/Half? Cages? Lockable or baker's racks? Shared with others?
Finally, ask for a listing of ALL fees.
That said, the colo should indeed answer all these questions. It's just that most small shops will not be able to make sense of the answers...
If you're a small shop looking to dip a toe into colo then a good summary question to ask is "Who else do you host?". If they have a few big-names to share then they'll probably be good enough for you, too.
Ideally ask for permission to ring up one or two of their reference customers and then actually do that. Colos don't like that very much - but if they outright refuse then you'll know to better look at another one.
But - yes, once you pick up a cabinet, particularly if you are doing any kind of BGP multihoming, pretty much all of those questions need answers (possible exception being their internal procedures - at a certain point you just have to trust them to do their job. )
No, it's fine information, if a bit much. But these questions, for the most part, shouldn't matter until you are big enough for these to be important questions but not for you (presumably a founder).
EDIT: I will expand on this a bit.
First, the real question you need to know, is, who owns the building your servers will be in? Lots of people claiming to have datacenters are actually renting from someone else, or even sub-leasing from somewhere else, putting you 2 places away from the owner. As much as possible, get into a situation where your monthly payments go directly to the true datacenter owner.
Second, what about the network? Rather than engage in a lot of hand-waving, just find out who they are directly connected to.
Third, do they have both UPS and diesel genset backup for your power? If you are able to physically visit the place, have them show you the physical stuff.
Ignore the "Tier" datacenter and "SAS70" crap, Tier X != reliability and SAS70 is a load of BS (CPAs trying to soak up some consulting dollars - yes, an SAS70 report can only be prepared by a CPA firm...)
Sure, it's an audit compliance document that does nothing of itself to ensure compliance. But: somebody asked and looked at this stuff, and there's absolutely no reason you can't do as much (or little) inspection as you want. Having been on a few DC audits myself, it's an educational process. And still ... onsite redundant generators, fuel, fuel provider contracts, backup batteries, etc., still mean little if there's a fault somewhere in the system (the past few major outages I've experience directly or have read about all included all of the above, but something somewhere failed to cut in or out appropriately).
Nobody's got anything on the Krell though.
Auditors are trained in, like, you know, auditing.
Which is rather more than just beancounting, preconceptions notwithstanding.
And again: what the audit entails is interviews, requesting and reviewing records, and the like. This doesn't guarantee that a SAS70 site is doing what it says it's doing. But if there are gross inconsistencies in the statements and documents, they should stand out.
From there, use the SAS70 report as a basis for your own questions. There are some very good summaries of things to as at sites such as ServerFault, WebHostingTalk, O'Reilly, and elsewhere.
If the SAS70 report says that all access is controlled, but you find you're able to casually stroll through the main door ... something's not adding up. Dig deeper.
If you are spending more than, say, two grand a month on co-lo, you probably want different providers for your rack and your bandwidth. Aside from the ease of moving providers to keep costs down, usually the 'house bandwidth' is often 10x market rate. I just got a quote the other day (I only need a little bandwidth, 100Mbps at this new place) and they wanted like $10/Mbps for single homed cogent. $0.50-$1.00/meg is market for single homed Cogent.)
Most of those are reasonable questions, though most salespeople won't know how to answer them. (and many of them, like 'local loop' charges will only make sense to telco guys. In data centers, we call them 'cross connect fees' and they are still important, but much less of a big deal than local loop charges when you are getting a telco line from someone off site)
Note, all of those questions should be asked of the bandwidth provider, not the rack provider. except the cross connect fee - this is what the datacenter/rack provider charges for a connection between you and a transit provider in the building. Sometimes it's one time, but it's usually monthly, and scales directly with how nice the data centre thinks it is. It's $50 one time for copper at he.net, but can be like $300/month+ at a coresite location for fiber.
Note, every bandwidth provider in the building is going to have different answers to all those questions, and even within one provider, the answers will be different based on what class of customer you are or how much you can pay.
They are single-homed to Cogent (at least that's how it appears; I didn't check looking glass), which is fine for a hobby server, but not really good for a startup.
http://www.datacentermap.com/as/47066_upstreams.html (47066 is me) http://www.datacentermap.com/as/18779_upstreams.html (18779 is egihosting)
Now, that's not a whole heck of a lot better from my point of view; but egihosting (they are another co-lo provider you might want to check out) is multihomed... so if you are the sort that thinks outsourcing means you don't have to worry about it, I have outsourced my multihomed bandwidth to egihosting.
The references to Cogent were referring my last attempt to get a second upstream. (the deal appears to have fallen through? we will see. I will have to write about it. It was a long story, in which a mistake I made 7 years ago comes back to bite me in the ass.)
I mean, yeah, the network isn't awesome, but eh, it is reasonable, I think, for the cost. I think that if you have physical hardware, you are better off spending the money on a second location (and for the difference between my level and 'premium' co-location, well, you can host with me, you can duplicate your setup with someone else in my price range, and probably have some money left over.
Two cheap ones are usually more reliable than one good one
(especially as this is way more of a 'market for lemons' than auto buying. It's really hard to get true information about the reliability of various providers.)
Being single homed to a small but competent provider who is multihomed is IMO better than being single homed to a large provider, in that the small provider is a lot more likely to actually help you out when things go wrong.
Cogent obviously would be a nice way to get some cheap extra bandwidth, too. I just fear depeering days :)
(of course, if you are looking for and willing to pay for serious bandwidth, you want to look for three or more upstreams. And in that case, having 3 tier 1 providers is better than having 3 tier 2 providers, as you are less likely to get route overlap, and tier 1 providers, all other things being equal, usually have better latency.)
- You'll most likely run out of power well before you run out of physical rack space.
- You'll (hopefully) spend less time in the data centre than you think, especially if you have good lights out access, remote power bar access etc. If you're locating in a large city you can often get a better deal somewhere in the suburbs rather than downtown with very little downside (at least that seems to be the case here in London).
- Interconnects between racks can be awkward to get set up, and you can't necessarily get the rack next door when you want more space.
- Redundant everything will save you a lot of hassle, but gets quite expensive. In particular redundant power will save you from downtime when the data centre decide they need to take one of the power feeds down for maintenance (it happens), and setting up redundant (external) networking has the same effect. Having a second network for admin access prevents you from being locked out in some failure cases. Hot swappable drives are useful for systems which are awkward to take offline (e.g. database servers), but not massively useful elsewhere if you have good redundancy at the server level.
- Data centre rack floors are about the most unpleasant place to work you can imagine :-)
- If you can get a handle for how competent the data centre are at mitigating DoS attacks it's a useful datapoint. If they aren't on the ball an attack against anyone hosted in the same place will take you offline too. having said that, I'm not entirely sure there's a fool-proof way to determine this ahead of time.
- You can generally borrow a cart with a keyboard and monitor on it to wheel up to your rack, so no need to have those permanently installed.
- Keep a stash of network cables, cable ties etc. at the bottom of your rack - it's annoying to show up only to find you've forgotten that kind of thing.
- You may need to register authorised employees before showing up, so make sure you do that before taking your new intern to help rack a server. Expect them to ask for ID before letting you in.
- Data centres (at least some of them) will take deliveries on your behalf, so there's normally no need to get things shipped to your office then drive them out yourself.
It also seems that we've entered an era where you're a fool if you colo a lean startup. Is there a best practice, book, really good blog post, etc that helps one determine when it makes sense to colo vs just spinning up more Heroku dynos?
I agree.
>They may make more sense in dedicated servers vs. coloing your own hardware, though.
depends on what skills you have already, and then your time to money exchange rate. If you use a lot of compute power, you can save a lot of money with a relatively small time investment, if you already have the skills.
Of course, this does eat the same amount of labor as building for the cloud /and/ building for co-lo, so you still have the time cost.
One project moved from AWS to a managed hosting provider with gear built to our specs mostly due to IO issues... continued to grow and then went back to AWS looking to do an easy DR process. Had some issues just with EC2 being able to keep up with replication at that point.
The one thing I'd like to try with AWS though is their Direct Connect service. Basically colo in a facility near them and get a cross connect to EC2 (think it focuses on VPC). http://aws.amazon.com/directconnect/
IO loads on real hardware, EC2 for all the other stuff.
But yeah. Active/active is a far better solution than what I describe, I mean, modulo the extra cash it costs.
[1]well, really, it's sharing spinning disk... the actual overhead of virtualization is pretty low by this point, but if you have two people accessing the same spinning disk? each gets rather less than half the performance they'd have gotten if they have the whole disk to themselves due to the physics of spinning disk and how much worse random access is than sequential. (two simultaneous sequential read/write streams to the same disk can look a lot like random access, depending on queueing.)
I have something like 30 sites up and running at the moment, some big, some BIG, some small. Adding a new one involves spending seven dollars on a domain name and fiddling around for a few minutes in Remote Desktop, safe in the knowledge that it can live comfortably on that server for its entire life without me needing to figure out what a dyno is.
So yeah, in the end it comes down to flexibility. Or rather how you define flexibility. Some people like to be able to spin instances up and down and squeeze their expenses down to the nearest micropenny. Others like to know that they have plenty of breathing room and no requirement to spin things in any direction at a price that's still on the same order of magnitude as the micropenny crowd.
Also, is his concern about earthquakes really warranted? I'd have thought that a DC would have been able to withstand more or less anything thrown its way.
2. A 20 A circuit breaker is allowed to trip at 20.000001 A. Derating the continuous load provides the safety margin for reliable operation, even using inexpensive circuit breakers.
The breakers in your home breaker box operate the exact same way. A 20A breaker will trip once you exceed a certain threshold, which I believe is 16 or 17. They're rated to go to 20 by the factory, but to prevent problems with overloading and causing fires/explosions/etc, they purposefully break at a lower threshold.
So this isn't just a colo price scam thing, this is the way the world works.
I'm not sure why this came about in the first place, though. AFAIK in Europe breakers are rated at 100%.
I think this is an NFPA/NEC[1] standard.
You could look at it another way: circuits are breakered such that you can temporarily peak 25% above their normal capacity without tripping the breaker. I don't know why the standard isn't more conservative, except to guess that it's because typical household/office uses are likely to have low continuous and high intermittent loads (e.g. vacuum cleaner), making it a good trade-off.
[1] USA (and Canada?) only. Anyone know how it is in other countries?
But kika's downthread comment is most on the mark - of course data centers would rather advertise the less relevant "circuit rating" as the number is higher. The load calculations are the same either way.
(btw speaking of the NEC, http://bulk.resource.org/codes.gov/ is a fantastic resource)
Here are my tips for people that are considering a colo. These are really all about working comfortably within a confined space (a half rack or smaller)
* 10U is not that much. Don't buy space that you aren't going to need but compare prices and find out about options for expansion. Space isn't expensive but you might have trouble buying space that isn't bundled with power.
* Keep your wiring organized. Don't say "I'll fix that later".
* This goes hand in hand with the previous: If everything has redundant power, are you going to run out of outlets before you fill your space? Consider installing a metered PDU (or PDUs) before you add any machines
* You probably don't need 2U servers. The 1Us that I currently buy have 2 CPUs, 10 x 2.5" drives (also handy for SSD) and redundant power: http://www.supermicro.com/products/system/1U/1027/SYS-1027R-...
Also, like justauser said, get a complete price list. Sucks to grow and find out that you are going to get hosed on cross connects or extra power or something like that.
Bandwidth - So, this seriously matters, but you shouldn't spend money on it. The biggest things about bandwidth, and connectivity are: reliability, latency, and capacity (in order of most important to least important)
Reliability: This is things like, packet loss, how often your connectivity is impaired, etc... This will have the largest Ux impact. Most bandwidth providers will have a pretty good story about this. Typically, the delta of reliability between a $15/mo provider and a $150/mo provider doesn't actually provide any realistic differences. Don't waste your money.
Latency: Although, most of the applications we'll be working on won't be sensitive to connectivity latency, it does create an upper bound of the speed of TCP data transfer. (Look at the big fat, long pipe problem). All of us have big fat, long pipes to our houses phones now. You can't do much for this than locate your datacenter well. The other thing i, you can ensure that your transit provider has good peering relationships, and transit. I would recommend looking at the list of peers through a looking glass (bgp.he.net is a particularly good looking one).
Capacity: This is the question of how much bandwidth can I soak up at any given point. You shouldn't ever have to think about this problem. You're probably not going to have to scale your bandwidth faster than what your upstream already has. Nonetheless, if you're running a bandwidth intensive site, you should consider this, because you could potentially choke your upstream. They will more than likely just go ahead and blackhole you in order to preserve the rest of their revenue stream.
The Dell/HP salesmen are worse, if you try to talk about power, especially if you do HPC. They will often flat-out refuse to believe that you're going to take 100 servers, stuff GPUs and all the RAM you can find in them, and run them at 100% for years on end.
But for a frugal startup it's perfect - cheap, good connectivity, awesome support, easy access. Not very reliable, right, but you don't have many users, don't ya? :-)
The entire city sits in a flood plain, and is protected by floodwalls, levies, dikes, and flood gates.
The last major flooding of major structures that I'm aware of was 1956 (mostly affecting areas north and east of Sacramento State), but given changing weather patterns and grave concerns over the delta levee system, nothing is certain.
Most of the major datacenters are clustered along US50 (which follows the American River), with others downtown and in West Sacramento, also within vulnerable zones.
Probabilities of a major flooding event exceed those of a quake. Though of course the Fremont/Hayward datacenters are direction on the Hayward fault.
Nope. I've worked there, they've got multiple petabytes of data, though a major portion of that is audio and video. Even the books aren't stored as txt files.