Google's fiber leeching caper | Hacker News Reader

Google's fiber leeching caper

dodgycoder.net

167 pointsdamian2000100 comments

Everybody who pays @95th dreams of doing something like this the very first time that 95th billing is explained to them. I mean Everbody.

What the Article fails to explain, is that there is both a port cost, and a minimum commit. For much less than what it would have cost them for the port, and minimum commit on a 1 Gig connection, they could have simply mailed their data via USPS and had it there in three days.

Also, if we do the math - and presume that they only ran 1 gigabit for 5% of the time - 1 gigabit/second at 36 hours (5% of a month) in terabytes = 16 Terabytes - which barely makes it into the "10s of terabytes" category, and well within the "Just mail an AIT-3 tape. [Edit - cheaper yet, just load up the hard drives that you are going to install anyways in your destination]

Edit 2:

Doh - http://www.mkomo.com/cost-per-gigabyte shows that the largest hard drives back then were 20 Gigabytes. We're talking a minimum of 1000 hard drives for "10s" of Terabytes.

Maybe that 1 GigCircuit Hack wasn't such a bad idea after all.

In the year 2000 storage cost about $12/GB [1]. This article claims 10s of TB, so let's say 20 TB, which is the most conservative possible estimate. 20 TB = 20000 GB. At $12/GB, that comes out to $240,000. That doesn't include retransmission cost (for busted drives) or shipping. I'm not sure the analysis is correct here.

Presumably their minimum commit was a lot lower than $240k, because otherwise they wouldn't have shrunk from just using the connection. They probably saved a lot of money by doing it their way.

[1] http://www.mkomo.com/cost-per-gigabyte

* I wasn't around then so I don't actually know.

* * It also depends what part of the year they needed to do the transfer. By [1], costs had dropped to $8/GB by the end of the year.

"storage cost about $12/GB "

Hard drives cost $12/GB. Tapes/optical media maybe $1/GB. Maybe I'm missing something but it seems like they would be fine for this.

Enterprise tape drives ain't cheap iirc. But anyway, I was responding specifically to the HDD idea.

Right - but they had to buy those hard drives anyways, so it didn't cost them anything more than the transportation cost of the drives to buy them at the source, load them up, and then ship them.

It would be hell to get the data off before imaging the drives. What do you transfer it to? The shuffling would require a lot of technical work, which also costs money. Also, we don't know if they were on homogenous hardware by then -- they may have still been using budget pcs and refurb'd parts. So it's hard to know if your plan could work. The investment of inventing a process for mass data transfer versus using the normal process but only during a certain time period is also significantly higher.

I have to think someone ran the numbers. It's not like the folks in 2000-google were dummies.

Admittedly, it would have been a lot of drives. Back in circa 2000, according to http://www.mkomo.com/cost-per-gigabyte, 20 Gigabyte Drives were in vogue @$200/pop. So, a Terabyte would have been 50 drives, 10 Terabytes @500 Drives.

10s of terabytes at a minimum 1000 drives. I'm beginning to question how "simple" an operation this might have been after all....

> It would be hell to get the data off before imaging the drives. What do you transfer it to?

I'd guess you'd transfer it to the same thing that was accepting the data when it was coming over the 1Gbps link?

The parameters of that scenario were that you'd ship the data on the disks that were eventually to store it (presumably without duplicating the OS on each during the transfer process). Therefore, there'd be nothing to move the data to.

The difference is that the hard drives themselves would be assets (and would probably have nostalgic value in 2013 if they wouldn't have decided to resell after the transfer)

You'd have to buy the drives anyway.

Even if you didn't consider cross-country-shipped drives to be data-center reliable (e.g.: the transfer media weren't re-used in the datacenter), you could simply cycle through drives. If your goal was 1 month to transfer 20 GB, you utilized overnight shipping, and required a day to load/unload data from drives at either end, that's 200 drives in flight, 50 of which are outbound, 50 returning, 50 writing, and 50 reading, on any given day.

You even get to take weekends off.

It's $48,000 in media. Plus shipping (not inconsiderable). You'd save on the cost of dedicated data transfer equipment (datacenter, router, links), though you'd need infrastructure for reading/writing data, and of course, storage at the remote side if you're not simply shipping disks for installation (which, frankly, is probably your best bet).

Resync should be fairly reasonably accomplished over the wire.

You can vary the math by taking longer (fewer drives, less cost) or shorter (more drives, higher cost), or even by utilizing ground shipping (7 days) at far lower cost.

What about the physical size of 1 GB in year 2000? How many HD do they need? and how much does it cost them to transport those drivers?

Your 16.2TB estimate is for one month. The article says this continued for multiple months.

You'd trust your hard drives with USPS?

Edit: I'm not implying that USPS is necessarily unreliable, I'm curious to hear if this is common practice.

Presumably I'm not sending the originals. Also, powered down, hard drives are pretty robust - their read/write heads are locked up. Remember - those drives have been bouncing around a LOT to get to you through retail/wholesale channels.

[edit] Also - they obviously had to buy hard drives for the destination, right? Presumably, early stage google was all about the open-chassis on plywood slam drives in approach, and wouldn't have been averse to loading up 30 or 40 drives, mailing them out, and simply diffing and resyncing as needed if 1 or 2 drives had issues, which would have been unlikely.

Also, presuming you needed to have one of your people go to the remote data center, there is always the old, move the datacenter with hard drives in my luggage trick. Even in 2000 you could put a lot of data into carry-on.

You've always got to expect a certain amount of packet loss over any connection.

RAID = Redundant Array of Independent Deliveries

You could make up any small number of botched drives with over-the-wire transfers. If it were a sufficient number of drives, you could simply re-image and re-transmit the data.

When they're adequately backed up, as in this circumstance, why not? Send the data in triplicate if you want, just to avoid the hassle of re-sending any that get damaged. Regardless, the cost would be trivial, even for young-Google.

I can assure you that there were drives larger than 20G in 2000. I had a server with 4 23G Seagate Elites at that time, and the Elite was an _old_ 5 1/4 FH SCSI drive. 18G Seagate Barracudas were certainly alive and kicking since 1997 at minimum (the year I started building real production servers).

Reminds me of a loophole Microsoft found in their utility contract:

http://www.nytimes.com/2012/09/24/technology/data-centers-in...

Threatened to waste $70,000 of electricity to circumvent a $210,000 contract penalty (for not buying enough electricity), and thus negotiated their way out of it.

(Note this is MSFT not GOOG, so be sure to interpret this as an act of disreputable evil and not a clever funny hack).

I think what's important here is that it's funny.

Google exploiting good faith rules to shirk payment? Funny.

Microsoft blowing $70K in power to meet a contract? Also Funny.

It is kind of stupid that the penalties were MORE than the actual electricity. Shouldn't it have just been the cost of the electricity that they failed to use?

That way loopholes like this wouldn't have existed.

No, because disposing of the unused electricity has costs for the provider.

The way this works is that the provider offers a cheaper rate to ensure constant energy output, since the power plants can't be easily and cheaply scaled back.

As an example, Ontario paid millions[1] to get others to use their surplus energy; MS' provider could have been forced to do the same, hence the penalty.

[1]: http://www.thestar.com/business/2011/08/29/ontario_pays_othe...

It's not just about using the excess generation capacity. In fact, at this scale it probably has nothing to do with generation. For distribution utilities, they need to install new infrastructure to connect new large loads (new datacenter, new manufacturing plant, etc.) - substation upgrades, new conductor, etc. Because that infrastructure only serves that one customer, the cost is recovered from them through a minimum-connected-load + capital contribution contract rather than from the entire customer base through a rate increase.

Keep in mind that utilities are heavily regulated and can't typically touch their rates at all without regulatory approval. In many jurisdictions, there is an "obligation to connect" for residences - the utility /must/ provide power, at a certain rate, to any residence in their area of operation. They aren't allowed to directly recover the cost of infrastructure upgrades (e.g. new distribution transformer when somebody builds an expansion to their house) directly from that customer (up to a point - if you're building a mansion that uses 10x as much power as everyone else, they'll come after you), only from the ratebase as a whole. Otherwise, people who live in rural areas wouldn't be able to afford electricity.

No such obligation exists for businesses which are typically the largest loads anyhow. If you build a new 50 MW datacenter on my distribution system, now all of a sudden I need to put in 5km x 3 phases x 5 feeders of new conductor, 2 new station transformers, plus civil infrastructure, etc. It could even be special equipment I wouldn't otherwise have on my system - like DC equipment. All of that is additional to what I have to pay the generator for that 50 MW of power on a continuous basis. So, I calculate the payback period on 50 MW of power I'm going to sell you at a 5% margin, and if it's longer than the lifespan of the equipment I'm going to buy I make you pay up front to bring it down to say, 25 years. But that only works if you actually buy 50 MW of power from me for 25 years, so if you don't I charge you a penalty equivalent to the amortized cost of that infrastructure that now sits idle.

Hope this helps explain the economics of it from a distribution point of view. I am a power engineer.

I'm curious. So how much power (energy) can a typical residential user use? I'm thinking average lines and transformers.

Here in Toronto the current standard for residential service is 200 A @ 120/240V (24 kW). Older homes might be 100 A. That would be the max that the conductor between the distribution transformer and the home's panel can handle without thermal failure. But you wouldn't size distribution transforms based solely on peak load because a mineral-oil immersed transformer has a lot more thermal mass than a cable and takes hours/days of sustained load to actually heat up. Average loads used for planning and distribution transformer sizing are more like 5-10kW/house.

Thanks! I actually own a 2-family house that is very old and I have two 60-AMP panels. If I renovate, it will need to be upgraded.

It seems...unlikely that they actually needed to pay people to take their power (though I'm no EE; it may well be possible). My intuitive sense from that article was that it was a contract issue (and possibly not even that; they also throw in something about "below costs", which would obviously not be negative price).

Electrical generation and distribution is a tough business. Places where you have generation capacity aren't necessarily accessible via the transmission network. So if NYC isn't using power, you can't just send it to Cleveland. Sometimes, you have a certain amount of "fixed" generation capacity that you must use, or must commit to taking offline for days before restarting. 80% of your power may come from a few big, centralized, cheap power plants (nukes, big hydro, coal) and a bunch of geographically dispersed, small, more expensive plants that you spin up to meet peak demands. (natural gas, small hydro)

Utilities sign contracts with big industrial users who have steady-state energy demands to ensure that they have a place to send lots of power to. They also have contracts that when peak demand is reached, the industrial customers shut down lines in exchange for discounts. (I worked at a place that financed a DR datacenter largely by committing to failing over 50% of our energy workload to another region during a peak demand emergency.)

I'm not doing the topic justice, but there's alot of behind the scenes stuff there -- I'm sure it wasn't just a contract dispute.

From the article it doesn't sound like there was any sort of dispute...my guess would be that the contract formalized payment as a function of capacity and demand, in such a way that payment could be negative. This might not even be a mistake or from a need to distribute the power, it could just be a low-cost (due to rarity) sweetener.

Depends on the scale.

From the article:

> At the end of the month, the top 5% of usage information was discarded, to eliminate spikes, which were assumed to be measurement errors.

Not actually true. The bandwidth measurements don't have accuracy problems; providers use 95th percentile billing to make connections "burstable", allowing servers to handle an small unexpected traffic spike without a huge bill. And on the flip side, many sites have a traffic pattern of highs during a subset of the day (customers' waking hours or business hours or non-business hours, depending on the site) and lows outside that period (sleeping, etc), for which 95th percentile billing produces a higher bill than just charging for bytes transferred.

https://en.wikipedia.org/wiki/Burstable_billing

Very few customers have the kind of traffic pattern that would allow for using bandwidth during only 5% of the month. And in this particular case, Google could just as easily have saturated a 50Mbps line rather than using 5% of a gigabit line, which would have massively reduced (though not completely eliminated) the bill.

Thanks, didn't know that, I will update the post to mention Burstable billing.

Even for a scrappy startup, I consider this to be exceedingly unethical. Somebody paid for that bandwidth, somebody who essentially invested in Google with no opportunity to see a return on the investment.

I can't picture the world where this would not be considered "evil".

I think it's funny how you start about ethics and then immediately follow by talking about money.

Whether it was unethical depends on whether anyone suffered negative consequences and whether Google knew, or could foresee that. For all we know the bandwidth was available, couldn't be sold to anyone else for lack of customers and would simply have gone unutilized if Google hadn't used it.

> I think it's funny how you start about ethics and then immediately follow by talking about money.

Why would that be funny? A lot of ethics is about not destroying someone else's value. Value is measured in money.

Money is a measurement of value, like popularity is a measurement of skill. They're both poor measurements, but very easy to apply, they cater to everyone's understanding and there's undeniably a correlation. Therefor they get used a lot and people lose sight of how poor they are in many situations. Numbers also allow us to decouple from the ethics of the situation.

Take healthcare. You would not put a price on your own life: it is priceless to you. Nevertheless we put prices on other people's lives, because it is the easy way out, is accepted by almost everyone and allows us to avoid having to answer nasty questions. No one wants to honestly say someone unknown is worth nothing to them to keep alive. So you say you will only support health care up to $10K a year, because you know the chance a loved one will ever need more is extremely small. You talk abstract numbers to prevent from having to admit to yourself that you just don't give a damn. That wouldn't fit in with our ethics, would it?

> Money is a measurement of value, like popularity is a measurement of skill.

Extremely good analogy. Thanks.

A lot of people equate money with value directly. Only half true...

People talk numbers because money/supplies are finite. We can't spend 200 million trying to save one person, as that will simply create a lack of resources for every other person who needs care...

(Derail) Why not?

In the US, we've got ten million unemployed and tens of millions underemployed even when we're not in the middle of the Great Recession. With the proper application of economics, we could easily train tens of millions of more healthcare professionals (note: it doesn't matter if the unemployed are qualified; you can train the qualified-but-otherwise-employed and let everyone else shuffle to fill in the gaps. Hermit crab principle.) and end up with a large enough surplus to treat everyone as to their need.

As employment strategies go, it beats paying people to dig holes and then fill them in.

The cost of healthcare doesn't have much to do with nurses, it is drugs and machines and surgeons.

The rules were set by the supplier and they played within the rules. Don't play the game if you don't like the outcome.

Ethics aren't about the "rules".

You will always have bad people. The more people that exploit the system, the faster the system will change to be more 'fair', otherwise you're just handicapping yourself. Look at all the patent lawsuits. As they said in my business ethics class: ethics schmethics.

(1) It's not clear that acting unethically will lead to better rules and more ethical outcomes, in general [and actually, in some cases the opposite will happen - if you're corrupt it makes it easier for others to be corrupt]

(2) Handicapping yourself is the whole idea behind ethics and morality. Sometimes it's more important to do the right thing than the selfish thing.

Attitudes like yours are so disheartening to read. I totally disagree with your sentiment.

Moral != Ethical. I'm guessing you think Foxconn replacing workers with robots is bad too?

How is moral different from ethical? In my mind, they are pretty synonymous.

And no, I don't think Foxconn is bad for replacing workers with robots. Why would I?

Strictly obeying the rules != good faith

You're completely right, but many of us realized "don't be evil" was nonsense long ago. That ones who need to hear your message would defend Google as not being evil no matter what they did (as you're already seeing in response to your post).

This just seems like you patting yourself on the back for 'realizing the Don't Be Evil was BS.' A lot of people on this forum would see this as a clever hack regardless of who did it. The fact that you bring up the whole "Don't Be Evil" bit and then immediately reference yourself is telling.

There's a lot of projecting in your post. I'm not patting myself on the back for seeing the obvious. I'm frustrated that other, otherwise very intelligent people, are so blind to it.

This "95th percentile" measuring method is still widely used by bandwidth providers, colocation facilities, etc. I am not sure why the article says it was a common billing practice "of the time". More info: http://en.wikipedia.org/wiki/Burstable_billing

I'd be curious who still uses this. None of my contracts are based upon the 95th percentile. We also don't bill using this method either.

I've never met an ISP that sells Racks in a colo that didn't bill transit at 95th percentile (if you chose to use theirs). Are you hosting bare metal, or is your colo providing the physical server?

If you deal with a Datacenter that is just providing you with a Cage + Power + Cooling + Physical Security, you are almost always going to pay for transit @95th percentile.

It's quite common if you ask for a quote for hosting in the UK.

If you ask for a single server you generally get a monthly allowance. For example, one of our servers sits gets 5TB a month on a 100Mbit port. Saturating that 100Mbit port would give me 30TB (and a large overuse bill!).

But if you ask for colo rackspace (1/4,1/2 or full 42U cabinets) then you generally get to choose what drops you get (and what speed), what connectivity and what the 95th percentile billing rate is.

Here's a random example (i've no connection with any person or company in this thread):-

http://www.webhostchat.co.uk/request-products-services/26778...

All the colos I've looked at use the 95th percentile. Our bandwidth is 100Mbps but our capacity is 10Mbps. This would be about 26Tb/month. Our contracts are also in the UK so it might be a regional thing.

Single servers tend to use the total usage model. If you're buying space by the rack, you're more likely to use 95th percentile. CDNs often offer both methods and let you sort out which is better.

Thanks, I've updated the post with this wikipedia link.

I'm not sure if I'm supposed to be impressed at the ingenuity, or disgusted at the dishonesty.

Early-Google really fucked moron salespeople at most of their vendors (well, technically the salespeople also got paid, so the only morons were the investors). People sold space with power/cooling included at ratios which were unsustainable, and Google's people were more than willing to sign contracts with the facilities at those prices. Then Google "fully utilized" what they had in the contract, which really screwed their colo providers.

OTOH, this was in the colo/bandwidth/fiber nuclear winter of 2001-2004, so maybe some vendors were happy to have some revenue, but there were cases where the Google stuff was at negative overall margin on variable costs (i.e. the vendor would have been better off shutting down the power plant vs. selling to Google).

Using what you paid for is not unethical. Selling what you can't provide is, so if the colo can't keep up it's not Google's fault.

It's like renting a hotel room and then running the water/power/etc. 24x7 (and via cable to outdoors to power a factory for your 24h stay).

Sure... If hotels advertised unlimited water and power. I similarly don't feel bad for a restaurant that loses money on an all you can eat buffet.

There is "unlimited" and there is "reasonable and customary so let's not waste energy metering it".

Heh. Everyone did that at the time, because the colos didn't know better. I had a small 300ish installation in a tier 1 provider in the Northeast, and at one point in 2004 they had to tell me they couldn't provide the power for my new racks because they were literally out of circuits. Violation of contract, but they had no choice.

> People sold space with power/cooling included at ratios which were unsustainable

Do we have an ethical duty to make sure our counter-parties are always making sustainable offers?

"Unconscionable" contracts are legally void, so to some extent, yes.

Depends. What's your view on Steve Jobs and Wozniak's phone phreaking?

What, exactly, is dishonest about it?

That is a pretty awesome hack, but I wonder why they didn't just use hard drives and ship it instead of wasting months. Cost / benefit seems off there.

I'm not sure this is a hack as much as abusing a supplier providing a service.I'd like to believe hacks are pie-enlarging rather than zero-sum.

As mentioned above around that time cost of data on hard drives was $20/gb so that would come out to around $240k at the time. It's crazy how things have changed and we can buy 16gb flash drives now for $5-10. I remember when the first flash drives came out and it was like 128mb for $20.

More like 4MB for $20. I have some in my desk.

I find this a little hard to believe; it has the whiff of an urban legend. Besides the fact of minimum monthlies and the probability that the bandwidth provider is not so stupid as not to notice that they're moving lots of terabytes30 hours a month and nothing at all the rest of the time (especially over a period of months), where's the time saving? you could just do all the copying in the data center, repack the drives and fly them to the east coast. This seems a lot more efficient than trying to do it in chunks of <72 minutes/day.

That's insanely brilliant if you ask me. And on a site called Hacker News, mos people should agree!

A fair number of bandwidth providers charge little or nothing for inbound datacenter traffic (it's not what they're constrained on), which makes services for which inbound traffic is high, like web crawling, very cost-effective. Google likely benefitted from this as well, though they're hardly the only ones.

I completely believe this story, because I've done similar stuff. In 1998 I was working for a company that was using a lot more incoming bandwidth than outgoing bandwidth, which is the reverse of the usual pattern. So when we outgrew our data shack and had to move to a real data center, I badgered my sales guy into showing me the overall bandwidth usage charts for the proposed location.

Unsurprisingly, there was way more data flowing out of the facility than there was flowing in, so I talked him into a substantial discount on inbound traffic. "Hey, nothing else is using that half of the duplex..."

That's optimizing for mutual benefit. not the same as stuffing.

I've just made some corrections to the post. I was going off memory when I originally wrote it. After re-reading the relevant passage from the book again I've corrected the number of terabytes that Google needed to transmit - it was more like 9 TB. And they transmitted the whole lot over two nights in one month, not three nights. Sorry for the confusion.

It was on page 187-188 of Steven Levy's book "In the Plex" and the two people he's quoting are Urs Hölzle and Jim Reese, who were both involved in Google's infrastructure.

They had a lot of flexibility around when they used the bandwidth. I don't understand why they couldn't just negotiate a deal whereby they would only send data during times of low demand.

I call BS. I'm supposed to believe that Google took months to do something that would take Fedex 2 days. I think Google is smarter than this.

If, by "10s" of terabytes, they mean at least 20 terabytes, then, in 2000, a "Large" hard drive was 20 Gigabytes. That comes to 1,000 Hard drives. If they had some sneaky way of getting onto a GigE connection, then perhaps this might have made sense...

They still had to purchase HDs at the other end. It's not like data just lives in ether. At some point it's got to be written to someone's spinning rust.

How is this any cheaper than copying the HDs, shipping to the Northeast and installing the HDs into waiting servers there?

The logistical challenge of swapping 1000+ drives into/out of a small number chassis, keeping track of what data you've copied, would have been painful (but not impossible if stretched over a week)

I think the key issues here would be was moving the data at all time sensitive (is a two month latency not a problem), and, did they really have a "free" 1 GigPort they could sneak under the 5% window with? If it really was free (or very, very cheap) - then maybe it would have been worth it.

Depends on their architecture. Maybe the data and the system image lived on the same drive. Maybe they were just provisioning cheap / refurbished parts.

Maybe I'm overly skeptical in my old age, but I have a difficult time believing this.

Surely there'd some sort of rental cost for the line in addition to the bandwidth cost (or perhaps a minimum spend commitment)? Wouldn't it have been cheaper, simpler and possibly quicker just to get a smaller line and saturate that?

Has anyone read the book? Does the author provide a credible source?

Smells like urban legend.

I was surprised to read this in the book, and even more surprised that it seemed no one had discussed it online yet either, as far as I could tell. Its on page 187 of the book. The author, Steven Levy's source for this story is Urs Hölzle who was the first VP of Engineering at Google. His title now is Senior Vice President for Technical Infrastructure.

This kind of traffic optimization happens more frequently than you might think. Particularly among CDNs.

CDNs don't optimize solely for performance. They optimize for a balance of performance and cost. That means saturating a given 95/5 link for 5% of the month, and then mapping the traffic away right before it starts to get expensive.

I don't understand the "didn't use the connection at all outside that time, they should be able to get some free bandwidth." How or why would Google shut down a data center for that long just to pump some data across the lines? Wouldn't that mean down time? I don't get it.

The purpose of this data transfer was internal - transferring search index data between west and east coast - and they had a supplier contract just for this purpose.

Thank makes sense. Silly me.

They probably had a separate line coming into a router specifically for that purpose and directed all of the traffic to transfer the data through this router.

It's common for internet providers who cap to run the 'capping' process in batches, so it's possible to get 'free' bandwidth by exploiting the time period (often up to 24 hours) between reaching your limit and when your provider caps you.

So is it that the index didn't change that much in the months the transfer was happening? or the changes were cleverly isolated so they could be copied over last and the switch flipped immediately after?

"a good deed never goes unpunished" (95% billing to prevent bursting megabills)

"don't be evil" (except for 3000 exceptions)

why not save it on an SSD and fedex it over?

The first 3 words of the (short) article are "Back in 2000".

You mean other than the fact that SSDs didn't exist back when Google worried about $250k?

At the time, Google was not hugely profitable like today, and were very conscious of costs.

Meanwhile their hosting company was making a trillion in profit /s.

Google plays fast and loose with ethics and definitions, from (shall we call it) penalizing competitors and then releasing a clone to working with Visa and MasterCard ala Wikileaks to cut off the money http://www.gizmodo.co.uk/2013/02/google-wants-to-stave-pirac... to sites with "piracy sites." Their PR hasn't worked on me for a few years and there's even a boomerang thanks to it.

Reply on news.ycombinator.com