Want an energy efficient datacenter? Build it underwater
spectrum.ieee.org
spectrum.ieee.org
a) Salt water is highly corrosive. Wouldn't maintenance costs be high?
b) Isn't marine biology highly sensitive to heat pollution?
For high latency services like Amazon Glacier, wouldn't it make sense to host in a place like Iceland? Really cheap hydrothermic/clean power. Highly educated local talent pool, and relatively consistent cool temperatures. If you're maintaining 80F (http://www.datacenterknowledge.com/archives/2008/10/14/googl...) the ambient temps outside should provide ample cooling year round.
For lower latency requirements, wouldn't it be worth it to install efficient cooling powered by electricity? (preferably renewable) With cheap solar who really cares about grid power loss and associated inefficiencies?
Essentially, I think it makes sense to get better at solar than to have Steve Zissou as a server admin.
To metals. Not so much to plastics.
> Isn't marine biology highly sensitive to heat pollution?
Isn't land-based biology highly sensitive to heat pollution?
No, land-based organisms comparatively have a much, much higher tolerance to temperature swings.
Water's high heat capacity and conductivity means that most marine environments have extremely stable temperature ranges (a few degrees one way or other), so most marine species are adapted to live in only that range, and quickly die otherwise. Tropical fish are hard to keep alive precisely for this reason.
Air temperature varies much more widely depending on time of day, season, and weather, so terrestrial organisms have all developed extensive strategies for thermoregulation (e.g. mammals, birds). For example, dogs can survive in both arctic and tropical environments.
Plus, while heat is naturally dissipated upward, underwater thermal plumes are more likely to affect fish than above-ground plumes would affect birds.
FYI, Cornell uses lake water to cool several campus buildings, and they needed to carefully review that the warmer return water wouldn't negatively affect fish [1].
[1] https://energyandsustainability.fs.cornell.edu/util/cooling/...
So how much will 2.4 kWh change the temperature of a cubic meter of water at the ocean surface?
Density of water = 1000kg/1m3 so mass = 1000 kg
2.4 kWh = 8640 kJ
Heat_Energy = Conductance * Mass * change_in_temp
8640kJ = 4.186 kJ/(kg⋅K) * 1000kg * xC
x = 2C
So solar heat will change the first meter of water by about 2C over the course of a day. That's a small number b/c solar radiation is not a huge contributer to temperature swings. Instead what will effect the temperature difference is the seasonal change. If we can calculate the heat energy gained or lost due to seasonal extremes and water temp, then we can figure out what the range of acceptable heat energy is for surface water.
[1] https://www.withouthotair.com/c6/page_38.shtml
ETA: logic.
I'm guessing you've never experienced the joys of owning a 30 to 40 ft sized, completely non metal (fiberglass hull) sailboat kept in salt water, with all of the great fun of paying to haul it and have the marine growth scaped off, pressure washed, double or triple coat bottom paint re-applied, etc. Just because it's not metal doesn't mean it doesn't attract the category of what the maritime industry lumps together as "marine fouling".
"They are made from a metal alloy with a more "active" voltage (more negative reduction potential / more positive electrochemical potential) than the metal of the structure. The difference in potential between the two metals means that the galvanic anode corrodes, so that the anode material is consumed in preference to the structure." https://en.wikipedia.org/wiki/Galvanic_anode
The trouble was that they're pretty expensive, and zinc has a great resale value - so we had to do extra watches around the lower decks with rigged firehoses to try and stop local divers stealing them who knew we'd just come out of drydock. (This was in the Philippines). We lost several that way.
Wikipedia ["Galvanic anode"]: The arrangement of the anodes is then planned so as to provide an even distribution of current over the whole structure. For example, if a particular design shows that a pipeline 10 kilometres (6.2 mi) long needs 10 anodes, then approximately one anode per kilometere would be more effective than putting all 10 anodes at one end or in the centre.
Impact on the marine environment is briefly mentioned in the last couple of paragraphs of the article, although they're a little dismissive and don't provide any evidence supporting their claim of negligible impact.
A bit more concerning is the notion of relocating these data centre pods. Long-term "reefs" that are relocated around the world could perhaps be a new and different vector for transmitting invasive organisms. I'm not an expert, but I'd want to hear a few of them opine as to whether this creates any risks that ordinary shipping does not.
A server sub would put 200W * (say) 1000 servers, or 0.0002 GW.
What are the requirements for cargo ships? I suspect they run into this regularly when in a dock for extended periods or undergoing maintenance that doesn't require a dry-dock.
The heated water from a power plant can actually provide a refuge for manatees and other wildlife that prefer warm water during cold winter. They're actually known as good fishing spots in Florida!
Nuclear plants also use cooling towers to lower the temperature before it dumped back into a lake/river/ocean. You can do a Google search, but those towers are hollow on the inside. Water runs down the inside walls and convection currents move up.
That may change, but I doubt there are less server farms in 2020 than now.
As a side note, isn't that also Google's plan - the server farm as a single entity, where nothing is really fixed, just turned off and eventually replaced? In such a world, where a server farm exists then dies, underwater seems a great place for them. No regulations, no repairs, locally powered via means that no one can complain about (NIMBYism is a powerful force) just run until decommissioned.
> The water just meters downstream of a Natick vessel would get a few thousandths of a degree warmer at most.
Sounds negligible to me, but I'm not a marine biologist. Would be interested to hear a specialist weigh in.
A single coal power plant is negligible to the environment too. Half the world powered by coal isn't.
Some of them apparently are at 500°F [1].
The issue is the amount of heat relative to the environment in which flora and fauna have evolved to survive, not relative to absolute zero.
It's like saying that a 313 Kelvin body temperature shouldn't bother me because I do fine with 310K (~37C). It's only a 3K difference, after all.
I'm not sure what that means (or if it's really true). The temperature at night doesn't drop to almost (absolute) zero; the oceans don't freeze; so plenty of the Sun's energy must still be present.
The second statement was that "OTOH, that 1GW drops to almost zero at night." It may be a literally true statement on its own, but it's misleading in response to a statement that absolute zero isn't a realistic basis for this discussion.
The ocean is really, really big.
1) In the original article, these server farms would be remote administration only.
2) Such facilities would make the secret construction of one's James Bond Villain underwater base much, much easier. (Wouldn't Elon Musk, Jeff Bezos, and Peter Thiel all make great Bond villains?)
Villains are sometimes just victims of their circumstances and trying to survive.
Elon and co are the secret villains.
Julius Caesar is the quintessential example of someone who becomes something just out of trying to survive. He was a hero to many and a villain to many as well. I wish more villains were written that way.
You know if Elon Musk tried to take over the world as a benevolent dictator I'm not entirely sure I'd fight against it...
Then why don't you "get better at solar" and become rich in the process? What did you say, it's too hard, practically impossible? Yeah, life's a bitch, isn't it?
But who is in charge of the seabed around the United States? I imagine that I can't just drop a big metal container on the ocean floor without informing someone or asking permission, otherwise there would be a lot of things just dumped out there.
I also wonder about jurisdiction. If the data center is outside of the Territorial Zone (12 nautical miles from shore) but is inside of the 200nm exclusive economic zone (EEZ), which country would jurisdiction fall under? The country that owns the EEZ? Would the data center fly a flag of its owning country? Perhaps the country where the cabling comes ashore?
Some interesting legal questions. Perhaps the datacenter would be treated like a submarine in terms of jurisdiction. It could allow Microsoft to have their Irish branch open data centers 13 nautical miles offshore of the West Coast to provide storage that's harder for the US to legally get into.
There are.
I imagine this plan means to situate in littoral waters, not treaty open seas.
But what hazard could chipping containers packed with servers be?
I guess there is rare element pollution all too plenty, oozing from recycles in countries where had currency at all, means only barely just hard, not harsh life. But relativism appeases not real concerns.
The idea i think has been mentioned a few times by others, if near/offline storage makes any sense, but to some imaginable degree, sea storage of data also offers unusual security against all but highly determined assailants.
Is there any thermal maintenance factor, that means e.g. disk drives last longer with less data loss if kept at stable temperatures? Would that be effective to do at sea - i think this already mentioned too, the narrow bands of temperature required for marine tropical fish, suggests to me that the cost of mitigating heat waves and cold seasons s sure not applicable.
Can wave power be harnessed to circulate enough air?
Can a inexpensive efficient massive area of the structure be made to be a heatsink?
(Marine fouling as mentioned, seems highly challenging, even if access to surface by buoyancy charges is easily achieved.
Can something organic, even living like a coral, be coaxed into providing natural dissipation of any heat excess?
Oh, i'm saddened now, thinking about barren, sterile coasts where coral reefs have died. But would such areas be better to offer minimal marine fouling by lack of support for life in those God forsaken waters of shame?
Are any challenges less, say, in the Great Lakes?
Or installed inline in channels parallel to canals, utilizing water flow controlled often by lock systems, adding emergency cooling capacity? The trenches might easily be dug. In London there are fiber rights of way along many canals as well. My company long looked at using arch space under rail bridges and viaducts, for they are naturally very cool places, and next to fiber and power.*
Is there a military application for designing the containers?
For forward deployment?
Oceanic monitoring of SONAR?
Or required replacement civilian facilities you need to park aways from looting?
Sounds to me like there is utility in having a reference design, and a potential number of platform customers who habitually fund conceptual capabilities.
*(If anyone is seriously interested even for long shot / entertainment value, in containerized colo under London railroad arches, I'd love to hear from you, i'll freely donate some real energy into getting further than we did before recognizing we couldn't sell it to investors as a solo undertaking. (not meaning selling to investors for the idea, but to those in us, who would balk at cost of diversion of time, if undertaken without sharing interests. That said, it was appreciated the excellent security possible, as well as a side idea about possibility to get planning permission for erecting sky poles for microwave parallel to known HTF line of sight paths, as potential backup capacity and even service to colocated compute. Variously quite a number of projects we researched as thoroughly as possible before spending real money progressing.k, i.e. time, thought, desk research, scouting, but nothing like hiring planning lawyers to approach Network Rail et.al., just modestly intelligent motivated daydreaming, as line of sight to a lot of major landmarks seemed very plausible at 1km and under. Absolutely please do drop me a line if it's even just a daydream you agree has a angle to work, email in profile...)
One thing that does stand out is the sheer greyness of it all - international legal matters like this move very slowly and moreover the ICJ seems very poorly equipped to answer questions like this.
For the cooling aspect, they're already using pumped water and heat exchangers. What is the benefit of submersion vs just pumping sea water from a pipe to a land, or barge-based datacenter and using that to cool?
50msw = 74psi
200msw = 293psi
Many of the challenges could be reduced or mitigated if the pods could be at least partially pressurized to reduce the pressure differential between the inside and out -- but that introduces another complexity -- can the server hardware handle it.
There are quite a few engineering challenges, none of them unsolvable but also none of them seem necessary when it seems the same benefits can be obtained for less effort by keeping the pods above water.
The marginal, incremental value of adding a single server once capacity is reached is currently quite high. Using bogus numbers, lets say a location can hold 100,000 servers, then the marginal cost to add the 100,001st server will be very high, and might cost millions. These costs include:
- Cost of planning - when? Which countries? Which locations? Do you serve Kenya out of say South Africa? Or do you need to go to the mountains of Kenya where the air is a more consistent temperature?
- Cost of finding locations - doing deals to get the land for server farms is likely costly.
- Actually buying/leasing the land - how in advance do you do this? Years? Months? Weeks? Leasing land that lays dormant for say 18 months is wasted money.
- Getting approval to put in the required infrastructure (data cables, electricity, redundancy etc)
- Engineering/architectural costs to design buildings each and every time for varying sizes, in different regulatory environments.
- Cost of construction.
- Wasted capacity. There may be a need worldwide for say 50,000 servers, but with multiple continents, that may equate to several locations with 100,000 capacity.
All those costs add up.
Compare that to the marginal cost is adding a standard pod, with a lower server numbers per pod, mass manufactured in one location and shipped to a regulation free spot in the ocean. The marginal cost is not millions to go from (bogus numbers of) 100,000 to 100,001, it is the cost of one additional pod, at a much lower number, e.g. to go from 10,000 to 10,001 costs a fraction of the 100,000 to 100,001 example. The waste is also reduced, as multiple countries/continents can have say an additional X% capacity, added in smaller increments, rather than huge leaps in (initially) wasted capacity.
Now, is that overall cheaper? Maybe, maybe not, who knows? But many large hosting businesses might prefer say a 10% increase in theoretical cost per server added in smaller discrete numbers of servers. Factor in adding capacity in smaller increments with less capacity waste, less need to acquire the rights to land in advance and it might be cheaper in aggregate, even if it is theoretically more expensive per server at currently active locations.
Incidentally, that is the core value proposition of cloud computing like AWS - that you can add capacity incrementally. Despite there being theoretically cheaper options running your own hardware, this is larger capital outlay, with longer turnarounds to add capacity.
Use immersed electronics.
ROVs and the like have quite complicated onboard control systems. They're immersed in mineral oil (or something like that). They don't get wet and hydrostatic pressure is no longer an issue.
Provides great thermal conductivity though.
Splitting the data center into thousands of "pods" will also greatly increase the cost of cabling and connectors for power and data distribution (we're talking $100 per connector here) and the number of cable failures will skyrocket. Did you know that fish like to chew on submarine cables? Unless you get steel-armored cables, which are defended against "fishbite"...
/source: worked for a small cloud computing VPS provider
In the context of the discussion about cost disease, this is quite remarkable for demonstrating how regulation and local differences have raised costs to the point that a large company is seriously considering building data centres beneath the waves.
Of course the big problem was latency. It was one I couldn't conceive of a good solution for.
This is much better. It takes advantage of the same idea but you can park it right next to a population center and move it around the world fairly quickly as demand requires.
I love this idea!
Sure, you might be at 50% capacity after 3 years, but that's just like lithium ion batteries.
I think GP is proposing you start with, say, 11000 servers when you have a planned capacity of 10000, and just cycle off bad ones for spares (or just diminished capacity) until you redo the whole system.
I believe that's how Microsoft operates their server farms, but I could be mistaken.
When you have a 100,000 of anything electrical MTBF will hit you every day.
Regardless of how it works, I wonder if it's possible to get significantly more performance out of chips that are explicitly designed to operate at a much lower maximum temperature? Let's say you want to build a datacenter somewhere like the Northern coast of Alaska or Antarctica, and you have year-round reliable and abundant access to sea water that's barely above freezing, and suppose that you're willing to order enough CPUs / GPUs / bitcoin mining ASICs or whatever to justify your supplier to design and manufacture custom chips to operate in a lower temperature range. Is there some potential boon to performance or reduced manufacturing cost (due to looser tolerances) that might make such an effort worthwhile?
Guessing, but in the balance, I would assume no. Power generation costs are higher and service is less reliable. Staffing is more costly. Transport for all other resources is also more costly and at a much higher delay than in many other areas that are traditionally used.
From my own experience with high power (~25kW) FM radio transmitters, some of which we have at the top of mountains in the middle of ski-resorts: while cooling costs are reduced throughout the winter, they /aren't/ totally eliminated, so you still need to size and maintain the equipment for summer loads anyways. Getting to the site to do work and maintenance is an absolute chore; replacing the transmitter was a 10 man job where similar rigs in other locations only required 3.
Although, if it was the summer, after we finished our work, we got to take the alpine slide down the hill. As to why we put it there in the first place, signal coverage was excellent, and they already had the power we required available due to the chair-lifts being nearby. If we could have put it somewhere else and got the same coverage, we would have.
But you wouldn't tell anyone about it lest you lose your competitive advantage!
[1] http://www.popsci.com/article/technology/jellyfish-shut-down...
With data centres these days trying to keep chip temperatures as low as possible, this really makes the temperature gradient very low.
An alternative would be to pump in a little bit more energy and use the exhaust heat as a source for a heat pump.
I might be missing some aspects but would it not be much easier to locate the datacenter on land but close to the ocean (or other large body of water), with similar principles as used for cooling nuclear reactors? Then you would get the benefits but would not have the same strict requirements on reliability of the components.
If you locate it on land you would also have the benefit of using the excess heat for house heating (if located in cold climate and the temperature difference to outside is big enough). This is done in a number of cases in Sweden with excess heat from large industrial plants [1].
[1] Report in Swedish; http://www.svenskfjarrvarme.se/Global/Rapporter%20och%20doku...
I don't know how it'd work for something like a submerged data center, but blocks of area are indeed sold off by the government bodies that administrate them to O&G producers. This is also not cheap.
Like the Toronto office towers.
I hope to see them succeed though, especially if the data centers can be self sustained with wind or ocean (wave, tidal, ...) power.
https://www.digitalocean.com/company/blog/announcing-atlanti...
>Our investigation was a great success: not only were we able to reduce our electricity costs by 35%, but we discovered our high-density SSD storage was even more dense at 87atm!
Time to replicate / H2O-as-a-service it.
http://www02.abb.com/global/seitp/seitp161.nsf/0/0eed46e5302...
Also, it wounld be awesome if we could moor these to offshore wind turbines.
There are ways to use cold water for cooling without submersing things. Many of the buildings in downtown Toronto (including the 151 Front IX point and datacenter) are cooled from deep lake loop cooling/heat exchange: https://www.google.com/search?q=toronto+deep+lake+cooling&ie...
For example smart mines that will be powered, but indistinguishable from the signature of these "server farms", then hydrophones and other such technology for locating and tracking submarines and so on.
"About the Authors" makes clear how no thoughts other than the clown game were entertained by the article, but aren't we past the times that things get written about 20-30-50 years down the line. :)
Securing a datacenter from intruders underwater would.
Amazon, Microsoft, Google.
Off by two.
Build on the land. Put a heat exchanger in the ocean. Pump nice friendly coolant to and fro. Forget the idea of setting up shop literally underwater.
What about putting the data center in some kind of low Earth orbit? Something within the magnetic field, but far enough out to radiate heat? Would the latency still be a factor if your web server and CDN were hosted "in a cloud" on numerous communications satellites?
Also cosmic rays will wreck havoc on all your delicate silicon chips.
To transfer heat, you need a medium for the heat to transfer through. I believe some heat can be lost due to radiation, and so they have these big metal fins as heat sinks to radiate as much heat as possible, and reflective material on the solar facing side to prevent more heat from being absorbed.
The water, on the other hand, is a fantastic absorber of heat, and I'm guessing it's easier to drop one in the ocean, rather than launch one to space.
In hard space all the heat is lost as radiation (minus and incredibly tiny loss from hitting the very few particles in space), the dark side can be treat as a blackbody for a rough approximation.
One of the reasons that the side exposed to the sun is mirrored is to reflect as much of the radiation falling on that side as possible as the heat is so hard to get rid off, people think the shiny part is gold but it's actually usually mylar or something stacked in a layered insulation.
Space is a horrible environment to make things work, metals spontaneous weld themselves (vacuum welding), radical differentials in temperature depending on orientation, your electronics are constantly hammered by high energy radiation, micro impacts by things travelling at velocities that make a sniper rifle look like a slingshot, it's fascinating as a layman what it takes to keep stuff working reliably up there at all.
http://www.projectrho.com/public_html/rocket/basicdesign.php...
Also is the issue of having these very long cables running through airspace. Airplanes would have to be made aware of these wires. I can also imagine a lot less electromagnetic interference going on under the water.
Also, heat transfer becomes more difficult. No longer is there conduction of hot metal to cold water. In space, you have one main option for transferring heat. Radiation. I think this is addressed in your edit, but yeah, there's the Sun adding energy while you're struggling to radiate out your own server heat...