What It's Like to Work in One of America's Data Centers
wsj.com
wsj.com
When I joined that ISP we had ~2000 subscribers, and one modem rack that could take 30 concurrent 33.6k dial-ins. The company had just been acquired by a small independent telco in Cheshire, headquartered in a small town that is most famous for a weekly farmers market and an annual steam rally.
We shared that headquarters at first with the founder's other business - a dating agency for rich people in later years - and slowly took over.
The building was a farmhouse of sorts. We'd turned into a rabbit warren of offices, and then, we decided to expand.
First we turned one of the offices into a "machine room". We didn't even bother with fire suppression that first time, we just ripped up the carpet and laid some laminate, put some 19" racks in and got the power upgraded. The air-con was insufficient in that room and the following summer I had to wheel in portable units and had a Cisco router in there report that the inlet fan had a temperature of ~80C. That was around the same time the chief of staff was asking us why we wore jeans and t-shirts and couldn't we wear suits? No, no we could not wear suits.
Bit by bit we took over more rooms through the building, we started working in a portacabin in the car park (where the neighbouring farmer's sheepdog would join us for lunch).
At some point we needed to install ~20k modems, (Nortel CVX1800s - 56kbps had arrived!), and the telephone switch to run it, so somebody bought a second hand DMS100 that went into another room we had converted.
By that point the dating agency had moved out.
We needed another ~100 servers as well to manage email, web hosting, and so on. Those came in parts (motherboards, cases, RAM sticks, hard drives - about £500k worth in 1999 money), and we had to build them and install BSD on them (first OpenBSD, then FreeBSD, long story). We also had to make our own ethernet cables, which is where I learned I can't wire ethernet cables if my life depended on it.
Within 12 months from when we started, we were handling 750,000 dial-in customers.
Around September of 1999 I left and that was when the company had heard about a custom built data centre that was up for sale in Milton Keynes, so decided to move everything there. Around that time they were pouring concrete under the floors of all those rooms we'd thrown racks in because the timber floor joists (!) were starting to give way under the weight of the kit.
That company got acquired and then went through a weird route, but basically it's still going as a major UK ISP. I'll keep the name private to protect the guilty. :)
2000 subscribers, and one modem rack to that and a business still operating 27 years later. A lot of sausage rolls, plenty of sticking plasters for the cuts from those shitty cases, and almost every day ending with a couple of pints in a pub somewhere.
One of my old colleagues and I often think back to that being one of the best years of work in our lives. But... data centre work is horrid, dirty, hot, and underpaid for what it is. But you can - sometimes - be part of something kind of cool and fun, and something very few people get to see or understand.
But there's a reason I've stuck to software ever since. :)
Are you me? would always take me about 10 attempts to even get one where all the pins would light up on a tester. But not good enough for high-speed data.
Kids these days, don't know when they were born. They have push-through RJ45 now, piece of piss and you never have one shorter than the other.
I don't miss on-site/DC infrastructure anymore. No more having to squeeze into tiny spaces, balance a lot of expensive kit on your head, and blood donations to the server gods.
Compute is so ubiquitous now I only have like 2 nodes in my lab anyway
It really bothered me that for how much enterprise gear costs, the manufacturers couldn't be bothered to deburr the edges of rails/servers/etc.
You left out "loud" haha. Just the nonstop buzz; it's like you're working at an airport tarmac or something.
We had a full site shutdown last week, which involved all power being shut off, which meant no running ovens.
The lack of sound felt so strange, like I was somewhere I shouldn't be, or in some sort of liminal space. It was unsettling in the silliest way.
Which is really odd, because just leaving the building doesn't have the same effect.
Most of the work in/around DCs is contract of some sort. Local structured cabling firms, electricians doing stints of various lengths, security, etc. Some of it good and lucrative, some of it not.
Most of the overlap I see is when there is a commercial building collapse (economically, not literally) and you end up seeing master electricians working on residential. Much more rare to see the opposite movement unless it's a permanent change.
Not only that, but the trades stick to their trades.
You don't get, for example, datacentre electricians rocking up pretenting they can install network cable to spec and running away having (if you are lucky) only done a continuity test on the cable they just installed.
Meanwhile every man and his dog, or even just the dog, will very happily turn up at your house and tell you they can install your CAT6 to spec for you when they've never seen a Fluke in their entire life.
Needless to say I've re-pulled a lot of those cables due to so many failures.
We had to redo and re-pull so many circuits in the last datacenter project.
I frequently have my electricians pull and terminate network cable if it’s only one or a few (daisy-chained VAV controllers or a network drop for a BAS panel), if the customer needs a formal TIA-568 test then I’ll sub that out. My electricians are already pulling and terminating building automation cable, CAT 6 isn’t gonna stop em :)
I’d like to learn more about PLCs and SCADA, but that scope of work usually falls on another contractor, we mostly provide installation for BAS systems that control HVAC and chilled/hot water plants.
I do some water treatment plant projects and there’s a local contractor that handles the PLC and SCADA stuff.
I want to know what country you live in that you can find domestic tradespeople who will do work to the same standard as datacentres, and do it at domestic prices. ;)
DC would be the kind of place with a lot of government contractors that would have this kind of stipulation, is my guess.
Power is the biggest one here in Texas, so much so even this laissez-faire state has stopped new permits for data centers. All the inner connect space for grids is used up and the wait for hardware and installs is massive.
So much so we're seeing private grids being setup for data centers.
To add, the numbers are actually this crazy.
https://dcmap.us/states/texas/
>Texas has 623 tracked data center facilities and projects: 349 operational, 101 under construction, 156 planned, 2 paused, and 15 inactive or historical. Reported or announced power totals 96.1 GW, including 85.4 GW tied to planned or under-construction projects.
This is for just one state.
>4,998 facilities and projects nationwide, with 374.7 GW in reported or announced power figures. Of that, 318.6 GW is tied to planned or under-construction projects. Climate-risk and water-stress labels summarize current records with those fields. The directory retains 150 inactive or historical records. States are ordered by current reported or announced power.
For example our office is heated using heatpump that has >22C of water heated by equipment as a source.
https://delska.com/about/news-resources/delska-newsroom/pres...
In the late 90s I worked at an insurance company in leafy Surrey[1]. I was told that they used to use their IBM mainframe to heat the swimming pool[2] effectively for free; when they upgraded they had to buy and run a heating system for the pool as the new mainframe didn't pump out enough heat.
[1] In England, south west suburbs of London to be precise.
[2] yes it had an indoor pool, badminton courts, outdoor tennis courts and other facilities - a great place to work.
And the way things are structured is that heating is amended and not exactly a direct connection to residential buildings at the mercy of DC load.
Thousands of people Google "why is my building cold" causing the AI DC to wake up and pump out heat. AI sees the heat and reports "heating is on, you're fine" then shuts down.
They are now stuck in a cycle.
You're using engineer in a highly context specific way; yes, in some contexts, engineering is a licensed field, but the general meaning of the word is:
> Engineering is the practice of systematically applying natural science and mathematics to design and improve systems, devices, or processes that solve problems under constraints.
Which is a word-for-word description of the sort of work (…some…) software engineers do.
(Perhaps less and less "systematically applying science" these days with the slop fest out there … but the field should be doing engineering.)
Wiping tears with my Portuguese tax return this year.
The licencing of some engineering is simply a relatively recent regulatory capture but that does not limit the types of work that engineering covers.
There, saved you from reading a story that isn't really informative and only was written because of the politics surrounding DCs are keeping them in the news.
https://news.vcu.edu/article/northern-virginia-data-center-a...
Presumably there are some data centers that use water cooling. I think google and amazon's big ones do. But even then, they're not "polluting", they're using them as heat sinks.
There was a lame rolling stone article that claimed amazon was "polluting" in Eastern Oregon, but what Amazon was doing was taking brined water from an aquifer and putting more concentrated brined water back into the water table. Air conditioning doesn't really add brine, so even the premise is ridiculous. I know that article has been referenced a lot by the "Data Centers Pollute The Water" folks.
Secondly drawing brine water from the aquifer to then putting the stronger brine back is essentially water consumption. I would also say it's externalising environmental cost of the stronger brine impact.
Yes, it does. As far as water is concerned, air conditioners are open loop. Air conditioners make water from condensation forming on the inside (cool) coils. Modern window air conditioners then splash that water on the outside hot coils, to benefit from evaporative cooling. I haven't looked into the details about data center water usage, but I would guess the cited usage is from evaporative cooling.