The slow rise of robots in the data center
datacenterdynamics.com
datacenterdynamics.com
Bizarro security roleplaying aside, Switch is probably the most professional and competent data center operator I've worked with.
Indeed it is. But sometimes real life has odd moments, and some data can be extraordinarily important:
> The dramatic rescue of title books from doomed abstract businesses proved a greater public good when all official land records were lost [in the Great Chicago Fire of 1871]. John G. Shortall, who forced a passing wagon driver at gunpoint to load his records, would thereafter be remembered for more than the arrangement of legal conveyances.
> Shortall and Hoard, Jones and Sellers and Chase Brothers and Co. - who have saved nearly all of their papers, including the indices to every piece of property in Cook County, and actual abstracts to a large proportion of this property. We have one firm - J.H. Rees and Co. - who has saved copies of all the maps and all the plats ever made of Cook County property.
> In April 1872, the Burnt Records Act was passed by the Illinois Legislature, and the existing abstract records of the three companies were made admissible as evidence in all courts of record.
It isn't. Switch is famous in the telecom/datacenter industry for weird security theatre. Notably they started having conspiciously armed-with-AR15 guards at their Las Vegas area facility 15+ years ago. Compare them to any major carrier hotel/critical IX point in the USA, they stand out as weird. And they're not even as critical as they claim to be.
I can't share the details but the real, hidden, non-theater security behind some of the US's major traffic exchange points very much exists but is not on public display for touring the rube customers.
-- Sith admin
So it might be a requirement?
"Prior to FedRAMP, the Security Control Assessor (SCA) had to visit the data center to check the “gates, guards and guns” every single time, even if that specific assessor had previously visited that data center. That is no longer necessary. The FedRAMP ATO takes care of all of that"
https://www.linkedin.com/pulse/security-control-spotlight-in...
(Ie any serious attacker will look past the theatre.)
Nothing to do with packet switched networks
Quick, someone tell him about floors, walls, ceilings, doors, and electric light.
There are of course planes that have flown for that long, and longer. The B-52s come to mind. But the SR-71 isn’t one of them.
The mechanism by which this is enforced is an "Airworthiness Directive" if you want to read more about it. Source: I used to fly a small 50+ year-old aircraft recreationally
Oddly enough, I found the reading pace of this article similar to the change it describes. Nothing negative mind you, just gently rolling in its delivery.
The magtapes used to hang on row after row of metal racks. The operators would retrieve the tapes, hold them on their forearms, and bring them back to what were usually a centralized set of tape drives.
Here's an offset into a video showing what it typically looked like. [0] The narrator refers to a "search tape" used to retrieve the person's information. The tape had to be retrieved from the racks.
The sheer weight and overall length of the rows/racks meant the rooms were often lower level with vinyl flooring over concrete (raised tile later). Imagine walking up and down for an 8-hour 5pm-1am shift. Not difficult to imagine skates.
Here's another video with a good representation of how it worked. [1]
As I have come to accept, the mid-80's was pre-everything. Deliberate change takes time.
[0]: https://youtu.be/Iddrm7mHPrY?t=111 [1]: https://www.youtube.com/watch?v=Cwj6pfhWBps
See how the vibrating loop passes back and forth across a hole in the middle of the column? That's a pressure sensor. As the loop moves across the hole, the pressure in the hole changes, telling the drive whether it's time to reel in or dispense more tape.
More info: https://en.wikipedia.org/wiki/9_track_tape
I was once in a datacenter that had a golf cart to get from one end to another (yes - it was that long).
Best part. You could roller skate to the music.
The main issues with robots in my experience is whether you can get the desired throughput while including planned and unplanned maintenance work, reconfigurations, etc. The upfront capital cost generally means you don't have the ability to have a drop-in replacement ready to go when a change is required. That results in downtime. Folk familiar with LEAN six sigma will know that hidden bottlenecks should be avoided or you build up large backlogs during downtime.
Good news for datacenter engineers is that fixing robots may become part of your remit in the future (interesting niche given the mix of software and mechanical). More interesting work will always exist!
The heart of any competent operations team, whether its in computer systems or plain old logistics, is drills. You can't simulate every possible scenario, but you try to analyze and get an initial set, and the build that up as you run into actual production issues. What you want to avoid is running scripts for the first time in the middle of a disaster.
As things are: even in well-managed data centers, the racks themselves are always somewhat finnicky, with varying levels of precision in assembly from rack to rack that require odd workarounds for equipment installation more often than one would expect. And that's not to mention how incredibly variable the rack enclosures themselves can be, which has big implications for cable routing. And nevermind the fact that there's basically no standard for port placement on rackmount systems.
Rack-and-stack labor is dirt cheap, or easily foisted off on your sysadmins for small deployments. I don't see a robot for this being competitive from a cost standpoint unless that robot is extremely general purpose and able to fulfill other roles.
Actually getting the parts on time has been more of an issue in my experience, or finding out that your entire rack is forfeit because it fell over at the docking bay.
I can count on one hand the number of times I've seen rack-and-stack go too slow for the consumers to actually use the hardware. Usually the hardware is sitting waiting to be provisioned for eons.
The real labor is pulling bad drives/boards and rejiggering the network after the racks are in. So many outages are caused by magic hands making an error and pulling the wrong cord.
Of course, at that point it you’re adding a bunch of cost and may be better off just designing the servers to last for like 10 years with enough redundancy (and hot/warm/cold spares) to not need any physical swapping, then replace the whole thing with a forklift at end of life.
He told me they had a robot like the ones from storagetek but for unplugging faulty blade servers and plug back new ones.
Makes sense for large scale installations.
I think the same thing when people talking about sending a 'plumbing robot' to your house.
Contingent workforces are cheap (minimum wage essentially).
It's also cheaper to use connecters that mate on insertion than to use connectors that humans are accustomed to.
This will take infinity to accomplish (barring disruption in compute/storage) because -
* Remote human hands are ridiculously cheap, and they can perform functions robots can't without extra pay or a paid for software upgrade (think shipment handling, etc). Only real use case that comes to mind is one without economic justification, but of necessity. Think datacenters in space or deep underwater, or hostile environments. So maybe innovation in this field will come from the government this time, and trickle to the private sector. Maybe.
* The cost of such investment far outweighs the financial gains. In the very competitive cloud business, it's the services you offer that matters most, and the reliability of your systems. Sending a person with a code scanner to verify and do a quick disk or cable swap is not a risky endeavour. Unplugging the wrong cable and causing an outage never happens on properly designed systems, with properly set maintenances, so this is irrelevant (when was the last time you've heard that in an outage retrospect?)