Hacker News Reader

Comment by rtp4me | Hacker News Reader

In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacements (velcro, tie wraps, etc). There is a fine line between cleanliness and usability.

For the folks that work with me, I explain the importance of air flow, proper cable length, and component serviceability. You definitely don't want "piano wire" stretched across the rack that prevents someone from replacing a PSU or NIC without affecting other gear in the rack.

View on HN

Couldn't agree more.

And the latter (tied down) quite often devolve to the former over time exactly because it's so much effort to maintain that suddenly an urgent repair happens and ties get cut and cables torn out and something new just thrown in, and the effort of re-doing the original setup means it gets deferred.

I stopped tieing cables together years ago for that reason. The more the perceived effort in getting what needs to be done out of the way + fixing the cabling afterwards, the less chance it gets done properly. Instead I'd aim to set an example of stretching the cables out of the way, labelling and getting length roughly right that is simple enough to work with in a rush and trivial to "fix up" afterwards.

In large environments where crises are more likely to involved routing traffic away from entire racks or data centres, you can afford being a stickler for these things. Most places are not that large, and being too strict creates more problems than it solves.

A particularly frustrating variant of the 'piano wire' issue:

A redundant pair of devices, except cables have been run in such a way that if one device failed, you could not remove it from the rack without removing cables from the remaining device.

/facepalm

this is the reason why (ideally) you should have three racks in a redundant setup:

2 for redundant component placement (with redundant power obviously). 1 for interconnections between the racks, having prepared cabling and MPO panels installed to make interconecting devices easily, without having to change anything in the equipment racks compared or vice versa.

Makes replacing hardware a breeze, and prevents a lot of issues with cable mess inside equipment racks because those cables are very short to MPO panels.

one of the key things to do in a 'serious' datacenter is separate the fiber from the power.

this is why serious facilities have a dedicated channel of yellow plastic overhead fiber tray which contains ONLY 1.6mm or 2.0mm jacketed 9/125 singlemode fiber, with nice smooth corners and spigots on it coming down above racks.

and then the AC power is in conduit and long rectangular metal enclosures.

and the -48VDC cabling on steel ladder rack, often with waxed string lacing if it was done by people who are being serious about it.

on a per rack cabinet level it's also really important to pick one vertical side for your power cables, and data cables vertically up the OTHER side, and stick to it consistently.

We often end up with AC power down both sides for A and B busses. Prefer the "zero U" type solutions for that.

Why is this? Does power interfere with the fiber?

No, not at all, it's for proper organization, management, labeling, flexibility to add and remove things, etc.

Then you get a bunch of boxes where "connectivity" and "power" is on opposite side than the other boxes ;/

what is also very common is having power ran across different power rails under the floor of the DC (different feeds obviously), which is then terminated upwards to a PDU into the customers rack.

Fiber comes in from the top, with a MPO or patchpanel at the top of rack for customer termination.

I don't know of anyone building new raised floor datacenters. Cabinets and equipment on concrete slab and everything power and communications related overhead.

Is there any good resources for learning more about this? I have a lot of legacy AV equipment mixed with modern digital components and networking and despite best efforts, I typically fall into either a rats nest or something that’s too constrained to repair/modify without undoing everything.

I mean honestly there's not too much expertise involved, it's mostly common sense. It's just that it takes so much longer to do it right at the beginning to ensure its maintainable and ordered rather than just doing it the easy way and hoping that it's someone else's problem by the time someone actually needs to find the other end or replace a cable.

Making your own cables that are the exact length they need to be instead of using preterminated ones and just stuffing the slack into the side of the rack, routing them down the side and through the floor or the cabling trays on the ceiling, standardized naming schemes for labeling devices/ports on either end of cables, documented rack diagrams, etc.

i once had to make up about a 100 ethernet cables of varying lengths at once. Man, untwisting, sorting, and separating the individual wires to insert into the connector before crimping made for some very sore fingers.

Not sure of any specific resources, just years of running cables :-)

Some of my lessons learned:

* Whatever you do, be very mindful of future events (repairs, replacements, etc). Spending an extra 5mins upfront when you are in a hurry will save you lots of time later.

* Label each end of the cable if possible. It is a PIA in the beginning but will be a real life-saver when you need to do maintenance

* Replace old (fat) cables with new, thinner cables of the proper length when possible. I typically add 1ft extra to any cable for a little slack.

* Think of airflow. Lots of cables bundled in the back of a rack can seriously cause airflow issues. Try to run all cables along the sides of the rack.

* NEVER run cables across the back of gear - especially for multi-node chassis that require rear access for maintenance

* Velcro is your friend.

* Stick with a cable color standard if possible. For me, I use white for IPMI, blue for data, red for external access

* Keep an updated diagram in the rack for the next guy/gal who needs to do work

* Always remove old/un-needed cables after a maintenance job.

* Keep the rack clean!

Velcro is your friend. So much this!

Abandoned-in-place wiring is the bane of every tech's and installer's existence. It's impossible to have worked on legacy equipment and not spent some time reterminating abandoned cable :/ Not to mention having to weed through it when tracing problems.

We're in an ongoing battle with abandoned-in-place wiring in our current building. There's data wiring back to ARCnet (93 ohm coax), and power wiring going back to knob and tube days.

> 93 ohm coax

ooo nifty. be on the lookout for vampire taps. they never truly die.

Wow - vampire taps. That brings back some memories :-)

Thicknet stuff's just another day at work here:

https://imgur.com/a/GDUR36j

Code requires abandoned cable to be removed from ceilings. Burning cable is toxic - reducing the amount of cable is beneficial if there is a fire.

HAHAHAHA CODE.

800A 120/208Y service entrance conductors live in the same raceway as 20A branch circuits. I got shocked off a lighting circuit with the main open once.

This is a former The Power Company building. I always heard it joked they could only use leftovers from customer jobs on their own stuff.

I understand that some landlords have it written into their leases that all network wiring will be removed when the lease ends because of this very reason.

One building we were in had Twinax above the drop-ceiling (used for System-36 and AS/400 terminals). Plus dozens of multi-pair phone cables that had been left there. I wish I had all that copper now that prices are much higher.

"Finger duct" is often the easiest way to get relatively clean results in a situation like AV where cabling is sort of poorly standardized (connections front and back, inline DC supplies, etc). It's fairly cheap and often run vertically on rack posts and horizontally above and below cable-dense areas like patch panels and switches, but in an AV situation you probably only need one vertical channel to get a big improvement in results. Finger duct can get pretty messy without planning but I think it's usually the best results to effort ratio.

Often used inside industrial equipment as well. Routes pneumatic lines pretty good.

I have been in some data centers over the years know one data center I visited multiple times where one blade chassis (their newest acquisition I think) stood on an old (wobbly IIRC) desk a meter or so from the closest rack and not near anything that could support it.

It was connected via a way too long fiber optic cable, of which most was placed on the end of a broomstick on the nearest server rack, and then the end of it was either wound around or tied to (I can't remember) the part of the broom stick that stuck out from the top of the rack towards the desk with the blade chassis before it fell down to the back of the chassis.

I'm thankful I never stumbled into it.

My first real IT job was doing all the grunt work (and whatever sysadmin tasks I could talk the greybeards into trusting the new kid with) in our two DCs, one at the corporate office and the other at our manufacturing plant.

Our DC at the corporate office was OK aside from the patch panel which was a fucking mess, that and when they originally built the thing however long ago they installed the switches at the top of the racks BACKWARDS so the ports where at the FRONT of the rack instead of the back with the server ports and they just ran with it instead of taking 20 minutes to flip them around, so instead of neat orderly cables going up and down between the ports you had these giant bundles on either side of the servers going from front to back I had to unbind and untangle anytime one needed replacing >.> This[0] is the what I managed to get our patch panels there reduced too and slightly cleaned too after removing all of these[1] cables that were still plugged into dead ports that my predecessor just never bothered to take out after retiring or recabling servers that had been on the other end. I'm 99% sure he was just running new cables and configuring a new port anytime someone called to complain about an unreachable server instead of trying to find the other end of the relevant cable.

[0] - https://i.imgur.com/PIPhAal.jpeg [1] - https://i.imgur.com/PIPhAal.jpeg

Wish I still had some pictures of the plant DC though because holy cow it was even worse, no patch panels anywhere, but they did have a drop floor so I could pull the tiles up and run cables down there if I was so inclined. My predecessor, not so much. He seemed to prefer running them between the tops of the racks, with random cables crisscrossing above your head, always drooping down and getting snagged on the step ladder as I carried it between racks, or getting caught in the doors constantly. And they would never let us have any downtime to clean up the cabling, nor could I talk them into just switching the network configs to use a different damn port on the actual servers so I could just run new fucking cables ;_; I had to settle for obsessively tracing and labeling each end of the cable with the port# and device that was on the other end of the cable, because otherwise every call about checking on a server turned into an hour of either tracing cables either under the floor, pulling tiles one by one to follow it to the other end, or a jungle safari sorting through the tangled mess of cables on top of the racks following the one in question as it arced over multiple aisles to it's final destination. Which of course I had to do anyways to get them all labeled, but at least this way I only had to do it once and it was done forever.

Good times.

[0][1]: both links are the same.

Crap you're right. Here's the pile of removed cables.

https://i.imgur.com/PLCphKA.jpeg

Square D panelboards, classy place :D

looks like the floor of a hairdresser.

> long ago they installed the switches at the top of the racks BACKWARDS so the ports where at the FRONT of the rack instead of the back with the server ports and they just ran with it instead of taking 20 minutes to flip them around

There might be a good reason for this, it sounds like they messed up the switch order and got front to airflow switches. Normally you would buy rear to front airflow, as the front of the cabinet is the cold aisle and the rear is hot.

In a shared colocation like Equinix, they will make you conform to these standards and have you rack a switch that way so the airflow is correct. Incorrect airflow lowers cooling the efficiency of a DC.

Front to airflow switches still have their place though, you typically find them on two post telco racks where the patch panels are.

They definitely just didn't care enough to mount them the correct way, I have very distinct memories of getting blasted in the face with hot air any time time I was standing in front of the racks. I'm reasonably tall and have longer hair, and the top U of the racks was juuuuust high enough so that my hair would be whipping around the entire time.

Yikes that’s pretty frustrating!

> In my experience, data center cabling is either a huge mess (random cables just hanging everywhere) or is tied down so much you can't do any repairs/replacements (velcro, tie wraps, etc). There is a fine line between cleanliness and usability.

We have 7 racks that went thru some changes and never in the history (12+ years) needed to replace any broken ones. It just doesn't happen.

What did happen were re-wires when some hardware were replaces but smaller ones were small enough that whether it was nicely bundled "permanently" or not didn't matter much, and bigger ones were "take everything out and recable" job anyway so again, what was in the rack (and there were some spaghettified ones too) also didn't matter.

Yeah routing it properly the first time helps but making it "permanent" by tying it down isn't really a problem, just put another bundle for next 5 servers.

It just doesn't happen

Depends on the datacenter. I work in a large testing environment, so equipment is being moved and parts are being swapped out all the time. The cabling needs to be nice enough that you have good airflow, can see all of the components, and to make swapping parts easier. Perfection isn't necessary but cleanliness makes life easier.

Reply on news.ycombinator.com