Cisco is making it more difficult to use used hardware | Hacker News Reader

Cisco is making it more difficult to use used hardware

ifixit.com

178 pointsturtlegrids92 comments

Cisco has been fighting second hand sales of their hardware for well over two decades now. There was a flood of surplus gear after the first dotcom crash. Lots of folks, myself included took advantage of those circumstances.

Eventually reputable vendors began to pop up. I recall doing a great deal of business (>$1m) with a company called "Network Hardware Resale". They're still in business as "Curvature" and I believe still moving second hand network and server gear as well as providing support.

There have been times that I recall when lead times for Cisco gear have exceeded 8 weeks, when I didn't have any other options other than using refurbished gear.

From my perspective, Cisco is nothing more than a company looking to extract as much money as possible out of Enterprise IT shops. They pretty much gave up the service provider market. IOS has gone a million different directions, and they've been slow to embrace IAC.

Back when I was buying refurb gear, I was also buying some new and some officially re-marketed gear. I'd get support for the chassis and supervisor modules (Cat 6500) so that I could get reliable software updates.

My sales rep's regional manager sent a threatening letter to the VP of my department with some dire warning about how we were in violation of the law or some such. We all ended up in a meeting in which they produced a list of serial numbers of gear that we'd sought support on that were supposedly grey-market. We produced the same list of serial numbers on invoices for legit Cisco certified re-marketed equipment. They'd just failed at record-keeping. My VP was livid and threw them out of his office. Pretty sure they got fired after.

Anyway, this is all just a tactic to extract the most money possible. It'd be irrational to choose Cisco for any new network deployment these days.

> It'd be irrational to choose Cisco for any new network deployment these days

I'm interested what you would buy instead? In the UK SME space I see mostly HP, Watchguard, Unifi. Lease lines still often come with Cisco gear (supplied by the provider). Otherwise I only see Cisco at the bigger entities.

Mikrotik has recently made bridging much saner, which makes their portfolio complete for datacenter use and pretty close on bridging users in the access layer.

Their CCR-series is very much bang per buck if L4-filtering is enough, which should be enough in today's world of end-to-end encrypted communications.

The CRS3xx-series as stated has become much saner with changes to bridging and with most features implemented are also worth the cost.

The upside with affordable devices and the licensing model they have, make it possible to keep cold spares available in case of disaster.

It does however make sense to study the management interfaces and disable all but SSH and HTTPS in order to minimize attack surface.

I have a phobia of MikroTik, especially in the hands of general IT people. Amazing bang per-buck, but I took over a network with 6 sites all with Mikrotik routers and there were some staggering config mistakes. The more I dug the more I became convinced that the OS was, if not to blame, but certainly was a major factor.

I used to test changes (when duties allowed) with nmap, and several times I showed experienced engineers that they had left a service open to the WAN by mistake! When a network is in the hands of general engineers, like it often is in the SME space, I like Watchguard firewalls. Very good defaults, helpful os.

I can imagine in a datacenter where you have network specialists, and robust working procedures Mikrotik could work well.

I use Unifi for Wifi, and have used their routers for a dedicated guest wifi network. For switches I know nothing other than HP, but don't see a lot of issues, especially with the Aruba kit.

I again have phobias regarding Watchguard and other products that have the consumer style special WAN-ports and related configuration restraints. Many Mikrotik-devices come preconfigured like consumer devices but the recent CCR-series does not.

The firewall in Mikrotik-devices is among the cleanest I've seen and very hard to miss-configure as long as the firewall is otherwise configured to not let unauthorized traffic through.

And yes, people are people and this is why we educate people when needed.

Depends on the type of network element required.

If I were doing data center switching, I'd buy something Broadcom based on which I could run something like Cumulus Linux.

For wifi, I like Unifi. For routers, Juniper is fine. Arista seems to be well-regarded in the switching space as well. Basically anything but Cisco.

That said, I haven't had to think about network elements for quite a few years.

UniFi is great for home, small to medium businesses, and anything up to a massive church or stadium. I use it at home and I know a few local installers who swear by it.

If you have a small network team UniFi would be a perfect fit. It’s very manageable and support is great.

Backing this. I use Unifi extensively and like it. There are some quirks, but I've had better experiences with it than Meraki.

in the service provider market juniper and arista seem to be king. There is very little on the market that beats an MX series router when looking for a network core for example.

in the enterprise, ubiquiti and HP seem to still be quite relevant. The amount of HP procurve gear in use is rather large.

I learned MX and JunOS mid-career. Routing protocols are solid and I much preferred the configuration over Cisco IOS especially for things like BGP, ISIS, routing policy, prefix lists, etc.

> Cat 6500

My first job was writing drivers for chips on that switch. My recent experience is with younger tech companies, so I'm sure recruiters get some false positives when they search for resumes with "ios."

Do you recall which ASICs? I was way more familiar with the guts of the 6500 than an end user should ever be.

Wow, it's been a while. Pretty much anything you could poke with "test platform firmware," but r2d2 and tiangang come to mind.

Any of this ring a bell?

https://null.53bits.co.uk/index.php?page=7600-forwarding-asi...

Got a little PTSD just looking at that page.

All this is going to do is increase the efforts for open source switching firmware. We already have working data plane and control plane for selected systems, but at this point it no longer is some far fetched dream.

Is this a firmware issue? It seems to be a change in the OS.

Regardless, ONIE + ONOS has been an excellent choice for a while now.

The firmware on a Cisco switch IS the OS.

It's both and neither. The protection is usually not really implemented in the switching ASICs or any part of the data plane but only in the control plane as you can't influence the data plan anyway.

In the short term, it's also going to encourage people to just crack the protection (if it hasn't already been done).

I used to work on drivers within IOS for a high-end switch. Between how complicated startup processes are, how specialized the hardware is, software workarounds for hardware issues, and the small demand for this, it doesn't seem likely. You're more likely to see Juniper take a different strategy to try to get more customers.

Juniper is just as bad as Cisco when it comes to resold/gray-market gear, and I don’t see that changing. Sucks because I freaking love JunOS and it kicks the ass of anything Cisco has put out along with the myriad copycat designs from HPE/Arista/etc.

I’m hoping someday cheap whitebox switches start entering the market, fs.com has a couple but they’re still out of reach for small shops and hobbyists compared to getting used gear off eBay.

Those switching ASICs are very complex and setup and bootstrapping is complicated on its own, but it's been done for (now legacy) switching and fabric ASICs and it's still being done for more current devices.

A lot of effort is going to the 10G and 100G top of rack models because that's usually where you can backport to 1G/10G access switches. The core switches and more complex models have not seem much success.

It's very unlikely. You must consider that those routers aren't standardized embedded systems, but they are highly specialized with ASIC chips, in-house operation systems with its homebrew network stack.

Even with perfect documentation, it's not something that you can just fire up a BSD on - getting it all supported and running requires many years of works. Many of the functionalities of those routers make special assumptions to boost performance and don't even fit into a standard network stack, so minimally, you would need to develop your own functional network stack for that.

Or at least that's my impression of them, correct me if I'm misinformed.

There are projects like ONIE that provide a common framework to bootstrap “white box” switches, and there are operating systems like Cumulus that can run on numerous platforms.

We’re not there yet but the situation is definitely changing / improving.

> We’re not there yet but the situation is definitely changing / improving.

But only if you can use that commodity hardware. If you need the stuff running on ASICs, it's as the parent describes and I don't see how that could change.

Can whitebox solutions ever offer the same performance as boxes from cisco etc? If not then I'm not convinced there will be much more improvement coming.

"whitebox solutions" are merchant silicon designs, typically Broadcom.

Cisco's entire Nexus line of switches was there entry into the market with merchant silicon. Their costs were much lower, but their pricing per port stayed roughly the same as their Catalyst line driving up margins. Adoption was pretty lackluster however until they started offering massive discounts on the Nexus products.

Anyway, you can generally get non-blocking line rate on all ports out of any Broadcom reference design. The only distinction between that and a Cisco is the how the device is managed.

I confused 'whitebox' and those switches based on x86-based hardware. Isn't the price differentiation also due to a reduced feature set?

Kinda depends.

Layer 3 forwarding is all the same. Whitebox stuff generally comes with simple routing protocol implementations.

Cisco, Juniper, et al. have routing protocol implementations generally guaranteed to work with their other network elements. That's how the L3 forwarding tables get populated.

Whether that makes a difference depends on the network architecture. FAANG companies and others are deploying whitebox switches, building "underlay" networks using eBGP, and using SDN as an overlay basically obviating the need to have much intelligence built into the network element.

Cumulus Linux replicates these efforts to some extent. One gets a whitebox switch running Linux with open source routing protocol implementations and configuration via something like ansible.

Yep, and SDN is also possible if you get switches that support things like being an openvswitch-slave or OpenFlow/NetFlow/whateverflow switch. You basically replace the control plane.

That's why a lot of engineering is based around getting data planes working, and ignoring most of the control plane that runs on the application processor. Even if you only get that to control the ASICs and run a simple REST server to accept data plane control commands it's already enough to turn a switch into something useful forever.

Yes, the ASICs require highly specialised setup and it's the main issue with creating firmware from a reverse-engineering perspective. Most firmwares are either created from reverse-engineering projects which due to the time consuming nature often target older switching chips like Broadcom chips from around 2009.

Setting them up is a PITA and almost an order of magnitude harder than DDR memory training or x86 bootstrapping without an FSP/BSP. But it's possible and has been shown to work.

Other projects like Open Compute and ONIE give a lot of insight for reverse engineering of newer fabric and data plane components which significantly shortens the process of reversing and re-using existing hardware.

The control plane side of things is much easier in that regard as they are commonly generic ARM or POWER architecture SoCs with generic Flash/PROM components running a more generic RTOS (i.e. one that is used across an entire line of products and supping many data planes) or even a Linux based OS. That, combined with easy to extract firmwares and JTAG + Serial makes the control plane a lot less complex than the data plane. This is also why control planes are usually left alone because you can drop in any control plane software we already have and focus on interfacing with the ASICs instead.

> Setting them up is a PITA and almost an order of magnitude harder than DDR memory training or x86 bootstrapping without an FSP/BSP. But it's possible and has been shown to work.

Wow, an order of magnitude harder that booting x86 without FSP, but it's still possible... I can't imagine that. I salute the efforts.

Eyeopening! Thanks.

It's possible but because there are other fabric/data-plane ASICs are are not an order of magnitude harder than Cisco's chips most time is spent on those instead.

The problem with the Cisco ones is mostly because they sit somewhere in between FPGAs and a Barrel CPU. Without the custom loading and init they are essentially useless.

Think of them as an FPGA with a softcore CPU. If they are empty they have no CPU so you can't run code on them. If you fill them with Cisco core data you get no use out of them without logging in to their DRM server. So you need to feed them the softcore but without the cisco runtime code.

This is extremely irritating. I have a fair amount of meraki hardware that was given to me as the reward for a hackathon, and I use it to run my home wifi networks. When the licenses expire the equipment will become completely useless and I’ll have to sell the perfectly functional stuff and replace it with unifi hardware...

You can still flash it, though

Some Meraki models support OpenWRT.

How does the phoning home work if they're on a network that doesn't route to the internet, or where the route to the internet is heavily firewalled? Or does it use a license server? Or is there some other procedure for refreshing the license in that situation? Or can you just not use these routers for that?

Typically (a) they whitelist a route to the licensing servers (e.g. enterprise, "soft" disconnected) or (b) there's a manual, airgap tolerant authentication procedure that must be performed regularly (e.g. generate key from licensing servers, save, move across gap, upload to hardware).

No idea what Cisco does specifically, but I've worked with other gear or software that worked like this.

One pulls the key off the network element, plugs that into a website which produces an authorization key which can then be cut and pasted into the network element.

jesus. in the age of cloud computing you copy/paste keys around :( this is just sad and makes me believe that you'd have to be borderline insane to choose cisco for anything that is started from scratch today.

It brings to focus exactly how anti-customer they are.

Umm, how else would you propose to authorize air-gapped gear?

how about don’t. take my money and give me freedom for MY devices

Licensing server on prem phones home. Devices talk to the internal licensing server.

https://www.cisco.com/c/en/us/td/docs/wireless/technology/me...

Too many customers are ISPs and gifts who do not want core infra connected to the internet.

> Smart Licensing works differently. Companies can acquire a pool of licenses for their account, which are shared automatically among devices they’ve deployed. Those devices phone home to Cisco regularly for validation, and if they aren’t able to do so will go back to “Evaluation Mode” after one year.

It doesn't sound unusual to me, I feel it's just the usual "Enterprise" bullshit, where you have little control or internal knowledge over your equipment, and must follow a restrictive licensing plan and use official support to keep it running. I think the most extreme version is the IBM mainframe computers, if you DRM license module is inoperative, the computer is essentially reduced to a huge piece of useless junk. Other enterprise hardware/software is not too restrictive, but still follows the same pattern, such as Oracle products.

the fact that devices need to phone home can be major issue, especially with networks not connected to the Internet.

Also, degrading a device because of a license issue can become a huge liability.

What if the smart licensing service has a technical error, and all switches in your datacenter go down because they no longer support the software features you require?

From my experience with other datacenter network vendors, most seem to just do a vanity kind of licensing where they basically audit you every X years for licenses.

This seems like a serious issue for the environment by forcing hardware into landfills.

I wonder what the landscape is like in Europe for things like this. I know some countries are more diligent about physical products having a longer useful life than others.

Here’s a thought: require such manufactures to completely recycle their discarded equipment. Suddenly reselling their ‘old’ equipment as refurbished at lower costs is a cost cutting option.

There already are quite strict rules along those lines in the EU. For example, here's the UK's interpretation:

https://www.gov.uk/guidance/regulations-waste-electrical-and...

How does that work? Does the user have to pay to send it back to the manufacturer or does the user have to pay? If the user does I can see a lot of them throwing it in the bin because it's cheaper. Also recycling often means just melting down the metal case and binning all of the insides.

You can make a market of it. Make it possible to transfer the obligations of recycling to third parties with necessary oversight and assets necessary to do the job completely. The responsibility still stays with the manufacturer. etc. see the comment with a link

Then everything gets shipped off to some third world country to be burned or land filled via a company that has Certified Recycling(tm)

This is already the case in the EU thanks to the WEEE directive.

https://www.cisco.com/c/en/us/about/product-innovation-stewa...

NetAPP has been the same way for decades: you can do what you want with the hardware but the licenses are nontransferable. Since I learned this in 2001 I have not bought any of their hardware.

> Those devices phone home to Cisco regularly for validation, and if they aren’t able to do so will go back to “Evaluation Mode” after one year.

I'd like to see their salesperson try and explain the advantages of this to my CIO.

you're put back to evaluation mode so that you can evaluate the choice you made when you picked them /s

The article's "if you can't fix it you don't own it" needs to be a mantra that's repeated at politicians until we're blue in the faces

Ubiquiti needs to invest in their router hardware team; golden opportunity in the market right now. Current gen of their routers are software based while even Cisco's SG3XX of routers have TCAM memory and can route at line speed. I really like their approach: cloud based but its your cloud and only if you want it.

I love Ubiquity. Their CLI is so user friendly.

i assume you are talking about the edgerouter series?

seem like good devices for a small environment or at home, but the stability left a lot to be desired.

The reason for this is counterfeit hardware. All nexus switches, APIC controllers, and UCS hw use TPM to validate software licensing. Invalid license, hardware is locked down. Witnessed this first hand on a bad APIC (ps call your favorite Cisco rep, they will hand out old ACI hardware like candy to a 3 year old)

Back in the IOS days there were many stories of calling into smartnet and finding the hardware you bought new from CDW had a serial different from Cisco’s master list. Cisco would cut you off for fear of liability, the owner would dump the hw onto the used market and lather-rinse-repeat with another unsuspecting fowl.

Why should it be possible for a third party to deactivate my hardware for any reason after I bought it?

Because if a product comes new, and is used in an attack against a customer, Cisco could be found liable. Considering the high stakes implementations high end Cisco gear can be found in, this is not a small concern.

Anyways, when you're buying hardware at this price point, you're not just paying for a box that moves packets.

I don't believe Cisco would be liable. Just like not a single manufacturer was liable for Mirai.

I don’t think we’re buying hardware anymore at this level. I think we’re buying software and support with hardware included. Not sure how I feel about this on all situations but I’d say that’s the only perspective the vendors can defend. It’s too bad for them too since eventually someone will build free software too.

It’s too bad for them too since eventually someone will build free software too.

In many cases, they already are, though it's debatable whether it's ready to compete head-on with the major commercial offerings. You can buy white box switches and open software platforms to control them for SDN purposes right now, for example. But I'd say so far the main SDN suites haven't built up a great reputation and a lot of IT groups still prefer to source everything from one supplier with an expensive but all-encompassing support contract.

You can run commodity / white box switches in regular old mode, with routing protocols etc.

Just because you have a Broadcom chip doesn’t mean you need to use some new-fangled centralized / crontroller based control plane.

Sure. My point was just that the more free/open alternatives to newer technologies in the industry aren't necessarily advanced enough yet to displace coughing up whatever it takes to buy big name brands and their big budget support contracts so everything comes from one place.

A lot of this bad serial number hardware was built by the "night shift" at the same factory making the official hardware. These things just happen when you outsource.

That sort of hardware can last forever. At a previous employer, we had in the office a second-hand Cisco Catalyst 3548XL switch (it still had an asset tag from the previous company the boss had founded). According to the Cisco documentation I looked at back then, not only was that model long out-of-life, but also its successor was long out-of-life. And other than a couple of dead ports, it still worked perfectly; it won't surprise me at all if it's still in use today.

I interacted with a company in Germany recently that still had 3548s in service. Not the XL model which was GigE, this was 10/100 only. The last order date for the OG 3548 was July 2002.

I'm sorry for companies that were planning on recovering part of their investment in Cisco hardware by reselling it a couple years down the road.

Maybe I'm naive but I'm surprised to learn there is much of a 2nd hand market for 9000 series -class network hardware. I would have assumed that the huge complexity in managing a device like that without cisco support, alongside the difficulty in sourcing replacement parts for an EOL system built around redundancy, would make it very unattractive compared to a newer mid-range alternative. Are people really using unsupported hardware like this in their network?!

Disclaimer: I work for Nokia, we also make big switches, I believe we have a similar licensing model for certain products.

I don't know which particular "9000 series" you're referring to, but C9K (which the article refers to) isn't EOL. It should be straightforward to get parts for these, via multiple channels.

The deployment model for grey market bit is usually on networks that run on cheap models. They'll do things like buy one of the devices through white channel market and get SW access. Then, they'll turn around and buy a bunch more grey channel devices and use the SW access to upgrade, etc. As long as the device can handle the PPS and bitrates needed, its good enough for these networks. If it dies, buy another one and let it hang until it gets replaced. There are plenty of networks that just don't care about the operational issues.. and they love to keep cisco because of talent pipeline and training ecosystem.

Anyway, running equipment without vendor support is fine. If its a config issue, the docs are okay. If it is a bug, 9 times out of 10 TAC throws a random bug ID and tells you to upgrade. Naturally, It becomes an issue more when you run on the periphery of what is supported.

The article talks about selling on hardware even once it's declared end of life and the article also gave the 9000 series as an example of hardware that requires a license. I assumed there was something similar going on already with the 9000's predecessor (the 8000?). I didn't mean to imply that the 9000 is already eol.

It must be hell for these guys when they hit a show-stopper in old sw and they have no access to the fixed sw for their old hw.

It's sad that used goods are considered a "grey market" and not a fully legitimate market like selling any used goods should be.

There are large parts of the world where engineers are cheaper than hardware.

So basically Cisco is useless on airgapped networks now?

Is this really any different from what MS does with Windows volume licensing?

If you don't like the license model of IOS (etc), buy hardware that is supported by free/open source OS instead.

Yes, it is different, but it doesn't matter; comparison to another companies strategy does not negate the issue because comparison to another company does not change what Cisco is doing.

I think to make that an apt comparison, you'd be buying hardware from Microsoft that only worked with Windows, and only with volume licences. So when sold second-hand they'd be essentially nonfunctional.

By having non-transferable licenses, Cisco will be able to differentiating pricing - ie. Governments pay extra, or non-profits get a discount.

I can only hope that going forward we'll see less cisco stuff in datacenters.

'Pre-owned' ? Second hand. All this whitewashing of terms is annoying. Just use the term that we all used to begin with.

“Used.”

Ok, we've used used above.

The whole language irks me. "Entitlement" calls out the new idea that you are allowed to use your hardware or not. The whole point of license code is to prevent the thing from working in some cases.

The last thing I want is another failure mode in my critical path.

I have a nice stack of used routers and firewalls, cost a small fortune. Good luck on ever trusting those in a critical application, and by their very nature that's what they are for. So I can see some of the reasoning behind this, but once you have physical possession of a device it would be a relatively small affair to open it up, flip a jumper to 'yes I mean it' and then to be able to get it back to the same condition in which it left the factory.

But since smartphones are mere terminals and computers are trending that way too it won't be long before a freely programmable computer will be a rarity.

Software distribution is now centralized to the point that we have lost a lot compared to just two decades ago. Freeware and shareware would be unthinkable today, direct-to-the-user by virtue of downloading from the creator. App-stores and self-bricking hardware are the new normal.

master / slave ....

It's not their job to make that easy. Their job is to serve their customers and those folks are not.

They have no business relationship with the buyer of second hand Cisco gear.

If you don't like it, but something else.

I disagree because being able to resell will also factor into the first hand buyers expectation to recooperate values by selling it. If it is made known reselling is difficult then there will be diminished first hand buyers. So being able to resell is also a feature of a product.

The laws need to be changed to make this illegal. It's extremely damaging to the environment and should be banned.

The argument for keeping Capitalism is that people like owning stuff and don't want to share their property.

But capitalism itself is going concentrate ownership, removing it from the vast majority of people. At that point what's the point?

Speaking of Cisco, if the Soviet Union lasted 10 more years, would we all have IPv6 by now? (Not to say that I want more Soviet Union.)

Reply on news.ycombinator.com