Cisco can't stop using hard-coded passwords
schneier.com
schneier.com
Can you already guess what has happened next? :)
The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.
PS: just saying, no need to involve NSA in something that is better and more often explained by stupidity. :)
Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked?
My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.
I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired.
Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.
Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.
The solution is trivial: no commission on this. Possibly even negative commission, can't convince your client not to take this? You get less money for the sale.
This way sales are incentivised to only talk about this if not doing so would lose them the sale entirely.
They're trying to build a car or a chair or an appliance. To them, anything that gets in the way of that goal is bad, anything that supports it is good. Sure, the customer's engineering department had a grand vision of skilled shift leads empowered to look up the fault diagnostics, call over a QC engineer, and get two-party authorization to ignore the fault and continue...but once it hit the floor, and getting QC over to the machine took a whopping 4 minutes, the operators all memorized their lead's and QC's passwords and were rewarded for cutting TAKT times by 3 minutes and 50 seconds. Yes, bad product containment is a huge deal in the boardroom, but there's almost no will to enforce it on the production floor.
For b2b or b2c your customers don’t want to sit with you to make your product better.
Yes they want your product better.
Myself as a customer I don’t spend time filling in customer reviews. I just want to go on with my life.
I see the same in company I work for, our customers don’t have time - they have stuff to be done - either our software helps them achieve their goals or they leave.
* https://techcrunch.com/2018/10/05/california-passes-law-that...
I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.
Hardcoded password is different from default passwords. You can't change a hardcoded password, it's always part of the source code/image.
A default password can be changed.
Both can be backdoors if kept secret, but a hardcoded password from the onset is a backdoor.
Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password?
Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer?
How would that be any improvement, if the latter?
Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.
This isn't a Latvian company, by chance, is it? Blink twice if yes.
The customer is usually wrong.
You could set that password just for that one customer, and there is no problem.
Daydream: A new law puts legal liability for the consequences of hard-coded passwords on the manufacturers and their distribution chains. (The latter so Amazon can't sell root-holed crap from Lucky Good Fast Router Co. this week, then that company vanishes and Amazon sells amazingly similar crap from Good Lucky Fast Router Co. next week, then that company vanishes and ...)
https://hg.mozilla.org/comm-central/file/tip/mailnews/base/s...
I don't quite understand how we ended up in this situation. These passwords are more like user-agent strings. But they are still passwords in the sense that without any of them, you can't access (the authentication providers for) those online services.
> A vulnerability in Cisco Emergency Responder could allow an unauthenticated, remote attacker to log in to an affected device using the root account, which has default, static credentials that cannot be changed or deleted.
And I'm not sure these client-side secrets/group passwords are entirely meaningless. Some people somewhere must ascribe some security-related property to them.
If Thunderbird shared their Netflix password in here, that wouldn’t say anything about the quality of Netflix’s security implementation or the security value of Netflix passwords in general - it tells you Thunderbird don’t care about how their Netflix account gets used.
And in terms of the structural difference - this is of course a hardcoded credential used to identify this client to a service - the case at hand is about a hardcoded credential authentication embedded in a service, used to authenticate clients. We can distinguish those cases regulatorily.
Blame anti-cybercrime laws and weird interpretations of these for that one. The idea was that, as application level secrets (client_id/client_secret) in OAuth are scoped to a specific legal entity requesting them, site operators like Facebook, Twitter and others could request a takedown of third party apps at app stores for using the secrets of other apps.
Even cheap routers start putting passwords on them that look randomized.
Look, it shouldn't be that hard. One would expect that since manufacturers surely can produce unique serial numbers and MAC addresses and put them both into hardware and on the sticker on the bottom, a unique pair of credentials should be a piece of cake. Right? Right?
I've worked for a lot of large enterprises, and they all are the same. Nothing special about Cisco or any others. Security is hard, and it does not provide a good return to invest in good security. You can stop using hardcoded passwords, but it won't make you any more money, and customers clearly aren't leaving because of them. The stock price is the only thing that matters. As long as that's safe, it's business as usual.
Cisco has security practices?
Now, if you're a big vendor, do you give more resources to the PBX team that builds features which sells PBX customers, or the add on team used by a fraction of a percent of the worldwide PBX install base? I'm not asking what you should do, but rather what your inner MBA would do...
0 - the fundamental purposes of Emergency Responder are to a) provide enhanced location information about a caller, say the general location within a building, and b) to allow emergency responders to call back the person who placed an emergency call, even if the phone they called from does not have a directly callable phone number. It does some other things, too, but those are the "primary" features.
1 - source: sold a whole bunch of PBXes.
If you wanted to make some juice at the cost of a lot of squeeze, legacy support is always needing self-abusive people who can keep fragile crap going.
What I don't get is why these passwords are baked into the firmware instead of a "split secret" scheme - basically, a user in trouble would read some sort of identifier from a label on the inside of the hardware in question to the support, who'd then use a secret on their side together with the identifier from the customer to generate the backdoor password.
That way they'd avoid someone brute-forcing or leaking the backdoor password and compromising fleets of devices in the field, while still creating a paper trail of who has access to that password.
Or, you know, do what Kevin Mitnick did - call up local staff and ask them to read it out (dial-in number to FBI phone tap system...)?
Cisco isn't perfect by any stretch of the imagination, but the corporation cares about security even if some of the individual employees don't care to the same extent.
("This is fine".)
Compared to other manufacturers that have default passwords printed on the device or something made from the serial number. When that hardware could be shoved in behind a ceiling panel somewhere (even chances of that hardware not being where it's supposed to be).
_____
It's clearly not the "right answer", but it is an answer.
Conspiracy people always gonna conspiracy though, and sometimes they're right, and whether it's an intentional backdoor or not, it is a backdoor.
There is a team working for Cisco who does not care for quality who decided it is a good idea to have a baked in password and who do not learn from past mistakes.
The core of the problem is outsourcing your core product to people who are neither competent nor interested in the quality of the product.
Every time I see any company thinking they can push their core responsibility to somebody else who's interests are not aligned, I can confidently say it is going to go downhill and fail at some point.
But I don't think it is malice. The engineer in me says that there are just too many eyes looking at the software and that it is just too easy for somebody to identify a password and that the consequences of it are too costly for the company.
And I can think of a zillion ways to leave a backdoor that is much more difficult to detect.
Of course, this does not absolve them from responsibility in any way. The core responsibility, the core product they provide is security. And not only they are showing they can't provide a secure product, they prove they simply do not learn and do not care to fix it.
You know what else is expensive? You need to troubleshoot some gear but you can't get in easily. Sophisticated procedures for getting into a router take time and skill to implement and to use, so they are, yes, you guessed it right: Expensive! What is not expensive? Having a surefire set of shared credentials that will work on bloody anything, that's what!
But inside CISCO even the most basic code review should have caught this, besides that even for test purposes they should have never ever implemented this.
Inside the MBA ideology, that "long term very expensive" doesn't exist. Lowering costs is good long term, because it grows the market and gives the company more money to invest into competitive advantage and grow its brand.
Getting your company disrupted or getting your market share diminished is extremely expensive and this is my understanding of what Jacques meant.
What kind of shiny rectangle do you have in your pocket. Is it the cheapest option? Which company in that market makes the most expensive product? Which company in the market fares best?
And Cisco itself, for decades, was not only the most expensive network equipment in multiple markets but also bringing highest margins and biggest bucks.
Do people who post these silly hyperboles sincerely believe them? There’s plenty of other reasons to outsource something, including many where it’s known from the outset that it would be the more expensive option.
I don't have anything against outsourcing. There is plenty of reasons to outsource, true. But there is one that is most important and IT IS NOT THE COST. I am not saying the cost is not important, I am just saying it is not the most important.
The most important reason is that you want to focus on your core product and not on everything else. If your core product that you sell to your customers are routers, if you are the CEO of the company, you probably do not want to oversee a division that produces accounting software or remote desktop software. If you need these things, you either buy them or outsource them.
You want to spend all your focus producing the best product for your customers, and this is routers. Not anything else that does not directly go into making better experience for your costumers.
It usually is or at least, in principle, should be cheaper. Because for some reason if somebody cares for the product, the total cost to produce the product at a given quality level will be lower.
Because from the looks of it, if I assume what you're saying is 100% true, the product is sales contracts for the routers and not the routers themselves.
I'm afraid the core of the problem is customers focusing on price more than on the quality of the product.
Lots of companies thrive without trying to be cheapest in the market. In most markets there is "expensive", but quality option and then there is a cheap option that everybody knows may be less quality. Companies make these choices all the time and those more expensive options are still successful.
And Cisco is actually positioned as an expensive option. For decades in many segments of the market, Cisco was the top option that you had to pay for through the nose.
Of course they can't. CIA and NSA need access.
The problem that i see is that this " stupid, lazy engineering" occurs very often at Cisco.
I would have expected that they have learned from past mistakes, but at the company i work for, we also don't learn.
Market forces just don't seem to work on these things.
I don't know what the answer is but putting the fate of millions of customers in the hands of one person's conscience, because of a knowledge imbalance, just doesn't seem right.
I can imagine a technocracy having a licensing scheme for suppliers, where Cisco would be told, this is substandard we'll pull your license. But, that sorry if structure just seems like it would fail to corruption so easily.
How do we improve standards over small, but impactful, parts of life that most politicians wouldn't understand; things that the market doesn't begin to address?
Like the fable of the Dog and the Wolf, do we need to forgo some liberty to acquire the comfort of progress?
No risk for free (as in beer) software like most OSS, since the total price is $0, but risk for paid services & software.
The problem is sloppy processes lacking protected networks for staging and deployment that properly configures said network gear with a unique password or TACACS+ auth.
I humbly disagree. However, this is much worse, anyway - it's a static root password:
> A vulnerability in Cisco Emergency Responder could allow an unauthenticated, remote attacker to log in to an affected device using the root account, which has default, static credentials that cannot be changed or deleted.
> This vulnerability is due to the presence of static user credentials for the root account that are typically reserved for use during development.
https://sec.cloudapps.cisco.com/security/center/content/Cisc...
I don't care about Cisco, so they can continue footgunning themselves in obviously stupid ways. It sounds like such a Cisco user should or must have an ability to be disabled. If not, that's fucking moronic. But really, what kind of morons put what should be their internal management network on the fucking internet and don't have TACACS+?
I didn't mean to imply you was talking about something else - I tried to make two separate points:
1) I don't think default passwords are "no problem"
2) This Cisco thing is anyway much worse than that.
As for what you bring up:
3) But really, what kind of morons put what should be their internal management network on the fucking internet and don't have TACACS+?
Indeed. Sadly 3) doesn't surprise me; it makes both 1 and 2 much worse (1 because the people that give us 3 aren't likely to change the default - even if they could - and with 2 they can't).
https://sec.cloudapps.cisco.com/security/center/content/Cisc...
Ed: ... But judging from this... "cisco"?
https://www.lifewire.com/cisco-default-password-list-2619151